Important

You are viewing the archived documentation of TiDB, which no longer receives updates. It is recommended that you use the latest LTS version of the TiDB database.

Deploy TiDB Binlog

This document describes how to maintain TiDB Binlog of a TiDB cluster in Kubernetes.

Prerequisites

Deploy TiDB Operator;
Install Helm and configure it with the official PingCAP chart.

Deploy TiDB Binlog in a TiDB cluster

TiDB Binlog is disabled in the TiDB cluster by default. To create a TiDB cluster with TiDB Binlog enabled, or enable TiDB Binlog in an existing TiDB cluster, take the following steps.

Deploy Pump

Modify the TidbCluster CR file to add the Pump configuration.

For example:

spec:
  ...
  pump:
    baseImage: pingcap/tidb-binlog
    version: v5.0.6
    replicas: 1
    storageClassName: local-storage
    requests:
      storage: 30Gi
    schedulerName: default-scheduler
    config:
      addr: 0.0.0.0:8250
      gc: 7
      heartbeat-interval: 2

Since v1.1.6, TiDB Operator supports passing raw TOML configuration to the component:

spec:
  ...
  pump:
    baseImage: pingcap/tidb-binlog
    version: v5.0.6
    replicas: 1
    storageClassName: local-storage
    requests:
      storage: 30Gi
    schedulerName: default-scheduler
    config: |
      addr = "0.0.0.0:8250"
      gc = 7
      heartbeat-interval = 2

Edit version, replicas, storageClassName, and requests.storage according to your cluster.

Set affinity and anti-affinity for TiDB and Pump.

If you enable TiDB Binlog in the production environment, it is recommended to set affinity and anti-affinity for TiDB and the Pump component; if you enable TiDB Binlog in a test environment on the internal network, you can skip this step.

By default, the affinity of TiDB and Pump is set to {}. Currently, each TiDB instance does not have a corresponding Pump instance by default. When TiDB Binlog is enabled, if Pump and TiDB are separately deployed and network isolation occurs, and ignore-error is enabled in TiDB components, TiDB loses binlogs.

In this situation, it is recommended to deploy a TiDB instance and a Pump instance on the same node using the affinity feature, and to split Pump instances on different nodes using the anti-affinity feature. For each node, only one Pump instance is required. The steps are as follows:

Configure spec.tidb.affinity as follows:

spec:
  tidb:
    affinity:
      podAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: "app.kubernetes.io/component"
                operator: In
                values:
                - "pump"
              - key: "app.kubernetes.io/managed-by"
                operator: In
                values:
                - "tidb-operator"
              - key: "app.kubernetes.io/name"
                operator: In
                values:
                - "tidb-cluster"
              - key: "app.kubernetes.io/instance"
                operator: In
                values:
                - ${cluster_name}
            topologyKey: kubernetes.io/hostname

Configure spec.pump.affinity as follows:

spec:
  pump:
    affinity:
      podAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: "app.kubernetes.io/component"
                operator: In
                values:
                - "tidb"
              - key: "app.kubernetes.io/managed-by"
                operator: In
                values:
                - "tidb-operator"
              - key: "app.kubernetes.io/name"
                operator: In
                values:
                - "tidb-cluster"
              - key: "app.kubernetes.io/instance"
                operator: In
                values:
                - ${cluster_name}
            topologyKey: kubernetes.io/hostname
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: "app.kubernetes.io/component"
                operator: In
                values:
                - "pump"
              - key: "app.kubernetes.io/managed-by"
                operator: In
                values:
                - "tidb-operator"
              - key: "app.kubernetes.io/name"
                operator: In
                values:
                - "tidb-cluster"
              - key: "app.kubernetes.io/instance"
                operator: In
                values:
                - ${cluster_name}
            topologyKey: kubernetes.io/hostname

Note

If you update the affinity configuration of the TiDB components, it will cause rolling updates of the TiDB components in the cluster.

Deploy Drainer

To deploy multiple drainers using the tidb-drainer Helm chart for a TiDB cluster, take the following steps:

Make sure that the PingCAP Helm repository is up to date:
```
helm repo update
```
```
helm search repo tidb-drainer -l
```

Get the default values.yaml file to facilitate customization:

helm inspect values pingcap/tidb-drainer --version=${chart_version} > values.yaml

Modify the values.yaml file to specify the source TiDB cluster and the downstream database of the drainer. Here is an example:
```
clusterName: example-tidb
clusterVersion: v5.0.6
baseImage:pingcap/tidb-binlog
storageClassName: local-storage
storage: 10Gi
initialCommitTs: "-1"
config: |
  detect-interval = 10
  [syncer]
  worker-count = 16
  txn-batch = 20
  disable-dispatch = false
  ignore-schemas = "INFORMATION_SCHEMA,PERFORMANCE_SCHEMA,mysql"
  safe-mode = false
  db-type = "tidb"
  [syncer.to]
  host = "downstream-tidb"
  user = "root"
  password = ""
  port = 4000
```
The clusterName and clusterVersion must match the desired source TiDB cluster.
The initialCommitTs is the starting commit timestamp of data replication when Drainer has no checkpoint. The value must be set as a string type, such as "424364429251444742".
For complete configuration details, refer to TiDB Binlog Drainer Configurations in Kubernetes.
Deploy Drainer:
```
helm install ${release_name} pingcap/tidb-drainer --namespace=${namespace} --version=${chart_version} -f values.yaml
```
If the server does not have an external network, refer to deploy the TiDB cluster to download the required Docker image on the machine with an external network and upload it to the server.
Note
This chart must be installed to the same namespace as the source TiDB cluster.

Enable TLS

Enable TLS between TiDB components

If you want to enable TLS for the TiDB cluster and TiDB Binlog, refer to Enable TLS between Components.

After you have created a secret and started a TiDB cluster with Pump, edit the values.yaml file to set the tlsCluster.enabled value to true, and configure the corresponding certAllowedCN:

...
tlsCluster:
  enabled: true
  # certAllowedCN:
  #  - TiDB
...

Enable TLS between Drainer and the downstream database

If you set the downstream database of tidb-drainer to mysql/tidb, and if you want to enable TLS between Drainer and the downstream database, take the following steps.

Create a secret that contains the TLS information of the downstream database.

kubectl create secret generic ${downstream_database_secret_name} --namespace=${namespace} --from-file=tls.crt=client.pem --from-file=tls.key=client-key.pem --from-file=ca.crt=ca.pem

tidb-drainer saves the checkpoint in the downstream database by default, so you only need to configure tlsSyncer.tlsClientSecretName and the corresponding cerAllowedCN:

tlsSyncer:
  tlsClientSecretName: ${downstream_database_secret_name}
  # certAllowedCN:
  #  - TiDB

To save the checkpoint of tidb-drainer to other databases that have enabled TLS, create a secret that contains the TLS information of the checkpoint database:

kubectl create secret generic ${checkpoint_tidb_client_secret} --namespace=${namespace} --from-file=tls.crt=client.pem --from-file=tls.key=client-key.pem --from-file=ca.crt=ca.pem

Edit the values.yaml file to set the tlsSyncer.checkpoint.tlsClientSecretName value to ${checkpoint_tidb_client_secret}, and configure the corresponding certAllowedCN:

...
tlsSyncer: {}
  tlsClientSecretName: ${downstream_database_secret_name}
  # certAllowedCN:
  #  - TiDB
  checkpoint:
    tlsClientSecretName: ${checkpoint_tidb_client_secret}
    # certAllowedCN:
    #  - TiDB
...

Remove Pump/Drainer nodes

For details on how to maintain the node state of the TiDB Binlog cluster, refer to Starting and exiting a Pump or Drainer process.

If you want to remove the TiDB Binlog component completely, it is recommended that you first remove Pump nodes and then remove Drainer nodes.

If TLS is enabled for the TiDB Binlog component to be removed, write the following content into binlog.yaml and execute kubectl apply -f binlog.yaml to start a Pod that is mounted with the TLS file and the binlogctl tool.

apiVersion: v1
kind: Pod
metadata:
  name: binlogctl
spec:
  containers:
  - name: binlogctl
    image: pingcap/tidb-binlog:${tidb_version}
    command: ['/bin/sh']
    stdin: true
    stdinOnce: true
    tty: true
    volumeMounts:
      - name: binlog-tls
        mountPath: /etc/binlog-tls
  volumes:
    - name: binlog-tls
      secret:
        secretName: ${cluster_name}-cluster-client-secret

Scale in Pump

To scale in Pump, you need to take a single Pump node offline, and execute kubectl edit tc ${cluster_name} -n ${namespace} to reduce the value of replicas of Pump by 1. Repeat the operations on each node.

The steps are as follows:

Take the Pump node offline.

Assume there are three Pump nodes in the cluster. You need to take the third node offline and replace ${ordinal_id} with 2. (${tidb_version} is the current TiDB version.)

If TLS is not enabled for Pump, create a Pod to take Pump offline:

kubectl run offline-pump-${ordinal_id} --image=pingcap/tidb-binlog:${tidb_version} --namespace=${namespace} --restart=OnFailure -- /binlogctl -pd-urls=http://${cluster_name}-pd:2379 -cmd offline-pump -node-id ${cluster_name}-pump-${ordinal_id}:8250

If TLS is enabled for Pump, use the previously started Pod to take Pump offline:

kubectl exec binlogctl -n ${namespace} -- /binlogctl -pd-urls "https://${cluster_name}-pd:2379" -cmd offline-pump -node-id ${cluster_name}-pump-${ordinal_id}:8250 -ssl-ca "/etc/binlog-tls/ca.crt" -ssl-cert "/etc/binlog-tls/tls.crt" -ssl-key "/etc/binlog-tls/tls.key"

View the log of Pump by executing the following command:

kubectl logs -f -n ${namespace} ${release_name}-pump-${ordinal_id}

If pump offline, please delete my pod is output, this node is successfully taken offline.

Delete the corresponding Pump Pod:
Execute kubectl edit tc ${cluster_name} -n ${namespace} to change spec.pump.replicas to 2, and wait until the Pump Pod is taken offline and deleted automatically.

(Optional) Force Pump to go offline:

If the offline operation fails, the Pump Pod will not output pump offline, please delete my pod. At this time, you can force Pump to go offline, that is, taking Step 2 to reduce the value of spec.pump.replicas and mark Pump as offline after the Pump Pod is deleted completely.

If TLS is not enabled for Pump, mark Pump as offline:

kubectl run update-pump-${ordinal_id} --image=pingcap/tidb-binlog:${tidb_version} --namespace=${namespace} --restart=OnFailure -- /binlogctl -pd-urls=http://${cluster_name}-pd:2379 -cmd update-pump -node-id ${cluster_name}-pump-${ordinal_id}:8250 --state offline

If TLS is enabled for Pump, mark Pump as offline using the previously started Pod:

kubectl exec binlogctl -n ${namespace} -- /binlogctl -pd-urls=https://${cluster_name}-pd:2379 -cmd update-pump -node-id ${cluster_name}-pump-${ordinal_id}:8250 --state offline -ssl-ca "/etc/binlog-tls/ca.crt" -ssl-cert "/etc/binlog-tls/tls.crt" -ssl-key "/etc/binlog-tls/tls.key"

Remove Pump nodes completely

Note

Before performing the following steps, you need to have at least one Pump node in the cluster. If you have scaled in Pump nodes to 0, you need to scale out Pump at least to 1 node before you perform the removing operation in this section.
To scale out the Pump to 1, execute kubectl edit tc ${tidb-cluster} -n ${namespace} and modify the spec.pump.replicas to 1.

Before removing Pump nodes, execute kubectl edit tc ${cluster_name} -n ${namespace} and set spec.tidb.binlogEnabled to false. After the TiDB Pods are rolling updated, you can remove the Pump nodes.
If you directly remove Pump nodes, it might cause TiDB failure because TiDB has no Pump nodes to write into.
Refer to Scale in Pump to scale in Pump to 0.
Execute kubectl edit tc ${cluster_name} -n ${namespace} and delete all configuration items of spec.pump.
Execute kubectl delete sts ${cluster_name}-pump -n ${namespace} to delete the StatefulSet resources of Pump.
View PVCs used by the Pump cluster by executing kubectl get pvc -n ${namespace} -l app.kubernetes.io/component=pump. Then delete all the PVC resources of Pump by executing kubectl delete pvc -l app.kubernetes.io/component=pump -n ${namespace}.

Remove Drainer nodes

Take Drainer nodes offline:
In the following commands, ${drainer_node_id} is the node ID of the Drainer node to be taken offline. If you have configured drainerName in values.yaml of Helm, the value of ${drainer_node_id} is ${drainer_name}-0; otherwise, the value of ${drainer_node_id} is ${cluster_name}-${release_name}-drainer-0.
- If TLS is not enabled for Drainer, create a Pod to take Drainer offline:
```
kubectl run offline-drainer-0 --image=pingcap/tidb-binlog:${tidb_version} --namespace=${namespace} --restart=OnFailure -- /binlogctl -pd-urls=http://${cluster_name}-pd:2379 -cmd offline-drainer -node-id ${drainer_node_id}:8249
```
- If TLS is enabled for Drainer, use the previously started Pod to take Drainer offline:
```
kubectl exec binlogctl -n ${namespace} -- /binlogctl -pd-urls "https://${cluster_name}-pd:2379" -cmd offline-drainer -node-id ${drainer_node_id}:8249 -ssl-ca "/etc/binlog-tls/ca.crt" -ssl-cert "/etc/binlog-tls/tls.crt" -ssl-key "/etc/binlog-tls/tls.key"
```
View the log of Drainer by executing the following command:
```
kubectl logs -f -n ${namespace} ${drainer_node_id}
```
If drainer offline, please delete my pod is output, this node is successfully taken offline.
Delete the corresponding Drainer Pod:
Execute helm uninstall ${release_name} -n ${namespace} to delete the Drainer Pod.
If you no longer need Drainer, execute kubectl delete pvc data-${drainer_node_id} -n ${namespace} to delete the PVC resources of Drainer.

(Optional) Force Drainer to go offline:

If the offline operation fails, the Drainer Pod will not output drainer offline, please delete my pod. At this time, you can force Drainer to go offline, that is, taking Step 2 to delete the Drainer Pod and mark Drainer as offline.

If TLS is not enabled for Drainer, mark Drainer as offline:

kubectl run update-drainer-${ordinal_id} --image=pingcap/tidb-binlog:${tidb_version} --namespace=${namespace} --restart=OnFailure -- /binlogctl -pd-urls=http://${cluster_name}-pd:2379 -cmd update-drainer -node-id ${drainer_node_id}:8249 --state offline

If TLS is enabled for Drainer, use the previously started Pod to take Drainer offline:

kubectl exec binlogctl -n ${namespace} -- /binlogctl -pd-urls=https://${cluster_name}-pd:2379 -cmd update-drainer -node-id ${drainer_node_id}:8249 --state offline -ssl-ca "/etc/binlog-tls/ca.crt" -ssl-cert "/etc/binlog-tls/tls.crt" -ssl-key "/etc/binlog-tls/tls.key"