TiDB Binlog 部署拓扑

本文介绍在部署最小拓扑集群的基础上,同时部署 TiDB Binlog。TiDB Binlog 是目前广泛使用的增量同步组件,可提供准实时备份和同步功能。

拓扑信息

实例个数物理机配置IP配置
TiDB316 VCore 32 GB10.0.1.1
10.0.1.2
10.0.1.3
默认端口配置;
开启 enable_binlog;
开启 ignore-error
PD34 VCore 8 GB10.0.1.4
10.0.1.5
10.0.1.6
默认端口配置
TiKV316 VCore 32 GB10.0.1.7
10.0.1.8
10.0.1.9
默认端口配置
Pump38 VCore 16GB10.0.1.1
10.0.1.7
10.0.1.8
默认端口配置;
设置 GC 时间 7 天
Drainer18 VCore 16GB10.0.1.12默认端口配置;
设置默认初始化 commitTS -1 为最近的时间戳
配置下游目标 TiDB 10.0.1.12:4000

拓扑模版

简单 TiDB Binlog 配置模板(下游为 MySQL)
# # Global variables are applied to all deployments and used as the default value of # # the deployments if a specific deployment value is missing. global: user: "tidb" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" server_configs: tidb: binlog.enable: true binlog.ignore-error: true pd_servers: - host: 10.0.1.4 - host: 10.0.1.5 - host: 10.0.1.6 tidb_servers: - host: 10.0.1.1 - host: 10.0.1.2 - host: 10.0.1.3 tikv_servers: - host: 10.0.1.7 - host: 10.0.1.8 - host: 10.0.1.9 pump_servers: - host: 10.0.1.1 - host: 10.0.1.2 - host: 10.0.1.3 drainer_servers: - host: 10.0.1.12 config: syncer.db-type: "tidb" syncer.to.host: "10.0.1.12" syncer.to.user: "root" syncer.to.password: "" syncer.to.port: 4000 monitoring_servers: - host: 10.0.1.10 grafana_servers: - host: 10.0.1.10 alertmanager_servers: - host: 10.0.1.10
简单 TiDB Binlog 配置模板(下游为 file)
# # Global variables are applied to all deployments and used as the default value of # # the deployments if a specific deployment value is missing. global: user: "tidb" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" server_configs: tidb: binlog.enable: true binlog.ignore-error: true pd_servers: - host: 10.0.1.4 - host: 10.0.1.5 - host: 10.0.1.6 tidb_servers: - host: 10.0.1.1 - host: 10.0.1.2 - host: 10.0.1.3 tikv_servers: - host: 10.0.1.7 - host: 10.0.1.8 - host: 10.0.1.9 pump_servers: - host: 10.0.1.1 - host: 10.0.1.2 - host: 10.0.1.3 drainer_servers: - host: 10.0.1.12 # drainer meta data directory path data_dir: "/path/to/save/data" config: syncer.db-type: "file" # directory to save binlog file, default same as data-dir(save checkpoint file) if this is not configured. # syncer.to.dir: "/path/to/save/binlog" monitoring_servers: - host: 10.0.1.10 grafana_servers: - host: 10.0.1.10 alertmanager_servers: - host: 10.0.1.10
详细 TiDB Binlog 配置模板
# # Global variables are applied to all deployments and used as the default value of # # the deployments if a specific deployment value is missing. global: user: "tidb" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" # # Monitored variables are applied to all the machines. monitored: node_exporter_port: 9100 blackbox_exporter_port: 9115 # deploy_dir: "/tidb-deploy/monitored-9100" # data_dir: "/tidb-data/monitored-9100" # log_dir: "/tidb-deploy/monitored-9100/log" # # Server configs are used to specify the runtime configuration of TiDB components. # # All configuration items can be found in TiDB docs: # # - TiDB: https://docs.pingcap.com/zh/tidb/stable/tidb-configuration-file # # - TiKV: https://docs.pingcap.com/zh/tidb/stable/tikv-configuration-file # # - PD: https://docs.pingcap.com/zh/tidb/stable/pd-configuration-file # # All configuration items use points to represent the hierarchy, e.g: # # readpool.storage.use-unified-pool # # # # You can overwrite this configuration via the instance-level `config` field. server_configs: tidb: log.slow-threshold: 300 binlog.enable: true binlog.ignore-error: true tikv: # server.grpc-concurrency: 4 # raftstore.apply-pool-size: 2 # raftstore.store-pool-size: 2 # rocksdb.max-sub-compactions: 1 # storage.block-cache.capacity: "16GB" # readpool.unified.max-thread-count: 12 readpool.storage.use-unified-pool: false readpool.coprocessor.use-unified-pool: true pd: schedule.leader-schedule-limit: 4 schedule.region-schedule-limit: 2048 schedule.replica-schedule-limit: 64 pd_servers: - host: 10.0.1.4 # ssh_port: 22 # name: "pd-1" # client_port: 2379 # peer_port: 2380 # deploy_dir: "/tidb-deploy/pd-2379" # data_dir: "/tidb-data/pd-2379" # log_dir: "/tidb-deploy/pd-2379/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.pd` values. # config: # schedule.max-merge-region-size: 20 # schedule.max-merge-region-keys: 200000 - host: 10.0.1.5 - host: 10.0.1.6 tidb_servers: - host: 10.0.1.1 # ssh_port: 22 # port: 4000 # status_port: 10080 # deploy_dir: "/tidb-deploy/tidb-4000" # log_dir: "/tidb-deploy/tidb-4000/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.tidb` values. # config: # log.slow-query-file: tidb-slow-overwrited.log - host: 10.0.1.2 - host: 10.0.1.3 tikv_servers: - host: 10.0.1.7 # ssh_port: 22 # port: 20160 # status_port: 20180 # deploy_dir: "/tidb-deploy/tikv-20160" # data_dir: "/tidb-data/tikv-20160" # log_dir: "/tidb-deploy/tikv-20160/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.tikv` values. # config: # server.grpc-concurrency: 4 # server.labels: { zone: "zone1", dc: "dc1", host: "host1" } - host: 10.0.1.8 - host: 10.0.1.9 pump_servers: - host: 10.0.1.1 ssh_port: 22 port: 8250 deploy_dir: "/tidb-deploy/pump-8250" data_dir: "/tidb-data/pump-8250" # The following configs are used to overwrite the `server_configs.pump` values. config: gc: 7 - host: 10.0.1.2 ssh_port: 22 port: 8250 deploy_dir: "/tidb-deploy/pump-8250" data_dir: "/tidb-data/pump-8250" # The following configs are used to overwrite the `server_configs.pump` values. config: gc: 7 - host: 10.0.1.3 ssh_port: 22 port: 8250 deploy_dir: "/tidb-deploy/pump-8250" data_dir: "/tidb-data/pump-8250" # The following configs are used to overwrite the `server_configs.pump` values. config: gc: 7 drainer_servers: - host: 10.0.1.12 port: 8249 deploy_dir: "/tidb-deploy/drainer-8249" data_dir: "/tidb-data/drainer-8249" # If drainer doesn't have a checkpoint, use initial commitTS as the initial checkpoint. # Will get a latest timestamp from pd if commit_ts is set to -1 (the default value). commit_ts: -1 # The following configs are used to overwrite the `server_configs.drainer` values. config: syncer.db-type: "tidb" syncer.to.host: "10.0.1.12" syncer.to.user: "root" syncer.to.password: "" syncer.to.port: 4000 syncer.to.checkpoint: schema: "tidb_binlog" type: "tidb" host: "10.0.1.14" user: "root" password: "123" port: 4000 - host: 10.0.1.13 port: 8249 deploy_dir: "/tidb-deploy/drainer-8249" data_dir: "/tidb-data/drainer-8249" # If Drainer does not have a checkpoint, use the initial commitTS as the initial checkpoint. # If commit_ts is set to -1 (the default value), you will get a latest timestamp from PD. commit_ts: -1 # The following configurations are used to overwrite the `server_configs.drainer` values. config: syncer.db-type: "kafka" syncer.replicate-do-db: - db1 - db2 syncer.to.kafka-addrs: "10.0.1.20:9092,10.0.1.21:9092,10.0.1.22:9092" syncer.to.kafka-version: "2.4.0" syncer.to.topic-name: "asyouwish" monitoring_servers: - host: 10.0.1.10 # ssh_port: 22 # port: 9090 # deploy_dir: "/tidb-deploy/prometheus-8249" # data_dir: "/tidb-data/prometheus-8249" # log_dir: "/tidb-deploy/prometheus-8249/log" grafana_servers: - host: 10.0.1.10 # port: 3000 # deploy_dir: /tidb-deploy/grafana-3000 alertmanager_servers: - host: 10.0.1.10 # ssh_port: 22 # web_port: 9093 # cluster_port: 9094 # deploy_dir: "/tidb-deploy/alertmanager-9093" # data_dir: "/tidb-data/alertmanager-9093" # log_dir: "/tidb-deploy/alertmanager-9093/log"

以上 TiDB 集群拓扑文件中,详细的配置项说明见通过 TiUP 部署 TiDB 集群的拓扑文件配置

关键参数介绍

拓扑配置模版的关键参数如下:

  • server_configs.tidb.binlog.enable: true

    开启 TiDB Binlog 服务,默认为 false。

  • server_configs.tidb.binlog.ignore-error: true

    高可用场景建议开启,如果设置为 true,发生错误时,TiDB 会停止写入 TiDB Binlog,并且在监控项 tidb_server_critical_error_total 上计数加 1;如果设置为 false,一旦写入 TiDB Binlog 失败,会停止整个 TiDB 的服务。

  • drainer_servers.config.syncer.db-type

    TiDB Binlog 的下游类型,目前支持 mysqltidbkafkafile

  • drainer_servers.config.syncer.to

    TiDB Binlog 的下游配置。根据 db-type 的不同,该选项可配置下游数据库的连接参数、Kafka 的连接参数、文件保存路径。详细说明可参见 TiDB Binlog 配置说明