TiFlash 部署拓扑

本文介绍在部署最小拓扑集群的基础上,部署 TiFlash 的拓扑结构。TiFlash 是列式的存储引擎,已经成为集群拓扑的标配,适合 Real-Time HTAP 业务。

拓扑信息

实例个数物理机配置IP配置
TiDB316 VCore 32GB * 110.0.1.7
10.0.1.8
10.0.1.9
默认端口
全局目录配置
PD34 VCore 8GB * 110.0.1.4
10.0.1.5
10.0.1.6
默认端口
全局目录配置
TiKV316 VCore 32GB 2TB (nvme ssd) * 110.0.1.1
10.0.1.2
10.0.1.3
默认端口
全局目录配置
TiFlash132 VCore 64 GB 2TB (nvme ssd) * 110.0.1.11默认端口
全局目录配置
Monitoring & Grafana14 VCore 8GB * 1 500GB (ssd)10.0.1.10默认端口
全局目录配置

拓扑模版

简单 TiFlash 配置模版
# # Global variables are applied to all deployments and used as the default value of # # the deployments if a specific deployment value is missing. global: user: "tidb" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" server_configs: pd: replication.enable-placement-rules: true pd_servers: - host: 10.0.1.4 - host: 10.0.1.5 - host: 10.0.1.6 tidb_servers: - host: 10.0.1.7 - host: 10.0.1.8 - host: 10.0.1.9 tikv_servers: - host: 10.0.1.1 - host: 10.0.1.2 - host: 10.0.1.3 tiflash_servers: - host: 10.0.1.11 data_dir: /tidb-data/tiflash-9000 deploy_dir: /tidb-deploy/tiflash-9000 monitoring_servers: - host: 10.0.1.10 grafana_servers: - host: 10.0.1.10 alertmanager_servers: - host: 10.0.1.10
详细 TiFlash 配置模版
# # Global variables are applied to all deployments and used as the default value of # # the deployments if a specific deployment value is missing. global: user: "tidb" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" # # Monitored variables are applied to all the machines. monitored: node_exporter_port: 9100 blackbox_exporter_port: 9115 # deploy_dir: "/tidb-deploy/monitored-9100" # data_dir: "/tidb-data/monitored-9100" # log_dir: "/tidb-deploy/monitored-9100/log" # # Server configs are used to specify the runtime configuration of TiDB components. # # All configuration items can be found in TiDB docs: # # - TiDB: https://docs.pingcap.com/zh/tidb/stable/tidb-configuration-file # # - TiKV: https://docs.pingcap.com/zh/tidb/stable/tikv-configuration-file # # - PD: https://docs.pingcap.com/zh/tidb/stable/pd-configuration-file # # All configuration items use points to represent the hierarchy, e.g: # # readpool.storage.use-unified-pool # # # # You can overwrite this configuration via the instance-level `config` field. server_configs: tidb: log.slow-threshold: 300 tikv: # server.grpc-concurrency: 4 # raftstore.apply-pool-size: 2 # raftstore.store-pool-size: 2 # rocksdb.max-sub-compactions: 1 # storage.block-cache.capacity: "16GB" # readpool.unified.max-thread-count: 12 readpool.storage.use-unified-pool: false readpool.coprocessor.use-unified-pool: true pd: schedule.leader-schedule-limit: 4 schedule.region-schedule-limit: 2048 schedule.replica-schedule-limit: 64 replication.enable-placement-rules: true tiflash: # Maximum memory usage for processing a single query. Zero means unlimited. profiles.default.max_memory_usage: 0 # Maximum memory usage for processing all concurrently running queries on the server. Zero means unlimited. profiles.default.max_memory_usage_for_all_queries: 0 tiflash-learner: # The allowable number of threads in the pool that flushes Raft data to storage. raftstore.apply-pool-size: 4 # The allowable number of threads that process Raft, which is the size of the Raftstore thread pool. raftstore.store-pool-size: 4 pd_servers: - host: 10.0.1.4 # ssh_port: 22 # name: "pd-1" # client_port: 2379 # peer_port: 2380 # deploy_dir: "/tidb-deploy/pd-2379" # data_dir: "/tidb-data/pd-2379" # log_dir: "/tidb-deploy/pd-2379/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.pd` values. # config: # schedule.max-merge-region-size: 20 # schedule.max-merge-region-keys: 200000 - host: 10.0.1.5 - host: 10.0.1.6 tidb_servers: - host: 10.0.1.7 # ssh_port: 22 # port: 4000 # status_port: 10080 # deploy_dir: "/tidb-deploy/tidb-4000" # log_dir: "/tidb-deploy/tidb-4000/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.tidb` values. # config: # log.slow-query-file: tidb-slow-overwrited.log - host: 10.0.1.8 - host: 10.0.1.9 tikv_servers: - host: 10.0.1.1 # ssh_port: 22 # port: 20160 # status_port: 20180 # deploy_dir: "/tidb-deploy/tikv-20160" # data_dir: "/tidb-data/tikv-20160" # log_dir: "/tidb-deploy/tikv-20160/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.tikv` values. # config: # server.grpc-concurrency: 4 # server.labels: # zone: "zone1" # dc: "dc1" # host: "host1" - host: 10.0.1.2 - host: 10.0.1.3 tiflash_servers: - host: 10.0.1.11 # ssh_port: 22 # tcp_port: 9000 # flash_service_port: 3930 # flash_proxy_port: 20170 # flash_proxy_status_port: 20292 # metrics_port: 8234 # deploy_dir: "/tidb-deploy/tiflash-9000" ## The `data_dir` will be overwritten if you define `storage.main.dir` configurations in the `config` section. # data_dir: "/tidb-data/tiflash-9000" # log_dir: "/tidb-deploy/tiflash-9000/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.tiflash` values. # config: # logger.level: "info" # ## Multi-disk deployment introduced in v4.0.9 # ## Check https://docs.pingcap.com/tidb/stable/tiflash-configuration#multi-disk-deployment for more details. # ## Example1: # # storage.main.dir: [ "/nvme_ssd0_512/tiflash", "/nvme_ssd1_512/tiflash" ] # # storage.main.capacity: [ 536870912000, 536870912000 ] # ## Example2: # # storage.main.dir: [ "/sata_ssd0_512/tiflash", "/sata_ssd1_512/tiflash", "/sata_ssd2_512/tiflash" ] # # storage.latest.dir: [ "/nvme_ssd0_150/tiflash" ] # # storage.main.capacity: [ 536870912000, 536870912000, 536870912000 ] # # storage.latest.capacity: [ 161061273600 ] # learner_config: # log-level: "info" # server.labels: # zone: "zone2" # dc: "dc2" # host: "host2" # - host: 10.0.1.12 # - host: 10.0.1.13 monitoring_servers: - host: 10.0.1.10 # ssh_port: 22 # port: 9090 # deploy_dir: "/tidb-deploy/prometheus-8249" # data_dir: "/tidb-data/prometheus-8249" # log_dir: "/tidb-deploy/prometheus-8249/log" grafana_servers: - host: 10.0.1.10 # port: 3000 # deploy_dir: /tidb-deploy/grafana-3000 alertmanager_servers: - host: 10.0.1.10 # ssh_port: 22 # web_port: 9093 # cluster_port: 9094 # deploy_dir: "/tidb-deploy/alertmanager-9093" # data_dir: "/tidb-data/alertmanager-9093" # log_dir: "/tidb-deploy/alertmanager-9093/log"# # Global variables are applied to all deployments and used as the default value of # # the deployments if a specific deployment value is missing. global: user: "tidb" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" # # Monitored variables are applied to all the machines. monitored: node_exporter_port: 9100 blackbox_exporter_port: 9115 # deploy_dir: "/tidb-deploy/monitored-9100" # data_dir: "/tidb-data/monitored-9100" # log_dir: "/tidb-deploy/monitored-9100/log" # # Server configs are used to specify the runtime configuration of TiDB components. # # All configuration items can be found in TiDB docs: # # - TiDB: https://docs.pingcap.com/zh/tidb/stable/tidb-configuration-file # # - TiKV: https://docs.pingcap.com/zh/tidb/stable/tikv-configuration-file # # - PD: https://docs.pingcap.com/zh/tidb/stable/pd-configuration-file # # All configuration items use points to represent the hierarchy, e.g: # # readpool.storage.use-unified-pool # # # # You can overwrite this configuration via the instance-level `config` field. server_configs: tidb: log.slow-threshold: 300 tikv: # server.grpc-concurrency: 4 # raftstore.apply-pool-size: 2 # raftstore.store-pool-size: 2 # rocksdb.max-sub-compactions: 1 # storage.block-cache.capacity: "16GB" # readpool.unified.max-thread-count: 12 readpool.storage.use-unified-pool: false readpool.coprocessor.use-unified-pool: true pd: schedule.leader-schedule-limit: 4 schedule.region-schedule-limit: 2048 schedule.replica-schedule-limit: 64 replication.enable-placement-rules: true tiflash: # Maximum memory usage for processing a single query. Zero means unlimited. profiles.default.max_memory_usage: 0 # Maximum memory usage for processing all concurrently running queries on the server. Zero means unlimited. profiles.default.max_memory_usage_for_all_queries: 0 tiflash-learner: # The allowable number of threads in the pool that flushes Raft data to storage. raftstore.apply-pool-size: 4 # The allowable number of threads that process Raft, which is the size of the Raftstore thread pool. raftstore.store-pool-size: 4 pd_servers: - host: 10.0.1.4 # ssh_port: 22 # name: "pd-1" # client_port: 2379 # peer_port: 2380 # deploy_dir: "/tidb-deploy/pd-2379" # data_dir: "/tidb-data/pd-2379" # log_dir: "/tidb-deploy/pd-2379/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.pd` values. # config: # schedule.max-merge-region-size: 20 # schedule.max-merge-region-keys: 200000 - host: 10.0.1.5 - host: 10.0.1.6 tidb_servers: - host: 10.0.1.7 # ssh_port: 22 # port: 4000 # status_port: 10080 # deploy_dir: "/tidb-deploy/tidb-4000" # log_dir: "/tidb-deploy/tidb-4000/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.tidb` values. # config: # log.slow-query-file: tidb-slow-overwrited.log - host: 10.0.1.8 - host: 10.0.1.9 tikv_servers: - host: 10.0.1.1 # ssh_port: 22 # port: 20160 # status_port: 20180 # deploy_dir: "/tidb-deploy/tikv-20160" # data_dir: "/tidb-data/tikv-20160" # log_dir: "/tidb-deploy/tikv-20160/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.tikv` values. # config: # server.grpc-concurrency: 4 # server.labels: # zone: "zone1" # dc: "dc1" # host: "host1" - host: 10.0.1.2 - host: 10.0.1.3 tiflash_servers: - host: 10.0.1.11 # ssh_port: 22 # tcp_port: 9000 # flash_service_port: 3930 # flash_proxy_port: 20170 # flash_proxy_status_port: 20292 # metrics_port: 8234 # deploy_dir: "/tidb-deploy/tiflash-9000" ## The `data_dir` will be overwritten if you define `storage.main.dir` configurations in the `config` section. # data_dir: "/tidb-data/tiflash-9000" # log_dir: "/tidb-deploy/tiflash-9000/log" # numa_node: "0,1" # # The following configs are used to overwrite the `server_configs.tiflash` values. # config: # logger.level: "info" # ## Multi-disk deployment introduced in v4.0.9 # ## Check https://docs.pingcap.com/tidb/stable/tiflash-configuration#multi-disk-deployment for more details. # ## Example1: # # storage.main.dir: [ "/nvme_ssd0_512/tiflash", "/nvme_ssd1_512/tiflash" ] # # storage.main.capacity: [ 536870912000, 536870912000 ] # ## Example2: # # storage.main.dir: [ "/sata_ssd0_512/tiflash", "/sata_ssd1_512/tiflash", "/sata_ssd2_512/tiflash" ] # # storage.latest.dir: [ "/nvme_ssd0_150/tiflash" ] # # storage.main.capacity: [ 536870912000, 536870912000, 536870912000 ] # # storage.latest.capacity: [ 161061273600 ] # learner_config: # log-level: "info" # server.labels: # zone: "zone2" # dc: "dc2" # host: "host2" # - host: 10.0.1.12 # - host: 10.0.1.13 monitoring_servers: - host: 10.0.1.10 # ssh_port: 22 # port: 9090 # deploy_dir: "/tidb-deploy/prometheus-8249" # data_dir: "/tidb-data/prometheus-8249" # log_dir: "/tidb-deploy/prometheus-8249/log" grafana_servers: - host: 10.0.1.10 # port: 3000 # deploy_dir: /tidb-deploy/grafana-3000 alertmanager_servers: - host: 10.0.1.10 # ssh_port: 22 # web_port: 9093 # cluster_port: 9094 # deploy_dir: "/tidb-deploy/alertmanager-9093" # data_dir: "/tidb-data/alertmanager-9093" # log_dir: "/tidb-deploy/alertmanager-9093/log"

以上 TiDB 集群拓扑文件中,详细的配置项说明见通过 TiUP 部署 TiDB 集群的拓扑文件配置

关键参数介绍

  • 需要将配置模板中 replication.enable-placement-rules 设置为 true,以开启 PD 的 Placement Rules 功能。

  • tiflash_servers 实例级别配置 "-host" 目前只支持 IP,不支持域名。

  • TiFlash 具体的参数配置介绍可参考 TiFlash 参数配置