Hybrid Deployment Topology

This document describes the topology and key parameters of the TiKV and TiDB hybrid deployment.

The hybrid deployment is usually used in the following scenario:

The deployment machine has multiple CPU processors with sufficient memory. To improve the utilization rate of the physical machine resources, multiple instances can be deployed on a single machine, that is, TiDB and TiKV's CPU resources are isolated through NUMA node bindings. PD and Prometheus are deployed together, but their data directories need to use separate file systems.

Topology information

Topology templates

• The simple template for the hybrid deployment
• The complex template for the hybrid deployment

For detailed descriptions of the configuration items in the above TiDB cluster topology file, see Topology Configuration File for Deploying TiDB Using TiUP.

Key parameters

This section introduces the key parameters when you deploy multiple instances on a single machine, which is mainly used in scenarios when multiple instances of TiDB and TiKV are deployed on a single machine. You need to fill in the results into the configuration template according to the calculation methods provided below.

• Optimize the configuration of TiKV

  • To configure readpool to be self-adaptive to the thread pool. By configuring the readpool.unified.max-thread-count parameter, you can make readpool.storage and readpool.coprocessor share a unified thread pool, and set the self-adaptive switch respectively.

    • Enable readpool.storage and readpool.coprocessor:

      readpool.storage.use-unified-pool: true
      readpool.coprocessor.use-unified-pool: true
      

    • The calculation method:

      readpool.unified.max-thread-count = cores * 0.8 / the number of TiKV instances
      

  • To configure the storage CF (all RocksDB column families) to be self-adaptive to memory. By configuring the storage.block-cache.capacity parameter, you can make CF automatically balance the memory usage.

    • storage.block-cache enables the CF self-adaptation by default. You do not need to modify it.

      storage.block-cache.shared: true
      

    • The calculation method:

      storage.block-cache.capacity = (MEM_TOTAL * 0.5 / the number of TiKV instances)
      

  • If multiple TiKV instances are deployed on the same physical disk, add the capacity parameter in the TiKV configuration:

    raftstore.capacity = disk total capacity / the number of TiKV instances
    

• The label scheduling configuration

  Since multiple instances of TiKV are deployed on a single machine, if the physical machines go down, the Raft Group might lose two of the default three replicas, which causes the cluster unavailability. To address this issue, you can use the label to enable the smart scheduling of PD, which ensures that the Raft Group has more than two replicas in multiple TiKV instances on the same machine.

  • The TiKV configuration

    The same host-level label information is configured for the same physical machine:

    config:
      server.labels:
        host: tikv1
    

  • The PD configuration

    To enable PD to identify and scheduling Regions, configure the labels type for PD:

    pd:
      replication.location-labels: ["host"]
    

• numa_node core binding

  • In the instance parameter module, configure the corresponding numa_node parameter and add the number of CPU cores.

  • Before using NUMA to bind cores, make sure that the numactl tool is installed, and confirm the information of CPUs in the physical machines. After that, configure the parameters.

  • The numa_node parameter corresponds to the numactl --membind configuration.