Get Started with TiDB Lightning
This tutorial assumes you use several new and clean CentOS 7 instances. You can use VMware, VirtualBox or other tools to deploy a virtual machine locally or a small cloud virtual machine on a vendor-supplied platform. Because TiDB Lightning consumes a large amount of computer resources, it is recommended that you allocate at least 16 GB memory and CPU of 32 cores for running it with the best performance.
Prepare full backup data
First, use dumpling
to export data from MySQL:
tiup dumpling -h 127.0.0.1 -P 3306 -u root -t 16 -F 256MB -B test -f 'test.t[12]' -o /data/my_database/
In the above command:
-B test
: means the data is exported from thetest
database.-f test.t[12]
: means only thetest.t1
andtest.t2
tables are exported.-t 16
: means 16 threads are used to export the data.-F 256MB
: means a table is partitioned into chunks and one chunk is 256 MB.
After executing this command, the full backup data is exported to the /data/my_database
directory.
Deploy TiDB Lightning
Step 1: Deploy a TiDB cluster
Before the data import, you need to deploy a TiDB cluster. In this tutorial, TiDB v5.4.0 is used as an example. For the deployment method, refer to Deploy a TiDB Cluster Using TiUP.
Step 2: Download TiDB Lightning installation package
The TiDB Lightning installation package is included in the TiDB Toolkit. To download the TiDB Toolkit, see Download TiDB Tools.
Step 3: Start tidb-lightning
Upload
bin/tidb-lightning
andbin/tidb-lightning-ctl
in the package to the server where TiDB Lightning is deployed.Upload the prepared data source to the server.
Configure
tidb-lightning.toml
as follows:[lightning] # Logging level = "info" file = "tidb-lightning.log" [tikv-importer] # Configure the import mode backend = "local" # Sets the directory for temporarily storing the sorted key-value pairs. # The target directory must be empty. sorted-kv-dir = "/mnt/ssd/sorted-kv-dir" [mydumper] # Local source data directory data-source-dir = "/data/my_datasource/" # Configures the wildcard rule. By default, all tables in the mysql, sys, INFORMATION_SCHEMA, PERFORMANCE_SCHEMA, METRICS_SCHEMA, and INSPECTION_SCHEMA system databases are filtered. # If this item is not configured, the "cannot find schema" error occurs when system tables are imported. filter = ['*.*', '!mysql.*', '!sys.*', '!INFORMATION_SCHEMA.*', '!PERFORMANCE_SCHEMA.*', '!METRICS_SCHEMA.*', '!INSPECTION_SCHEMA.*'] [tidb] # Information of the target cluster host = "172.16.31.2" port = 4000 user = "root" password = "rootroot" # Table schema information is fetched from TiDB via this status-port. status-port = 10080 # The PD address of the cluster pd-addr = "172.16.31.3:2379"After configuring the parameters properly, use a
nohup
command to start thetidb-lightning
process. If you directly run the command in the command-line, the process might exit because of the SIGHUP signal received. Instead, it's preferable to run a bash script that contains thenohup
command:#!/bin/bash nohup tiup tidb-lightning -config tidb-lightning.toml > nohup.out &
Step 4: Check data integrity
After the import is completed, TiDB Lightning exits automatically. If the import is successful, you can find tidb lightning exit
in the last line of the log file.
If any error occurs, refer to TiDB Lightning FAQs.
Summary
This tutorial briefly introduces what TiDB Lightning is and how to quickly deploy a TiDB Lightning cluster to import full backup data to the TiDB cluster.
For detailed features and usage about TiDB Lightning, refer to TiDB Lightning Overview.