Compatibility of TiDB Lightning and IMPORT INTO with TiCDC and Log Backup
This document describes TiDB Lightning and IMPORT INTO
compatibility with log backup and TiCDC, as well as some special usage scenarios.
IMPORT INTO
vs. TiDB Lightning
IMPORT INTO
integrates with the physical import mode of TiDB Lightning, but there are some differences. See IMPORT INTO
vs. TiDB Lightning for details.
Compatibility with log backup and TiCDC
TiDB Lightning logical import mode is compatible with log backup and TiCDC.
TiDB Lightning physical import mode is not compatible with log backup and TiCDC. The reason is that physical import mode directly ingests encoded KV pairs of the source data to TiKV, causing TiKV not to generate corresponding change logs during this process. Without such change logs, the relevant data cannot be backed up via log backup and cannot be replicated by TiCDC.
IMPORT INTO
is not compatible with log backup and TiCDC. The reason is thatIMPORT INTO
also ingests encoded KV pairs of the source data directly to TiKV.
Scenarios for TiDB Lightning logical import mode
If TiDB Lightning logical import mode can meet your application's performance requirements and your application requires imported tables to be backed up or replicated downstream using TiCDC, it is recommended to use TiDB Lightning logical import mode.
Scenarios for TiDB Lightning physical import mode
This section describes how to use TiDB Lightning together with log backup and TiCDC.
If TiDB Lightning logical import mode does not meet your application's performance requirements, you have to use TiDB Lightning physical import mode, and imported tables need to be backed up or replicated downstream using TiCDC, then the following scenarios are recommended.
Used with log backup
You can perform in different scenarios as follows:
Scenario 1: tables in physical import mode do not need to be backed up
In this scenario, if PITR is enabled, the compatibility check will report an error after TiDB Lightning starts. If you are sure that these tables do not need backup or log backup, you can change the
Lightning.check-requirements
parameter in the TiDB Lightning configuration file tofalse
and restart the import task.Scenario 2: after the physical import is finished, there will be no new DML operations on the table
This scenario does not involve incremental data writes, so it is sufficient to perform a table-level snapshot backup of the table after completing the data import in TiDB Lightning physical import mode, as described in Back up a table.
During data recovery, the snapshot data of the table is restored. See Restore a table for the procedure.
Scenario 3: after the physical import is finished, there will be new DML operations on the table (not supported)
In this scenario, you can only choose either full snapshot backup or log backup for this table. You cannot back up and restore both the full snapshot data and log backup data of this table.
Used with TiCDC
Using TiCDC with physical import mode is not compatible in the short term, because TiCDC cannot keep up with the write speed of TiDB Lightning physical import mode, which might result in increasing cluster replication latency.
You can perform in different scenarios as follows:
Scenario 1: the table does not need to be replicated downstream by TiCDC.
In this scenario, if TiCDC changefeed is enabled, the compatibility check will report an error after TiDB Lightning starts. If you are sure that these tables do not need backup or log backup, you can change the
Lightning.check-requirements
parameter in the TiDB Lightning configuration file tofalse
and restart the import task.Scenario 2: the table needs to be replicated downstream by TiCDC.
In this scenario, if TiCDC changefeed is enabled, the compatibility check will report an error after TiDB Lightning starts. You need to change the
Lightning.check-requirements
parameter in the TiDB Lightning configuration file in the upstream TiDB cluster tofalse
and restart the import task.After the import task for the upstream TiDB cluster is finished, use TiDB Lightning to import the same data in the downstream TiDB cluster. If you have databases such as Redshift and Snowflake in the downstream, you can configure them to read CSV, SQL, or Parquet files from a cloud storage service and then write the data to the database.
Scenarios for IMPORT INTO
This section describes how to use IMPORT INTO
together with log backup and TiCDC.
Used with log backup
You can perform in different scenarios as follows:
Scenario 1: tables do not need to be backed up
In this scenario, if PITR is enabled, the compatibility check will report an error after you submit the
IMPORT INTO
statement. If you are sure that these tables do not need backup or log backup, you can includeDISABLE_PRECHECK
(introduced in v8.0.0) inWithOptions
of that statement, and then resubmit it. In this way, the data import task ignores the compatibility check and imports the data directly.Scenario 2: after the import is finished, there will be no new DML operations on the table
This scenario does not involve incremental data writes, so it is sufficient to perform a table-level snapshot backup of the table after completing the data import, as described in Back up a table.
During data recovery, the snapshot data of the table is restored. See Restore a table for the procedure.
Scenario 3: after the import is finished, there will be new DML operations on the table (not supported)
In this scenario, you can only choose either full snapshot backup or log backup for this table. You cannot back up and restore both the full snapshot data and log backup data of this table.
Used with TiCDC
You can perform in different scenarios as follows:
Scenario 1: the table does not need to be replicated downstream by TiCDC.
In this scenario, if TiCDC changefeed is enabled, the compatibility check will report an error after you submit the
IMPORT INTO
statement. If you are sure that these tables do not need to be replicated by TiCDC, you can includeDISABLE_PRECHECK
(introduced in v8.0.0) inWithOptions
of that statement, and then resubmit it. In this way, the data import task ignores the compatibility check and imports the data directly.Scenario 2: the table needs to be replicated downstream by TiCDC.
In this scenario, if TiCDC changefeed is enabled, the compatibility check will report an error after you submit the
IMPORT INTO
statement. You can includeDISABLE_PRECHECK
(introduced in v8.0.0) inWithOptions
of that statement, and then resubmit it. In this way, the data import task ignores the compatibility check and imports the data directly.After the import task for the upstream TiDB cluster is finished, use
IMPORT INTO
to import the same data in the downstream TiDB cluster. If you have databases such as Redshift and Snowflake in the downstream, you can configure them to read CSV, SQL, or Parquet files from a cloud storage service and then write the data to the database.