Upgrade TiDB Data Migration Between 1.0.x Versions
This document introduces how to upgrade a TiDB DM cluster from a lower 1.0.x version to a higher 1.0.x version, and the major changes and other information about the upgraded version.
Upgrade to v1.0.5
Version information
Release Version: v1.0.5
Git Commit Hash: a8e9f53f91e29756b09a22cdc37a6a6efcdfe55b
Git Branch: release-1.0
UTC Build Time: 2020-04-27 06:56:31
Go Version: go version go1.13 linux/amd64
Main changes
- Improve the incremental replication speed when the
UNIQUE KEY
column has theNULL
value - Add retry for the
Write conflict
(9007 and 8005) error returned by TiDB - Fix the issue that the
Duplicate entry
error might occur during the full data import - Fix the issue that the
stop-task
/pause-task
command may not work when no data written upstream after the full import is completed - Fix the issue that the monitoring metrics still display data after the migration task is stopped
Upgrade operation example
- Download the new version of DM-Ansible, and confirm that there is
dm_version = v1.0.5
in theinventory.ini
file. - Run
ansible-playbook local_prepare.yml
to download the new DM binary file to the local disk. - Run
ansible-playbook rolling_update.yml
to perform a rolling update for the DM cluster components. - Run
ansible-playbook rolling_update_monitor.yml
to perform a rolling update for the DM monitoring components.
Upgrade to v1.0.4
Version information
Release Version: v1.0.4-1-gd681c67
Git Commit Hash: d681c6731d3432f4d8f38ea651f44d49d6860269
Git Branch: release-1.0
UTC Build Time: 2020-03-16 09:45:29
Go Version: go version go1.13 linux/amd64
Main changes
- Add English UI for DM Portal
- Add the
--more
parameter in thequery-status
command to show complete migration status information - Fix the issue that
resume-task
might fail to resume the migration task which is interrupted by the abnormal connection to the downstream TiDB server - Fix the issue that the online DDL operation cannot be properly migrated after a failed migration task is restarted because the online DDL meta information has been cleared after the DDL operation failure
- Fix the issue that
query-error
might cause the DM-worker to panic afterstart-task
goes into error - Fix the issue that the relay log file and
relay.meta
cannot be correctly recovered when restarting an abnormally stopped DM-worker process beforerelay.meta
is successfully written
Upgrade operation example
- Download the new version of DM-Ansible, and confirm that there is
dm_version = v1.0.4
in theinventory.ini
file. - Run
ansible-playbook local_prepare.yml
to download the new DM binary file to the local disk. - Run
ansible-playbook rolling_update.yml
to perform a rolling update for the DM cluster components. - Run
ansible-playbook rolling_update_monitor.yml
to perform a rolling update for the DM monitoring components.
Upgrade to v1.0.3
Version information
Release Version: v1.0.3
Git Commit Hash: 41426af6cffcff9a325697a3bdebeadc9baa8aa6
Git Branch: release-1.0
UTC Build Time: 2019-12-13 07:04:53
Go Version: go version go1.13 linux/amd64
Main changes
- Add the command mode in dmctl
- Support migrating the
ALTER DATABASE
DDL statement - Optimize the error message output
- Fix the panic-causing data race issue occurred when the full import unit pauses or exits
- Fix the issue that
stop-task
andpause-task
might not take effect when retrying SQL operations to the downstream
Upgrade operation example
- Download the new version of DM-Ansible, and confirm that there is
dm_version = v1.0.3
in theinventory.ini
file. - Run
ansible-playbook local_prepare.yml
to download the new DM binary file to the local disk. - Run
ansible-playbook rolling_update.yml
to perform a rolling update for the DM cluster components. - Run
ansible-playbook rolling_update_monitor.yml
to perform a rolling update for the DM monitoring components.
Upgrade to v1.0.2
Version information
Release Version: v1.0.2
Git Commit Hash: affc6546c0d9810b0630e85502d60ed5c800bf25
Git Branch: release-1.0
UTC Build Time: 2019-10-30 05:08:50
Go Version: go version go1.12 linux/amd64
Main changes
- Support automatically generating some configuration items for DM-worker to reduce manual configuration cost
- Support automatically generating the parameters of Mydumper database and tables to reduce manual configuration cost
- Optimize the default output of
query-status
to highlight important information - Directly manage the DB connection to the downstream instead of using the built-in connection pool to optimize the handling of and retry for SQL errors
- Fix the panic that might occur when the DM-worker process is started or when the DML statement is failed to execute
- Fix the bug that the timeout of executing the sharding DDL statements (for example,
ADD INDEX
) might cause that the subsequent sharding DDL statements cannot be correctly coordinated - Fix the bug that the
start-task
command cannot be executed when some DM-workers are inaccessible - Improve the automatic retry policy for the
1105
error
Upgrade operation example
- Download the new version of DM-Ansible, and confirm that there is
dm_version = v1.0.2
in theinventory.ini
file. - Run
ansible-playbook local_prepare.yml
to download the new DM binary file to the local disk. - Run
ansible-playbook rolling_update.yml
to perform a rolling update for the DM cluster components. - Run
ansible-playbook rolling_update_monitor.yml
to perform a rolling update for the DM monitoring components.
Upgrade to v1.0.1
Version information
Release Version: v1.0.1
Git Commit Hash: e63c6cdebea0edcf2ef8c91d84cff4aaa5fc2df7
Git Branch: release-1.0
UTC Build Time: 2019-09-10 06:15:05
Go Version: go version go1.12 linux/amd64
Main changes
- Fix the issue that DM frequently re-establishes the database connection in some situations
- Fix the panic that might occur when using the
query-status
command
Upgrade operation example
- Download the new version of DM-Ansible, and confirm that there is
dm_version = v1.0.1
in theinventory.ini
file. - Run
ansible-playbook local_prepare.yml
to download the new DM binary file to the local disk. - Run
ansible-playbook rolling_update.yml
to perform a rolling update for the DM cluster components. - Run
ansible-playbook rolling_update_monitor.yml
to perform a rolling update for the DM monitoring components.
Upgrade to v1.0.0-10-geb2889c9 (1.0 GA)
Version information
Release Version: v1.0.0-10-geb2889c9
Git Commit Hash: eb2889c9dcfbff6653be9c8720a32998b4627db9
Git Branch: release-1.0
UTC Build Time: 2019-09-06 03:18:48
Go Version: go version go1.12 linux/amd64
Main changes
- Support automatically recovering migration tasks for some abnormal situations
- Improve compatibility with DDL syntaxes
- Fix the bug that the abnormal connection to the upstream database might cause data loss
Upgrade operation example
- Download the new version of DM-Ansible, and confirm that there is
dm_version = v1.0.0
in theinventory.ini
file. - Run
ansible-playbook local_prepare.yml
to download the new DM binary file to the local disk. - Run
ansible-playbook rolling_update.yml
to perform a rolling update for the DM cluster components. - Run
ansible-playbook rolling_update_monitor.yml
to perform a rolling update for the DM monitoring components.
Upgrade to v1.0.0-rc.1-12-gaa39ff9
Version information
Release Version: v1.0.0-rc.1-12-gaa39ff9
Git Commit Hash: aa39ff981dfb3e8c0fa4180127246b253604cc34
Git Branch: dm-master
UTC Build Time: 2019-07-24 02:26:08
Go Version: go version go1.11.2 linux/amd64
Main changes
Starting from this release, DM checks all configurations strictly. Unrecognized configuration triggers an error. This is to ensure that users always know exactly what the configuration is.
Upgrade notes
Before starting the DM-master or DM-worker, ensure that the obsolete configuration information has been deleted and there are no redundant configuration items.
Otherwise, the starting might fail. In this situation, you can delete the deprecated configuration based on the failure information. These are two possible deprecated configurations:
meta-file
indm-worker.toml
server-id
inmysql-instances
intask.yaml