Manage Data Source Configurations in TiDB Data Migration

This document introduces how to manage data source configurations, including encrypting the MySQL password, operating the data source, and changing the bindings between upstream MySQL instances and DM-workers using dmctl.

Encrypt the database password

In DM configuration files, it is recommended to use the password encrypted with dmctl. For one original password, the encrypted password is different after each encryption.

./dmctl encrypt 'abc!@#123'

MKxn0Qo3m3XOyjCnhEMtsUCm83EhGQDZ/T4=

Operate data source

You can use the operate-source command to load, list or remove the data source configurations to the DM cluster.

help operate-source

`create`/`stop`/`show` upstream MySQL/MariaDB source.

Usage:
  dmctl operate-source <operate-type> [config-file ...] [--print-sample-config] [flags]

Flags:
  -h, --help                  help for operate-source
  -p, --print-sample-config   print sample config file of source

Global Flags:
  -s, --source strings   MySQL Source ID

Flags description

• create: Creates one or more upstream database sources. When creating multiple data sources fails, DM rolls back to the state where the command was not executed.

• stop: Stops one or more upstream database sources. When stopping multiple data sources fails, some data sources might be stopped.

• show: Shows the added data source and the corresponding DM-worker.

• config-file: Specifies the file path of source.yaml and can pass multiple file paths.

• --print-sample-config: Prints the sample configuration file. This parameter ignores other parameters.

Usage example

Use the following operate-source command to create a source configuration file:

operate-source create ./source.yaml

For the configuration of source.yaml, refer to Upstream Database Configuration File Introduction.

The following is an example of the returned result:

{
    "result": true,
    "msg": "",
    "sources": [
        {
            "result": true,
            "msg": "",
            "source": "mysql-replica-01",
            "worker": "dm-worker-1"
        }
    ]
}

Check data source configurations

If you know the source-id, you can run dmctl --master-addr <master-addr> config source <source-id> to get the data source configuration.

config source mysql-replica-01

{
  "result": true,
    "msg": "",
    "cfg": "enable-gtid: false
      flavor: mysql
      source-id: mysql-replica-01
      from:
        host: 127.0.0.1
        port: 8407
        user: root
        password: '******'
}

If you don't know the source-id, you can run dmctl --master-addr <master-addr> operate-source show to list all data sources first.

operate-source show

{
    "result": true,
    "msg": "",
    "sources": [
        {
            "result": true,
            "msg": "source is added but there is no free worker to bound",
            "source": "mysql-replica-02",
            "worker": ""
        },
        {
            "result": true,
            "msg": "",
            "source": "mysql-replica-01",
            "worker": "dm-worker-1"
        }
    ]
}

Change the bindings between upstream MySQL instances and DM-workers

You can use the transfer-source command to change the bindings between upstream MySQL instances and DM-workers.

help transfer-source

Transfers an upstream MySQL/MariaDB source to a free worker.
Usage:
  dmctl transfer-source <source-id> <worker-id> [flags]
Flags:
  -h, --help   help for transfer-source
Global Flags:
  -s, --source strings   MySQL Source ID.

Before transferring, DM checks whether the worker to be unbound still has running tasks. If the worker has any running tasks, you need to pause the tasks first, change the binding, and then resume the tasks.

Usage example

If you do not know the bindings of DM-workers, you can run dmctl --master-addr <master-addr> list-member --worker to list the current bindings of all workers.

list-member --worker

{
    "result": true,
    "msg": "",
    "members": [
        {
            "worker": {
                "msg": "",
                "workers": [
                    {
                        "name": "dm-worker-1",
                        "addr": "127.0.0.1:8262",
                        "stage": "bound",
                        "source": "mysql-replica-01"
                    },
                    {
                        "name": "dm-worker-2",
                        "addr": "127.0.0.1:8263",
                        "stage": "free",
                        "source": ""
                    }
                ]
            }
        }
    ]
}

In the above example, mysql-replica-01 is bound to dm-worker-1. The below command transfers the binding worker of mysql-replica-01 to dm-worker-2.

transfer-source mysql-replica-01 dm-worker-2

{
    "result": true,
    "msg": ""
}

Check whether the command takes effect by running dmctl --master-addr <master-addr> list-member --worker.

list-member --worker

{
    "result": true,
    "msg": "",
    "members": [
        {
            "worker": {
                "msg": "",
                "workers": [
                    {
                        "name": "dm-worker-1",
                        "addr": "127.0.0.1:8262",
                        "stage": "free",
                        "source": ""
                    },
                    {
                        "name": "dm-worker-2",
                        "addr": "127.0.0.1:8263",
                        "stage": "bound",
                        "source": "mysql-replica-01"
                    }
                ]
            }
        }
    ]
}