URI Formats of External Storage Services
This document describes the URI formats of external storage services, including Amazon S3, GCS, and Azure Blob Storage.
The basic format of the URI is as follows:
[scheme]://[host]/[path]?[parameters]
Amazon S3 URI format
scheme
:s3
host
:bucket name
parameters
:access-key
: Specifies the access key.secret-access-key
: Specifies the secret access key.session-token
: Specifies the temporary session token. BR does not support this parameter yet.use-accelerate-endpoint
: Specifies whether to use the accelerate endpoint on Amazon S3 (defaults tofalse
).endpoint
: Specifies the URL of custom endpoint for S3-compatible services (for example,<https://s3.example.com/>
).force-path-style
: Use path style access rather than virtual hosted style access (defaults totrue
).storage-class
: Specifies the storage class of the uploaded objects (for example,STANDARD
orSTANDARD_IA
).sse
: Specifies the server-side encryption algorithm used to encrypt the uploaded objects (value options: `,
AES256, or
aws:kms`).sse-kms-key-id
: Specifies the KMS ID ifsse
is set toaws:kms
.acl
: Specifies the canned ACL of the uploaded objects (for example,private
orauthenticated-read
).role-arn
: When you need to access Amazon S3 data from a third party using a specified IAM role, you can specify the corresponding Amazon Resource Name (ARN) of the IAM role with therole-arn
URL query parameter, such asarn:aws:iam::888888888888:role/my-role
. For more information about using an IAM role to access Amazon S3 data from a third party, see AWS documentation.external-id
: When you access Amazon S3 data from a third party, you might need to specify a correct external ID to assume the IAM role. In this case, you can use thisexternal-id
URL query parameter to specify the external ID and make sure that you can assume the IAM role. An external ID is an arbitrary string provided by the third party together with the IAM role ARN to access the Amazon S3 data. Providing an external ID is optional when assuming an IAM role, which means if the third party does not require an external ID for the IAM role, you can assume the IAM role and access the corresponding Amazon S3 data without providing this parameter.
The following is an example of an Amazon S3 URI for TiDB Lightning and BR. In this example, you need to specify a specific file path testfolder
.
s3://external/testfolder?access-key=${access-key}&secret-access-key=${secret-access-key}
The following is an example of an Amazon S3 URI for TiCDC sink-uri
.
tiup cdc:v7.5.0 cli changefeed create \
--server=http://172.16.201.18:8300 \
--sink-uri="s3://cdc?endpoint=http://10.240.0.38:9000&access-key=${access-key}&secret-access-key=${secret-access-key}" \
--changefeed-id="cdcTest" \
--config=cdc_csv.toml
The following is an example of an Amazon S3 URI for IMPORT INTO
. In this example, you need to specify a specific filename test.csv
.
s3://external/test.csv?access-key=${access-key}&secret-access-key=${secret-access-key}
GCS URI format
scheme
:gcs
orgs
host
:bucket name
parameters
:credentials-file
: Specifies the path to the credentials JSON file on the migration tool node.storage-class
: Specifies the storage class of the uploaded objects (for example,STANDARD
orCOLDLINE
)predefined-acl
: Specifies the predefined ACL of the uploaded objects (for example,private
orproject-private
)
The following is an example of a GCS URI for TiDB Lightning and BR. In this example, you need to specify a specific file path testfolder
.
gcs://external/testfolder?credentials-file=${credentials-file-path}
The following is an example of a GCS URI for IMPORT INTO
. In this example, you need to specify a specific filename test.csv
.
gcs://external/test.csv?credentials-file=${credentials-file-path}
Azure Blob Storage URI format
scheme
:azure
orazblob
host
:container name
parameters
:account-name
: Specifies the account name of the storage.account-key
: Specifies the access key.sas-token
: Specifies the shared access signature (SAS) token.access-tier
: Specifies the access tier of the uploaded objects, for example,Hot
,Cool
, orArchive
. The default value is the default access tier of the storage account.encryption-scope
: Specifies the encryption scope for server-side encryption.encryption-key
: Specifies the encryption key for server-side encryption, which uses the AES256 encryption algorithm.
The following is an example of an Azure Blob Storage URI for BR. In this example, you need to specify a specific file path testfolder
.
azure://external/testfolder?account-name=${account-name}&account-key=${account-key}