TiDB Lightning Monitoring

tidb-lightning supports metrics collection via Prometheus. This document introduces the monitor configuration and monitoring metrics of TiDB Lightning.

Monitor configuration

If TiDB Lightning is manually installed, follow the instructions below.

The metrics of tidb-lightning can be gathered directly by Prometheus as long as it is discovered. You can set the metrics port in tidb-lightning.toml:

[lightning]
# HTTP port for debugging and Prometheus metrics pulling (0 to disable)
pprof-port = 8289

...

You need to configure Prometheus to make it discover the servers. For instance, you can directly add the server address to the scrape_configs section:

...
scrape_configs:
  - job_name: 'tidb-lightning'
    static_configs:
      - targets: ['192.168.20.10:8289']

Grafana dashboard

Grafana is a web interface to visualize Prometheus metrics as dashboards.

Row 1: Speed

Sometimes the import speed will drop to zero allowing other parts to catch up. This is normal.

Row 2: Progress

Row 3: Resource

Row 4: Quota

Row 5: Read speed

If any of the duration is too high, it indicates that the disk used by TiDB Lightning is too slow or busy with I/O.

Row 6: Storage

Row 7: Import speed

Monitoring metrics

This section explains the monitoring metrics of tidb-lightning.

Metrics provided by tidb-lightning are listed under the namespace lightning_*.

• lightning_importer_engine (Counter)

  Counts open and closed engine files. Labels:

  • type:
    • open
    • closed

• lightning_idle_workers (Gauge)

  Counts idle workers. Labels:

  • name:
    • table: the remainder of table-concurrency, normally 0 until the end of the process
    • index: the remainder of index-concurrency, normally 0 until the end of the process
    • region: the remainder of region-concurrency, normally 0 until the end of the process
    • io: the remainder of io-concurrency, normally close to configured value (default 5), and close to 0 means the disk is too slow
    • closed-engine: number of engines which have been closed but not yet cleaned up, normally close to index + table-concurrency (default 8). A value close to 0 means TiDB Lightning is faster than TiKV Importer, which might cause TiDB Lightning to stall

• lightning_kv_encoder (Counter)

  Counts open and closed KV encoders. KV encoders are in-memory TiDB instances that convert SQL INSERT statements into KV pairs. The net values need to be bounded in a healthy situation. Labels:

  • type:
    • open
    • closed

• lightning_tables (Counter)

  Counts processed tables and their statuses. Labels:

  • state: the status of the table, indicating which phase should be completed
    • pending: not yet processed
    • written: all data encoded and sent
    • closed: all corresponding engine files closed
    • imported: all engine files have been imported into the target cluster
    • altered_auto_inc: AUTO_INCREMENT ID altered
    • checksum: checksum performed
    • analyzed: statistics analysis performed
    • completed: the table has been fully imported and verified
  • result: the result of the current phase
    • success: the phase completed successfully
    • failure: the phase failed (did not complete)

• lightning_engines (Counter)

  Counts number of engine files processed and their status. Labels:

  • state: the status of the engine, indicating which phase should be completed
    • pending: not yet processed
    • written: all data encoded and sent
    • closed: engine file closed
    • imported: the engine file has been imported into the target cluster
    • completed: the engine has been fully imported
  • result: the result of the current phase
    • success: the phase completed successfully
    • failure: the phase failed (did not complete)

• lightning_chunks (Counter)

  Counts number of chunks processed and their status. Labels:

  • state: a chunk's status, indicating which phase the chunk is in
    • estimated: (not a state) this value gives total number of chunks in current task
    • pending: loaded but not yet processed
    • running: data are being encoded and sent
    • finished: the entire chunk has been processed
    • failed: errors happened during processing

• lightning_import_seconds (Histogram)

  Bucketed histogram for the time needed to import a table.

• lightning_row_read_bytes (Histogram)

  Bucketed histogram for the size of a single SQL row.

• lightning_row_encode_seconds (Histogram)

  Bucketed histogram for the time needed to encode a single SQL row into KV pairs.

• lightning_row_kv_deliver_seconds (Histogram)

  Bucketed histogram for the time needed to deliver a set of KV pairs corresponding to one single SQL row.

• lightning_block_deliver_seconds (Histogram)

  Bucketed histogram for the time needed to deliver a block of KV pairs to Importer.

• lightning_block_deliver_bytes (Histogram)

  Bucketed histogram for the uncompressed size of a block of KV pairs delivered to Importer.

• lightning_chunk_parser_read_block_seconds (Histogram)

  Bucketed histogram for the time needed by the data file parser to read a block.

• lightning_checksum_seconds (Histogram)

  Bucketed histogram for the time needed to compute the checksum of a table.

• lightning_apply_worker_seconds (Histogram)

  Bucketed histogram for the time needed to acquire an idle worker (see also the lightning_idle_workers gauge). Labels:

  • name:
    • table
    • index
    • region
    • io
    • closed-engine