tdtableexport>: Treasure Data table export to S3

NOTE: We're limiting export capability to only us-east region S3 bucket. In general, please use Result Output to S3 feature using td operator.

tdtableexport> operator exports data from Treasure Data to S3.

Copy
Copied
+step1:
  td_table_export>:
  database: mydb
  table: mytable
  file_format: jsonl.gz
  from: 2016-01-01 00:00:00 +0800
  to:   2016-02-01 00:00:00 +0800
  s3_bucket: my_backup_backet
  s3_path_prefix: mydb/mytable

Secrets

When you don't know how to set secrets, please refer to Managing Workflow Secret

  • td.apikey : API_KEY

    The Treasure Data API key to use when running Treasure Data table exports.

  • aws.s3.access_key_id : ACCESS KEY ID

    The AWS Access Key ID to use when writing to S3.

    Examples:

    Copy
    Copied
    aws.s3.access_key_id: ABCDEFGHJKLMNOPQRSTU
  • aws.s3.secret_access_key : SECRET ACCESS KEY

    The AWS Secret Access Key to use when writing to S3.

    Examples:

    Copy
    Copied
    aws.s3.secret_access_key: QUtJ/QUpJWTQ3UkhZTERNUExTUEEQUtJQUpJWTQ3

Options

  • database : NAME

    Name of the database.

    Examples:

    Copy
    Copied
    database: my_database
  • table : NAME

    Name of the table to export.

    Examples:

    Copy
    Copied
    table: my_table
  • file_format : TYPE

    Output file format. Available formats are tsv.gz (tab-separated values per line) and jsonl.gz (json record per line).

    json.gz and line-json.gz are available only for backward compatibility purpose.

    Examples:

    Copy
    Copied
    file_format: jsonl.gz
  • from : yyyy-MM-dd HH:mm:ss [ Z]

    Export records from this time (inclusive). Actual time range is :command:[from, to). Value should be a UNIX timestamp integer (seconds) or string in yyyy-MM-dd HH:mm:ss[ Z] format.

    Examples:

    Copy
    Copied
    from: 2016-01-01 00:00:00 +0800
  • to : yyyy-MM-dd HH:mm:ss [ Z]

    Export records to this time (exclusive). Actual time range is :command:[from, to). Value should be a UNIX timestamp integer (seconds) or string in yyyy-MM-dd HH:mm:ss[ Z] format.

    Examples:

    Copy
    Copied
    to: 2016-02-01 00:00:00 +0800
  • s3_bucket : NAME

    S3 bucket name to export records to.

    Examples:

    Copy
    Copied
    s3_bucket: my_backup_backet
  • s3_path_prefix : NAME

    S3 file name prefix.

    Examples:

    Copy
    Copied
    s3_path_prefix: mytable/mydb
  • endpoint : ADDRESS

    API endpoint (default: api.treasuredata.com).

  • use_ssl : BOOLEAN

    Enable SSL (https) to access to the endpoint (default: true).

Output parameters

  • td.last_job_id or td.last_job.id

    The job id this task executed.

    Examples:

    Copy
    Copied
    52036074
  • td.last_job.num_records

    The number of records of this job output.

    Examples:

    Copy
    Copied
    10