Treasure Data TD-API Quickstart

The TD-API makes it easy to integrate with other tools to better manipulate your data. Through the TD API you can programatically access the functionality typically provided through the web based TD Console.

This functionality includes:

  • Manage bulk imports of data
  • List databases and tables
  • Schedule jobs
  • Manage connectors
  • Manage users
  • Access control policies

By leveraging the TD-API endpoints you can create an external application, tool, or CLI to automate common tasks. For example, you may want to integrate TD-API with your continous deployment service to run tests or setup complex automations.

Setup

Before you can use a Treasure Data REST API, you need two things:

  • An endpoint or baseURL for your API call
  • An API key to authenticate your API call

Determining Endpoints or BaseURLs

Each regional Treasure Data server has specific baseURLs that correspond to it. Here is a list of regional servers and baseURLs for TD APIs.

Getting Your API Keys

You can get your API keys from the My Settings section of the TD Console.

Choosing the Right Type of API Key

There are two types of API keys.

API Key Type Description
master Master API keys can be used to perform all permitted operations that correspond to the user’s permissions in TD Console.
write-only Based on the permissions and access levels associated with a user, write-only API keys only allow importing data into Treasure Data to those databases the user has write access to. This provides an additional layer of security in controlling access to a Treasure Data account through the REST APIs.
Write-only API keys are useful when access has to be provided to 3rd parties or API keys need to be embedded in ingestion libraries (for example, with web page integrations).
warning

Keep your API Master key secure. Master keys grant both read and write access to Treasure Data, and anyone in possession of your Master key could possibly access, corrupt, or delete your data. Do not share your Master keys and be careful not to expose or release your Master key publicly.

Click here to see how to get a write-only API Key.

  1. Login to the Treasure Data Console and go to the API Key page .
  2. If you dont already have a write-only API key, then create one. From the top right, select Actions > Create API Key .
  3. Name the new API Key.
  4. From the Type drop-down menu, select Write-only .
  5. Select Save .
  6. Copy your new Write-Only API Key and use it authenticate the APIs in your project.

Basic Use

List Databases

In this case, you want to return a list of databases. This example issues a call to the API to return a list of your databases.

Sample Request

Copy
Copied
curl -H "AUTHORIZATION: TD1 YOUR_API_KEY_HERE" \ 
          "https://api.treasuredata.com/v3/database/list"

Sample Response

Copy
Copied
{
  "databases": [
    {
      "name": "database1",
      "count": 5000,
      "created_at": "2013-11-01 16:48:41 -0700",
      "updated_at": "2013-11-01 16:48:41 -0700",
      "organization": null
    },
    {
      "name": "database2",
      "count": 5000,
      "created_at": "2013-11-08 17:47:22 -0800",
      "updated_at": "2013-11-08 17:47:22 -0800",
      "organization": null
    }
  ]
}

Issue a Query

You might want to run a query to count how many visitors have come to your website.

URI: https://api.treasuredata.com/v3/job/issue/presto/td_audit_log

Sample Request

Copy
Copied
curl --location --request POST          
     'https://api.treasuredata.com/v3/job/issue/presto/td_audit_log' \
     --header 'AUTHORIZATION: TD1 YOUR_API_KEY_HERE' \
     --header 'Content-Type: application/json' \
     --data-raw '{
       "query": "select * from access limit 5",
       "domain_key": 261,
       "Priority":0
}

Sample Response

Copy
Copied
{
  "job": "1120145314",
  "job_id": "1120145314",
  "database": "td_audit_log"
}

Get Query Results

You might want to get query results for a specific job.

Sample Request

Copy
Copied
curl --location --request GET 'https://api.treasuredata.com/v3/job/result/1145768692?format=json' \
    --header 'AUTHORIZATION: TD1 YOUR_API_KEY_HERE' \
    --header 'Content-Type: application/json' \

Sample Response

Copy
Copied
[1,"400","num_requests",1601378369]
[2,"404","http_status_code",1601378369]
[3,"200","num_requests",1601378369]

List Connectors

You might need to retrieve a list of connectors or to check if one exists or see if one has been added.

Sample Request

Copy
Copied
curl -I -X GET \
  https://api.treasuredata.com/v3/result/list \
  -H 'Authorization: TD1 YOUR_API_KEY_HERE'

Sample Response

Copy
Copied
{
    "name": "mb_s3_sep_6",
    "url": "s3://test:xxx@s3-us-west-1.amazonaws.com/",
    "organization": null
}

Further Reading

Now that you have the basics down, here are some suggested resources for you.

  • Tutorial - Use a continuous integration / continuous deployment service (CI / CD) tools such as Travis CI or Circle CI with TD-API
  • Treasure Toolbelt - Command Line Interface for Treasure Data services.

Treasure Data Clients - native language wrappers for the TD-API REST API.