Skip to content
/
API Reference

First class Data Warehouse Integration API (1.0.0)

Download OpenAPI description
dwh_integration_api-public.json
dwh_integration_api-public.yaml
Languages
Servers
Production AWS server
https://dwh-integration-api.treasuredata.com/
Production EU01 server
https://dwh-integration-api.eu01.treasuredata.com/
Production AP02 server
https://dwh-integration-api.ap02.treasuredata.com/
Production AP03 server
https://dwh-integration-api.ap03.treasuredata.com/
Production AWS Tokyo server
https://dwh-integration-api.treasuredata.co.jp/

List all DWH Integrations for the current user.

Request

Security
apiKey
curl -i -X GET \
  https://dwh-integration-api.treasuredata.com/v1/dwh/ \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

DWH Integrations

Bodyapplication/json
itemsArray of objects(DwhIntegration)
paginationobject(Pagination)
Response
application/json
{
  "items": [
    { … }
  ],
  "pagination": {
    "page": 0,
    "size": 0,
    "total_pages": 0
  }
}

Create a new DWH Integration

Request

Note: the DWH Integration will not automatically create the TD database and tables. You need to create the TD database and tables before creating the DWH Integration.

Security
apiKey
Bodyapplication/jsonrequired
schedule_typestring
Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"
schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

namestringrequired
basic_configsobject(BasicConfigs)required

The basic configs of the DWH Integration.

basic_configs.​typestringrequired

Connector type

Discriminator
basic_configs.​td_authentication_idstringrequired

The existing Connector Authentication ID

basic_configs.​rolestring(Role Name)

Specify the role name, Leave blank to use default role.

basic_configs.​warehousestring(Snowflake Warehouse)required

The Snowflake data Warehouse to use for the connection, e.g. "COMPUTE_WH"

basic_configs.​dbstring(Snowflake Database)required
basic_configs.​schemastring(Schema, e.g. "PUBLIC")required

The schema to use for the connection

basic_configs.​incrementalboolean(Incremental Loading?)

Enable incremental loading for this connector

Default false
advanced_configsArray of objects(AdvancedConfigs)required

The DWH Integration advanced configs. The configs contain array of of table configs which combine with the basic configs will be used to create Connector Bulkload Session.

advanced_configs[].​target_tablestring

The name of the TD table to import data into.

advanced_configs[].​configobject(Advanced Config)

The configuration which will merge with the basic_configs to form a complete Connector configuration for Bulkload job. Some of the config from the basic_configs can declare here to override the basic_configs.

output_configobject(OutConfigs)required

Config for how data will be imported into TD.

output_config.​databasestring

The name of the TD database to import data into. The database must be created before the DWH Integration is created.

output_config.​modestring

Specify the import mode. The default value is append.

output_config.​time_columnstring

The value of the column will be used as the TD timestamp column.

filter_configsArray of objects(FilterConfigs)

Config for how data will be filtered before importing into TD. See the connector filters for more info.

curl -i -X POST \
  https://dwh-integration-api.treasuredata.com/v1/dwh/ \
  -H 'Authorization: YOUR_API_KEY_HERE' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "test",
    "schedule_type": "daily",
    "schedule_option": "10:00",
    "timezone": "Asia/Tokyo",
    "basic_configs": {
      "td_authentication_id": 11,
      "database": "database",
      "table": "table",
      "schema": "test"
    },
    "advanced_configs": [
      {
        "target_table": "test",
        "config": {
          "select": "1",
          "table": "test",
          "source_type": "presto"
        }
      }
    ],
    "output_config": {
      "type": "td",
      "database": "test",
      "table": "test",
      "schema": "test"
    }
  }'

Responses

DWH Integration Created

Bodyapplication/json
schedule_typestring
Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"
schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

idinteger(int64)
namestring

The name of the DWH Integration.

user_idinteger(int64)

The ID of the user who created the DWH Integration.

account_idinteger(int64)

The TD Account which the DWH Integration belongs to.

workflow_project_idinteger(int64)
workflow_project_namestring
dwh_config_idinteger(int64)
created_atstring(date-time)
updated_atstring(date-time)
Response
application/json
{
  "schedule_type": "none",
  "schedule_option": "string",
  "timezone": "string",
  "id": 0,
  "name": "string",
  "user_id": 0,
  "account_id": 0,
  "workflow_project_id": 0,
  "workflow_project_name": "string",
  "dwh_config_id": 0,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z"
}

Get a DWH Integration.

Request

Security
apiKey
Path
dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X GET \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

DWH Integration

Bodyapplication/json
schedule_typestring
Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"
schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

idinteger(int64)
namestring

The name of the DWH Integration.

user_idinteger(int64)

The ID of the user who created the DWH Integration.

account_idinteger(int64)

The TD Account which the DWH Integration belongs to.

workflow_project_idinteger(int64)
workflow_project_namestring
dwh_config_idinteger(int64)
created_atstring(date-time)
updated_atstring(date-time)
Response
application/json
{
  "schedule_type": "none",
  "schedule_option": "string",
  "timezone": "string",
  "id": 0,
  "name": "string",
  "user_id": 0,
  "account_id": 0,
  "workflow_project_id": 0,
  "workflow_project_name": "string",
  "dwh_config_id": 0,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z"
}

Update a DWH Integration.

Request

Security
apiKey
Path
dwh_idinteger(int64)required

The ID of the DWH Integration.

Bodyapplication/jsonrequired
schedule_typestring
Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"
schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

namestring
basic_configsobject(BasicConfigs)

The basic configs of the DWH Integration.

advanced_configsArray of objects(AdvancedConfigs)

The DWH Integration advanced configs. The configs contain array of of table configs which combine with the basic configs will be used to create Connector Bulkload Session.

output_configobject(OutConfigs)

Config for how data will be imported into TD.

filter_configsArray of objects(FilterConfigs)

Config for how data will be filtered before importing into TD. See the connector filters for more info.

curl -i -X PUT \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}' \
  -H 'Authorization: YOUR_API_KEY_HERE' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "test",
    "schedule_type": "daily",
    "schedule_option": "11:00",
    "timezone": "Asia/Tokyo",
    "basic_configs": {
      "td_authentication_id": 11,
      "database": "database",
      "table": "table",
      "schema": "test"
    },
    "advanced_configs": [
      {
        "target_table": "test",
        "config": {
          "select": "1",
          "table": "test",
          "source_type": "presto"
        }
      }
    ],
    "output_config": {
      "type": "td",
      "database": "test",
      "table": "test",
      "schema": "test"
    }
  }'

Responses

DWH Integration Updated

Bodyapplication/json
schedule_typestring
Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"
schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

idinteger(int64)
namestring

The name of the DWH Integration.

user_idinteger(int64)

The ID of the user who created the DWH Integration.

account_idinteger(int64)

The TD Account which the DWH Integration belongs to.

workflow_project_idinteger(int64)
workflow_project_namestring
dwh_config_idinteger(int64)
created_atstring(date-time)
updated_atstring(date-time)
Response
application/json
{
  "schedule_type": "none",
  "schedule_option": "string",
  "timezone": "string",
  "id": 0,
  "name": "string",
  "user_id": 0,
  "account_id": 0,
  "workflow_project_id": 0,
  "workflow_project_name": "string",
  "dwh_config_id": 0,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z"
}

Delete a DWH Integration.

Request

This will destroy the integration, generated Workflow project and all generated Bulkload Sessions.

Security
apiKey
Path
dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X DELETE \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

DWH Integration Deleted

Bodyapplication/json
schedule_typestring
Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"
schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

idinteger(int64)
namestring

The name of the DWH Integration.

user_idinteger(int64)

The ID of the user who created the DWH Integration.

account_idinteger(int64)

The TD Account which the DWH Integration belongs to.

workflow_project_idinteger(int64)
workflow_project_namestring
dwh_config_idinteger(int64)
created_atstring(date-time)
updated_atstring(date-time)
Response
application/json
{
  "schedule_type": "none",
  "schedule_option": "string",
  "timezone": "string",
  "id": 0,
  "name": "string",
  "user_id": 0,
  "account_id": 0,
  "workflow_project_id": 0,
  "workflow_project_name": "string",
  "dwh_config_id": 0,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z"
}

List all Bulkload sessions of the current DWH Integration.

Request

Security
apiKey
Path
dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X GET \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}/bulkload_sessions' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

Bulkload sessions

Bodyapplication/jsonArray [
namestring

The Bulkload Session name, which will be used for accessing Connector Bulkload Session

bulkload_session_idinteger(int64)

The Bulkload Session ID

target_tablestring

The table where data will be imported into

configs_digeststring

The digest of the configs used to create the Bulkload Session

]
Response
application/json
[
  {
    "name": "string",
    "bulkload_session_id": 0,
    "target_table": "string",
    "configs_digest": "string"
  }
]

Get a DWH Integration configuration.

Request

Security
apiKey
Path
dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X GET \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}/configs' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

DWH Integration configuration

Bodyapplication/json
idinteger(int64)
config_digeststring
created_atstring(date-time)
updated_atstring(date-time)
basic_configsobject(BasicConfigs)

The basic configs of the DWH Integration.

advanced_configsArray of objects(AdvancedConfigs)

The DWH Integration advanced configs. The configs contain array of of table configs which combine with the basic configs will be used to create Connector Bulkload Session.

output_configobject(OutConfigs)

Config for how data will be imported into TD.

filter_configsArray of objects(FilterConfigs)

Config for how data will be filtered before importing into TD. See the connector filters for more info.

Response
application/json
{
  "id": 0,
  "config_digest": "string",
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "basic_configs": {
    "type": "string",
    "td_authentication_id": "string",
    "role": "string",
    "warehouse": "string",
    "db": "string",
    "schema": "string",
    "incremental": false
  },
  "advanced_configs": [
    { … }
  ],
  "output_config": {
    "database": "string",
    "mode": "string",
    "time_column": "string"
  },
  "filter_configs": [
    { … }
  ]
}

Trigger run the DWH Integration workflow.

Request

Security
apiKey
Path
dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X PUT \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}/run_workflow' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

See the response of GET /api/attempts https://docs.digdag.io/api/ for more info

Bodyapplication/json
statusstring
idstring
indexinteger
projectobject
workflowobject
sessionIdstring
sessionUuidstring
sessionTimestring(date-time)
retryAttemptNamestring
doneboolean
successboolean
cancelRequestedboolean
createdAtstring(date-time)
finishedAtstring(date-time)
Response
application/json
{
  "status": "string",
  "id": "string",
  "index": 0,
  "project": {
    "id": "string",
    "name": "string"
  },
  "workflow": {
    "name": "string",
    "id": "string"
  },
  "sessionId": "string",
  "sessionUuid": "string",
  "sessionTime": "2019-08-24T14:15:22Z",
  "retryAttemptName": "string",
  "done": true,
  "success": true,
  "cancelRequested": true,
  "createdAt": "2019-08-24T14:15:22Z",
  "finishedAt": "2019-08-24T14:15:22Z"
}

List all DWH Workflow session attempts.

Request

Security
apiKey
Path
dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X GET \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}/session_attempts' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

Session Attempts

Bodyapplication/json
itemsArray of objects(SessionAttempt)
paginationobject(Pagination)
Response
application/json
{
  "items": [
    { … }
  ],
  "pagination": {
    "page": 0,
    "size": 0,
    "total_pages": 0
  }
}

Guess DWH Integration configuration

Request

Perform Guess operation for complete DWH Integration configuration by providing basic_configs.

Security
apiKey
Bodyapplication/jsonrequired
basic_configsobject(BasicConfigs)

The basic configs of the DWH Integration.

property name*anyadditional property
curl -i -X POST \
  https://dwh-integration-api.treasuredata.com/v1/guess \
  -H 'Authorization: YOUR_API_KEY_HERE' \
  -H 'Content-Type: application/json' \
  -d '{
    "basic_configs": {
      "td_authentication_id": 11,
      "type": "snowflake",
      "warehouse": "test_wh",
      "db": "database",
      "schema": "test"
    }
  }'

Responses

Guess Successful

Bodyapplication/json
basic_configsobject(BasicConfigs)

The basic configs of the DWH Integration.

advanced_configsArray of objects(AdvancedConfigs)

The DWH Integration advanced configs. The configs contain array of of table configs which combine with the basic configs will be used to create Connector Bulkload Session.

Response
application/json
{
  "basic_configs": {
    "type": "string",
    "td_authentication_id": "string",
    "role": "string",
    "warehouse": "string",
    "db": "string",
    "schema": "string",
    "incremental": false
  },
  "advanced_configs": [
    { … }
  ]
}