List all DWH Integrations for the current user.

Request

Security

apiKey

curl -i -X GET \
  https://dwh-integration-api.treasuredata.com/v1/dwh/ \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

DWH Integrations

Bodyapplication/json

itemsArray of objects(DwhIntegration)

paginationobject(Pagination)

Response

application/json

{
  "items": [
    { … }
  ],
  "pagination": {
    "page": 0,
    "size": 0,
    "total_pages": 0
  }
}

Create a new DWH Integration

Request

Note: the DWH Integration will not automatically create the TD database and tables. You need to create the TD database and tables before creating the DWH Integration.

Security

apiKey

Bodyapplication/jsonrequired

schedule_typestring

Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"

schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

namestringrequired

basic_configsobject(BasicConfigs)required

The basic configs of the DWH Integration.

basic_configs.typestringrequired

Connector type

Discriminator

basic_configs.td_authentication_idstringrequired

The existing Connector Authentication ID

basic_configs.rolestring(Role Name)

Specify the role name, Leave blank to use default role.

basic_configs.warehousestring(Snowflake Warehouse)required

The Snowflake data Warehouse to use for the connection, e.g. "COMPUTE_WH"

basic_configs.dbstring(Snowflake Database)required

basic_configs.schemastring(Schema, e.g. "PUBLIC")required

The schema to use for the connection

basic_configs.incrementalboolean(Incremental Loading?)

Enable incremental loading for this connector

Default false

advanced_configsArray of objects(AdvancedConfigs)required

The DWH Integration advanced configs. The configs contain array of of table configs which combine with the basic configs will be used to create Connector Bulkload Session.

advanced_configs[].target_tablestring

The name of the TD table to import data into.

advanced_configs[].configobject(Advanced Config)

The configuration which will merge with the basic_configs to form a complete Connector configuration for Bulkload job. Some of the config from the basic_configs can declare here to override the basic_configs.

output_configobject(OutConfigs)required

Config for how data will be imported into TD.

output_config.databasestring

The name of the TD database to import data into. The database must be created before the DWH Integration is created.

output_config.modestring

Specify the import mode. The default value is append.

output_config.time_columnstring

The value of the column will be used as the TD timestamp column.

filter_configsArray of objects(FilterConfigs)

Config for how data will be filtered before importing into TD. See the connector filters for more info.

curl -i -X POST \
  https://dwh-integration-api.treasuredata.com/v1/dwh/ \
  -H 'Authorization: YOUR_API_KEY_HERE' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "test",
    "schedule_type": "daily",
    "schedule_option": "10:00",
    "timezone": "Asia/Tokyo",
    "basic_configs": {
      "td_authentication_id": 11,
      "database": "database",
      "table": "table",
      "schema": "test"
    },
    "advanced_configs": [
      {
        "target_table": "test",
        "config": {
          "select": "1",
          "table": "test",
          "source_type": "presto"
        }
      }
    ],
    "output_config": {
      "type": "td",
      "database": "test",
      "table": "test",
      "schema": "test"
    }
  }'

Responses

DWH Integration Created

Bodyapplication/json

schedule_typestring

Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"

schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

idinteger(int64)

namestring

The name of the DWH Integration.

user_idinteger(int64)

The ID of the user who created the DWH Integration.

account_idinteger(int64)

The TD Account which the DWH Integration belongs to.

workflow_project_idinteger(int64)

workflow_project_namestring

dwh_config_idinteger(int64)

created_atstring(date-time)

updated_atstring(date-time)

Response

application/json

{
  "schedule_type": "none",
  "schedule_option": "string",
  "timezone": "string",
  "id": 0,
  "name": "string",
  "user_id": 0,
  "account_id": 0,
  "workflow_project_id": 0,
  "workflow_project_name": "string",
  "dwh_config_id": 0,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z"
}

Get a DWH Integration.

Request

Security

apiKey

Path

dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X GET \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

DWH Integration

Bodyapplication/json

schedule_typestring

Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"

schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

idinteger(int64)

namestring

The name of the DWH Integration.

user_idinteger(int64)

The ID of the user who created the DWH Integration.

account_idinteger(int64)

The TD Account which the DWH Integration belongs to.

workflow_project_idinteger(int64)

workflow_project_namestring

dwh_config_idinteger(int64)

created_atstring(date-time)

updated_atstring(date-time)

Response

application/json

{
  "schedule_type": "none",
  "schedule_option": "string",
  "timezone": "string",
  "id": 0,
  "name": "string",
  "user_id": 0,
  "account_id": 0,
  "workflow_project_id": 0,
  "workflow_project_name": "string",
  "dwh_config_id": 0,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z"
}

Update a DWH Integration.

Request

Security

apiKey

Path

dwh_idinteger(int64)required

The ID of the DWH Integration.

Bodyapplication/jsonrequired

schedule_typestring

Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"

schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

namestring

basic_configsobject(BasicConfigs)

The basic configs of the DWH Integration.

advanced_configsArray of objects(AdvancedConfigs)

The DWH Integration advanced configs. The configs contain array of of table configs which combine with the basic configs will be used to create Connector Bulkload Session.

output_configobject(OutConfigs)

Config for how data will be imported into TD.

filter_configsArray of objects(FilterConfigs)

Config for how data will be filtered before importing into TD. See the connector filters for more info.

curl -i -X PUT \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}' \
  -H 'Authorization: YOUR_API_KEY_HERE' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "test",
    "schedule_type": "daily",
    "schedule_option": "11:00",
    "timezone": "Asia/Tokyo",
    "basic_configs": {
      "td_authentication_id": 11,
      "database": "database",
      "table": "table",
      "schema": "test"
    },
    "advanced_configs": [
      {
        "target_table": "test",
        "config": {
          "select": "1",
          "table": "test",
          "source_type": "presto"
        }
      }
    ],
    "output_config": {
      "type": "td",
      "database": "test",
      "table": "test",
      "schema": "test"
    }
  }'

Responses

DWH Integration Updated

Bodyapplication/json

schedule_typestring

Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"

schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

idinteger(int64)

namestring

The name of the DWH Integration.

user_idinteger(int64)

The ID of the user who created the DWH Integration.

account_idinteger(int64)

The TD Account which the DWH Integration belongs to.

workflow_project_idinteger(int64)

workflow_project_namestring

dwh_config_idinteger(int64)

created_atstring(date-time)

updated_atstring(date-time)

Response

application/json

{
  "schedule_type": "none",
  "schedule_option": "string",
  "timezone": "string",
  "id": 0,
  "name": "string",
  "user_id": 0,
  "account_id": 0,
  "workflow_project_id": 0,
  "workflow_project_name": "string",
  "dwh_config_id": 0,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z"
}

Delete a DWH Integration.

Request

This will destroy the integration, generated Workflow project and all generated Bulkload Sessions.

Security

apiKey

Path

dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X DELETE \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

DWH Integration Deleted

Bodyapplication/json

schedule_typestring

Enum"none""daily""hourly""weekly""monthly""minutes_interval""cron"

schedule_optionstring

The schedule value for the schedule_type. For example, if schedule_type is daily, schedule_option should be a time string like 10:00.

timezonestring

The timezone of the schedule. For example, UTC or Asia/Tokyo.

idinteger(int64)

namestring

The name of the DWH Integration.

user_idinteger(int64)

The ID of the user who created the DWH Integration.

account_idinteger(int64)

The TD Account which the DWH Integration belongs to.

workflow_project_idinteger(int64)

workflow_project_namestring

dwh_config_idinteger(int64)

created_atstring(date-time)

updated_atstring(date-time)

Response

application/json

{
  "schedule_type": "none",
  "schedule_option": "string",
  "timezone": "string",
  "id": 0,
  "name": "string",
  "user_id": 0,
  "account_id": 0,
  "workflow_project_id": 0,
  "workflow_project_name": "string",
  "dwh_config_id": 0,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z"
}

List all Bulkload sessions of the current DWH Integration.

Request

Security

apiKey

Path

dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X GET \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}/bulkload_sessions' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

Bulkload sessions

Bodyapplication/jsonArray [

namestring

The Bulkload Session name, which will be used for accessing Connector Bulkload Session

bulkload_session_idinteger(int64)

The Bulkload Session ID

target_tablestring

The table where data will be imported into

configs_digeststring

The digest of the configs used to create the Bulkload Session

]

Response

application/json

[
  {
    "name": "string",
    "bulkload_session_id": 0,
    "target_table": "string",
    "configs_digest": "string"
  }
]

Get a DWH Integration configuration.

Request

Security

apiKey

Path

dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X GET \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}/configs' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

DWH Integration configuration

Bodyapplication/json

idinteger(int64)

config_digeststring

created_atstring(date-time)

updated_atstring(date-time)

basic_configsobject(BasicConfigs)

The basic configs of the DWH Integration.

advanced_configsArray of objects(AdvancedConfigs)

The DWH Integration advanced configs. The configs contain array of of table configs which combine with the basic configs will be used to create Connector Bulkload Session.

output_configobject(OutConfigs)

Config for how data will be imported into TD.

filter_configsArray of objects(FilterConfigs)

Config for how data will be filtered before importing into TD. See the connector filters for more info.

Response

application/json

{
  "id": 0,
  "config_digest": "string",
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "basic_configs": {
    "type": "string",
    "td_authentication_id": "string",
    "role": "string",
    "warehouse": "string",
    "db": "string",
    "schema": "string",
    "incremental": false
  },
  "advanced_configs": [
    { … }
  ],
  "output_config": {
    "database": "string",
    "mode": "string",
    "time_column": "string"
  },
  "filter_configs": [
    { … }
  ]
}

Trigger run the DWH Integration workflow.

Request

Security

apiKey

Path

dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X PUT \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}/run_workflow' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

See the response of GET /api/attempts https://docs.digdag.io/api/ for more info

Bodyapplication/json

statusstring

idstring

indexinteger

projectobject

workflowobject

sessionIdstring

sessionUuidstring

sessionTimestring(date-time)

retryAttemptNamestring

doneboolean

successboolean

cancelRequestedboolean

createdAtstring(date-time)

finishedAtstring(date-time)

Response

application/json

{
  "status": "string",
  "id": "string",
  "index": 0,
  "project": {
    "id": "string",
    "name": "string"
  },
  "workflow": {
    "name": "string",
    "id": "string"
  },
  "sessionId": "string",
  "sessionUuid": "string",
  "sessionTime": "2019-08-24T14:15:22Z",
  "retryAttemptName": "string",
  "done": true,
  "success": true,
  "cancelRequested": true,
  "createdAt": "2019-08-24T14:15:22Z",
  "finishedAt": "2019-08-24T14:15:22Z"
}

List all DWH Workflow session attempts.

Request

Security

apiKey

Path

dwh_idinteger(int64)required

The ID of the DWH Integration.

curl -i -X GET \
  'https://dwh-integration-api.treasuredata.com/v1/dwh/{dwh_id}/session_attempts' \
  -H 'Authorization: YOUR_API_KEY_HERE'

Responses

Session Attempts

Bodyapplication/json

itemsArray of objects(SessionAttempt)

paginationobject(Pagination)

Response

application/json

{
  "items": [
    { … }
  ],
  "pagination": {
    "page": 0,
    "size": 0,
    "total_pages": 0
  }
}

Guess DWH Integration configuration

Request

Perform Guess operation for complete DWH Integration configuration by providing basic_configs.

Security

apiKey

Bodyapplication/jsonrequired

basic_configsobject(BasicConfigs)

The basic configs of the DWH Integration.

property name*anyadditional property

curl -i -X POST \
  https://dwh-integration-api.treasuredata.com/v1/guess \
  -H 'Authorization: YOUR_API_KEY_HERE' \
  -H 'Content-Type: application/json' \
  -d '{
    "basic_configs": {
      "td_authentication_id": 11,
      "type": "snowflake",
      "warehouse": "test_wh",
      "db": "database",
      "schema": "test"
    }
  }'

Responses

Guess Successful

Bodyapplication/json

basic_configsobject(BasicConfigs)

The basic configs of the DWH Integration.

advanced_configsArray of objects(AdvancedConfigs)

The DWH Integration advanced configs. The configs contain array of of table configs which combine with the basic configs will be used to create Connector Bulkload Session.

Response

application/json

{
  "basic_configs": {
    "type": "string",
    "td_authentication_id": "string",
    "role": "string",
    "warehouse": "string",
    "db": "string",
    "schema": "string",
    "incremental": false
  },
  "advanced_configs": [
    { … }
  ]
}