tdforeach>: Repeat using Treasure Data queries

tdforeach> operator loops subtasks for each result rows of a Hive or Presto query on Treasure Data.

Subtasks set at _do section can reference results using ${td.each.COLUMNNAME} syntax where COLUMNNAME is a name of column.

For example, if you run a query select email, name from users and the query returns 3 rows, this operator runs subtasks 3 times with ${td.each.email} and ${td.each.name} parameters.

Copy

Copied

_export:
  td:
    database: www_access

+for_each_users:
  td_for_each>: queries/users.sql
  _do:
    +show:
      echo>: found a user ${td.each.name} email ${td.each.email}

Secrets

When you don't know how to set secrets, please refer to Managing Workflow Secret

td.apikey : API_KEY
The Treasure Data API key to use when running Treasure Data queries.

Options

tdforeach> : FILE.sql
Path to a query template file. This file can contain ${...} syntax to embed variables.

Examples:
Copy

Copied
```
td>: queries/step1.sql
```
_parallel : BOOLEAN
Runs the repeating tasks in parallel.

Examples:
Copy

Copied
```
_parallel: true
```

_do : TASKS

Tasks to run.

Examples:

Copy

Copied

# Runs a single task
_do:
echo>: ${td.each}

Copy

Copied

# Runs multiple tasks
_do:
+show_email:
  echo>: ${td.each.email}
+show_name:
  echo>: ${td.each.name}

database : NAME
Name of a database.

Examples:
Copy

Copied
```
database: my_db
```
endpoint : ADDRESS
API endpoint (default: api.treasuredata.com).
use_ssl : BOOLEAN
Enable SSL (https) to access to the endpoint (default: true).
engine : presto
Query engine (presto or hive).

Examples:
Copy

Copied
```
engine: hive
```
Copy

Copied
```
engine: presto
```
priority : 0
Set Priority (From -2 (VERY LOW) to 2 (VERY HIGH) , default: 0 (NORMAL)).
job_retry : 0
Set automatic job retry count (default: 0).

We recommend that you not set retry count over 10. If the job is not succeessful less than 10 times retry, it needs some fix a cause of failure.
presto_pool_name : NAME
Name of a resource pool to run the query in. Applicable only when engine is presto.

Examples:
Copy

Copied
```
presto_pool_name: poc
```
hive_pool_name : NAME
Name of a resource pool to run the query in. Applicable only when engine is hive.

Examples:
Copy

Copied
```
engine: hive
pool_name: poc
```
engine_version : NAME
Specify engine version for Hive and Presto.

Examples:
Copy

Copied
```
engine: hive
engine_version: stable
```
hive_engine_version : NAME
Specify engine version for Hive. This option overrides engine_version if engine is hive.

Examples:
Copy

Copied
```
engine: hive
hive_engine_version: stable
```

Output parameters

td.last_job_id or td.last_job.id
The job id this task executed.

Examples:
Copy

Copied
```
52036074
```
td.last_job.num_records
The number of records of this job output.

Examples:
Copy

Copied
```
10
```