redshift_load>: Redshift load operations

redshift_load> operator runs COPY statement to load data from external storage on Redshift.

Copy

Copied

_export:
  redshift:
    host: my-redshift.1234abcd.us-east-1.redshift.amazonaws.com
    # port: 5439
    database: production_db
    user: app_user
    ssl: true
    # strict_transaction: false

+load_from_dynamodb_simple:
    redshift_load>:
    schema: myschema
    table: transactions
    from: dynamodb://transaction-table
    readratio: 123

+load_from_s3_with_many_options:
    redshift_load>:
    schema: myschema
    table: access_logs
    from: s3://my-app-bucket/access_logs/today
    manifest: true
    encrypted: true
    region: us-east-1
    csv: "'"
    delimiter: "$"
    # json: s3://my-app-bucket/access_logs/jsonpathfile
    # avro: auto
    # fixedwidth: host:15,code:3,method:15
    gzip: true
    # bzip2: true
    # lzop: true
    acceptanydate: true
    acceptinvchars: "&"
    blanksasnull: true
    dateformat: yyyy-MM-dd
    emptyasnull: true
    encoding: UTF8
    escape: false
    explicit_ids: true
    fillrecord: true
    ignoreblanklines: true
    ignoreheader: 2
    null_as: nULl
    removequotes: false
    roundec: true
    timeformat: YYYY-MM-DD HH:MI:SS
    trimblanks: true
    truncatecolumns: true
    comprows: 12
    compupdate: ON
    maxerror: 34
    # noload: true
    statupdate: false
    role_session_name: federated_user
    session_duration: 1800
    # temp_credentials: false

Secrets

When you don't know how to set secrets, please refer to Managing Workflow Secret

aws.redshift.password : NAME
Optional user password to use when connecting to the Redshift database.
aws.redshift_load.access_key_id, aws.redshift.access_key_id, aws.access_key_id
The AWS Access Key ID to use when accessing data source. This value is used to get temporary security credentials by default. See temp_credentials option for details.
aws.redshift_load.secret_access_key, aws.redshift.secret_access_key, aws.secret_access_key
The AWS Secret Access Key to use when accessing data source. This value is used to get temporary security credentials by default. See temp_credentials option for details.
aws.redshift_load.role_arn, aws.redshift.role_arn, aws.role_arn
Optional Amazon resource names (ARNs) used to copy data to the Redshift. The role needs AssumeRole role to use this option. Requires temp_credentials to be true. If this option isn't specified, this operator tries to use a federated user

Options

database : NAME
Database name.

Examples:
Copy

Copied
```
database: my_db
```
host : NAME
Hostname or IP address of the database.

Examples:
Copy

Copied
```
host: db.foobar.com
```
port : NUMBER
Port number to connect to the database. Default: 5439.

Examples:
Copy

Copied
```
port: 2345
```
user : NAME
User to connect to the database

Examples:
Copy

Copied
```
user: app_user
```
ssl : BOOLEAN
Enable SSL to connect to the database. Default: false.

Examples:
Copy

Copied
```
ssl: true
```
schema : NAME
Default schema name. Default: public.

Examples:
Copy

Copied
```
schema: my_schema
```
strict_transaction : BOOLEAN
Whether this operator uses a strict transaction to prevent generating unexpected duplicated records just in case. Default: true. This operator creates and uses a status table in the database to make an operation idempotent. But if creating a table isn't allowed, this option should be false.

Examples:
Copy

Copied
```
strict_transaction: false
```
status_table_schema : NAME
Schema name of status table. Default: same as the value of schema option.

Examples:
Copy

Copied
```
status_table_schema: writable_schema
```
status_table : NAME
Table name prefix of status table. Default: __digdag_status.

Examples:
Copy

Copied
```
status_table: customized_status_table
```
table : NAME
Table name in Redshift database to be loaded data

Examples:
Copy

Copied
```
table: access_logs
```
from : URI
Parameter mapped to FROM parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
from: s3://my-app-bucket/access_logs/today
```
manifest : BOOLEAN
Parameter mapped to MANIFEST parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
manifest: true
```
encrypted : BOOLEAN
Parameter mapped to ENCRYPTED parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
encrypted: true
```
readratio : NUMBER
Parameter mapped to READRATIO parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
readratio: 150
```
region : NAME
Parameter mapped to REGION parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
region: us-east-1
```
csv : CHARACTER
Parameter mapped to CSV parameter of Redshift's COPY statement. If you want to just use default quote character of CSV parameter, set empty string like csv: ''

Examples:
Copy

Copied
```
csv: "'"
```
delimiter : CHARACTER
Parameter mapped to DELIMITER parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
delimiter: "$"
```
json : URI
Parameter mapped to JSON parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
json: auto
```
Examples:
Copy

Copied
```
json: s3://my-app-bucket/access_logs/jsonpathfile
```
avro : URI
Parameter mapped to AVRO parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
avro: auto
```
Copy

Copied
```
avro: s3://my-app-bucket/access_logs/jsonpathfile
```
fixedwidth : CSV
Parameter mapped to FIXEDWIDTH parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
fixedwidth: host:15,code:3,method:15
```
gzip : BOOLEAN
Parameter mapped to GZIP parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
gzip: true
```
bzip2 : BOOLEAN
Parameter mapped to BZIP2 parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
bzip2: true
```
lzop : BOOLEAN
Parameter mapped to LZOP parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
lzop: true
```
acceptanydate : BOOLEAN
Parameter mapped to ACCEPTANYDATE parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
acceptanydate: true
```
acceptinvchars : CHARACTER
Parameter mapped to ACCEPTINVCHARS parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
acceptinvchars: "&"
```
blanksasnull : BOOLEAN
Parameter mapped to BLANKSASNULL parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
blanksasnull: true
```
dateformat : STRING
Parameter mapped to DATEFORMAT parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
dateformat: yyyy-MM-dd
```
emptyasnull : BOOLEAN
Parameter mapped to EMPTYASNULL parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
emptyasnull: true
```
encoding : TYPE
Parameter mapped to ENCODING parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
encoding: UTF8
```
escape : BOOLEAN
Parameter mapped to ESCAPE parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
escape: false
```
explicit_ids : BOOLEAN
Parameter mapped to EXPLICIT_IDS parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
explicit_ids: true
```
fillrecord : BOOLEAN
Parameter mapped to FILLRECORD parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
fillrecord: true
```
ignoreblanklines : BOOLEAN
Parameter mapped to IGNOREBLANKLINES parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
ignoreblanklines: true
```
ignoreheader : NUMBER
Parameter mapped to IGNOREHEADER parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
ignoreheader: 2
```
null_as : STRING
Parameter mapped to NULL AS parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
null_as: nULl
```
removequotes : BOOLEAN
Parameter mapped to REMOVEQUOTES parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
removequotes: false
```
roundec : BOOLEAN
Parameter mapped to ROUNDEC parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
roundec: true
```
timeformat : STRING
Parameter mapped to TIMEFORMAT parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
timeformat: YYYY-MM-DD HH:MI:SS
```
trimblanks : BOOLEAN
Parameter mapped to TRIMBLANKS parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
trimblanks: true
```
truncatecolumns : BOOLEAN
Parameter mapped to TRUNCATECOLUMNS parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
truncatecolumns: true
```
comprows : NUMBER
Parameter mapped to COMPROWS parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
comprows: 12
```
compupdate : TYPE
Parameter mapped to COMPUPDATE parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
compupdate: ON
```
maxerror : NUMBER
Parameter mapped to MAXERROR parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
maxerror: 34
```
noload : BOOLEAN
Parameter mapped to NOLOAD parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
noload: true
```
statupdate : TYPE
Parameter mapped to STATUPDATE parameter of Redshift's COPY statement

Examples:
Copy

Copied
```
statupdate: off
```
temp_credentials : BOOLEAN
Whether this operator uses temporary security credentials. Default: true. This operator tries to use temporary security credentials as follows:
- If role_arn is specified, it calls AssumeRole action
- If not, it calls GetFederationToken action
See details about AssumeRole and GetFederationToken in the documents of AWS Security Token Service.

So either of AssumeRole or GetFederationToken action is called to use temporary security credentials by default for secure operation. But if this option is disabled, this operator uses credentials as-is set in the secrets insread of temporary security credentials.

Examples:
Copy

Copied
```
temp_credentials: false
```
session_duration INTEGER
Session duration of temporary security credentials. Default: 3 hour. This option isn't used when disabling temp_credentials

Examples:
Copy

Copied
```
session_duration: 1800
```