bq_ddl>: Managing Google BigQuery Datasets and Tables

bq_ddl> operator can be used to create, delete and clear Google BigQuery Datasets and Tables.

Copy

Copied

_export:
  bq:
    dataset: my_dataset

+prepare:
  bq_ddl>:
  create_datasets:
    - my_dataset_${session_date_compact}
  empty_datasets:
    - my_dataset_${session_date_compact}
  delete_datasets:
    - my_dataset_${last_session_date_compact}
  create_tables:
    - my_table_${session_date_compact}
  empty_tables:
    - my_table_${session_date_compact}
  delete_tables:
    - my_table_${last_session_date_compact}

Secrets

When you don't know how to set secrets, please refer to Managing Workflow Secret

gcp.credential : CREDENTIAL
See gcp.credential.

Options

create_datasets : LIST
Create new datasets.

For detailed information about dataset configuration parameters, see the Google BigQuery Datasets Documentation.

Examples:

Copy

Copied

  create_datasets:
    - foo
    - other_project:bar

Copy

Copied

  create_datasets:
    - foo_dataset_${session_date_compact}
    - id: bar_dataset_${session_date_compact}
      project: other_project
      friendly_name: Bar dataset ${session_date_compact}
      description: Bar dataset for ${session_date}
      default_table_expiration: 7d
      location: EU
      labels:
        foo: bar
        quux: 17
      access:
        - domain: example.com
          role: READER
        - userByEmail: ingest@example.com
          role: WRITER
        - groupByEmail: administrators@example.com
          role: OWNER

empty_datasets : LIST
Create new datasets, deleting them first if they already exist. Any tables in the datasets will also be deleted.

For detailed information about dataset configuration parameters, see the Google BigQuery Datasets Documentation.

Examples:

Copy

Copied

  empty_datasets:
    - foo
    - other_project:bar

Copy

Copied

  empty_datasets:
    - foo_dataset_${session_date_compact}
    - id: bar_dataset_${session_date_compact}
      project: other_project
      friendly_name: Bar dataset ${session_date_compact}
      description: Bar dataset for ${session_date}
      default_table_expiration: 7d
      location: EU
      labels:
        foo: bar
        quux: 17
      access:
        - domain: example.com
          role: READER
        - userByEmail: ingest@example.com
          role: WRITER
        - groupByEmail: administrators@example.com
          role: OWNER

delete_datasets : LIST
Delete datasets, if they exist.

Examples:

Copy

Copied

  delete_datasets:
    - foo
    - other_project:bar

Copy

Copied

  delete_datasets:
    - foo_dataset_${last_session_date_compact}
    - other_project:bar_dataset_${last_session_date_compact}

create_tables : LIST
Create new tables.

For detailed information about table configuration parameters, see the Google BigQuery Tables Documentation.

Examples:

Copy

Copied

  create_tables:
    - foo
    - other_dataset.bar
    - other_project:yet_another_dataset.baz

Copy

Copied

  create_tables:
    - foo_dataset_${session_date_compact}
    - id: bar_dataset_${session_date_compact}
      project: other_project
      dataset: other_dataset
      friendly_name: Bar dataset ${session_date_compact}
      description: Bar dataset for ${session_date}
      expiration_time: 2016-11-01-T01:02:03Z
      schema:
        fields:
          - {name: foo, type: STRING}
          - {name: bar, type: INTEGER}
      labels:
        foo: bar
        quux: 17
      access:
        - domain: example.com
          role: READER
        - userByEmail: ingest@example.com
          role: WRITER
        - groupByEmail: administrators@example.com
          role: OWNER

empty_tables : LIST Create new tables, deleting them first if they already exist.
For detailed information about table configuration parameters, see the Google BigQuery Tables Documentation.

Examples:

Copy

Copied

  empty_tables:
    - foo
    - other_dataset.bar
    - other_project:yet_another_dataset.baz

Copy

Copied

  empty_tables:
    - foo_table_${session_date_compact}
    - id: bar_table_${session_date_compact}
      project: other_project
      dataset: other_dataset
      friendly_name: Bar dataset ${session_date_compact}
      description: Bar dataset for ${session_date}
      expiration_time: 2016-11-01-T01:02:03Z
      schema:
        fields:
          - {name: foo, type: STRING}
          - {name: bar, type: INTEGER}
      labels:
        foo: bar
        quux: 17
      access:
        - domain: example.com
          role: READER
        - userByEmail: ingest@example.com
          role: WRITER
        - groupByEmail: administrators@example.com
          role: OWNER

delete_tables : LIST Delete tables, if they exist.
Examples:

Copy

Copied

  delete_tables:
    - foo
    - other_dataset.bar
    - other_project:yet_another_dataset.baz

Copy

Copied

  delete_tables:
    - foo_table_${last_session_date_compact}
    - bar_table_${last_session_date_compact}