bq_ddl>: Managing Google BigQuery Datasets and Tables

bq_ddl> operator can be used to create, delete and clear Google BigQuery Datasets and Tables.

_export:
  bq:
    dataset: my_dataset

+prepare:
  bq_ddl>:
  create_datasets:
    - my_dataset_${session_date_compact}
  empty_datasets:
    - my_dataset_${session_date_compact}
  delete_datasets:
    - my_dataset_${last_session_date_compact}
  create_tables:
    - my_table_${session_date_compact}
  empty_tables:
    - my_table_${session_date_compact}
  delete_tables:
    - my_table_${last_session_date_compact}

Secrets

When you don't know how to set secrets, please refer to Managing Workflow Secret

• gcp.credential : CREDENTIAL

  See gcp.credential.

Options

• create_datasets : LIST

  Create new datasets.

  For detailed information about dataset configuration parameters, see the Google BigQuery Datasets Documentation.

  Examples:

  Copy

  Copied

  create_datasets:
  - foo
  - other_project:bar

  Copy

  Copied

  create_datasets:
  - foo_dataset_${session_date_compact}
  - id: bar_dataset_${session_date_compact}
    project: other_project
    friendly_name: Bar dataset ${session_date_compact}
    description: Bar dataset for ${session_date}
    default_table_expiration: 7d
    location: EU
    labels:
      foo: bar
      quux: 17
    access:
      - domain: example.com
        role: READER
      - userByEmail: ingest@example.com
        role: WRITER
      - groupByEmail: administrators@example.com
        role: OWNER

• empty_datasets : LIST

  Create new datasets, deleting them first if they already exist. Any tables in the datasets will also be deleted.

  For detailed information about dataset configuration parameters, see the Google BigQuery Datasets Documentation.

  Examples:

  Copy

  Copied

  empty_datasets:
  - foo
  - other_project:bar

  Copy

  Copied

  empty_datasets:
  - foo_dataset_${session_date_compact}
  - id: bar_dataset_${session_date_compact}
    project: other_project
    friendly_name: Bar dataset ${session_date_compact}
    description: Bar dataset for ${session_date}
    default_table_expiration: 7d
    location: EU
    labels:
      foo: bar
      quux: 17
    access:
      - domain: example.com
        role: READER
      - userByEmail: ingest@example.com
        role: WRITER
      - groupByEmail: administrators@example.com
        role: OWNER

• delete_datasets : LIST

  Delete datasets, if they exist.

  Examples:

  Copy

  Copied

  delete_datasets:
  - foo
  - other_project:bar

  Copy

  Copied

  delete_datasets:
  - foo_dataset_${last_session_date_compact}
  - other_project:bar_dataset_${last_session_date_compact}

• create_tables : LIST

  Create new tables.

  For detailed information about table configuration parameters, see the Google BigQuery Tables Documentation.

  Examples:

  Copy

  Copied

  create_tables:
  - foo
  - other_dataset.bar
  - other_project:yet_another_dataset.baz

  Copy

  Copied

  create_tables:
  - foo_dataset_${session_date_compact}
  - id: bar_dataset_${session_date_compact}
    project: other_project
    dataset: other_dataset
    friendly_name: Bar dataset ${session_date_compact}
    description: Bar dataset for ${session_date}
    expiration_time: 2016-11-01-T01:02:03Z
    schema:
      fields:
        - {name: foo, type: STRING}
        - {name: bar, type: INTEGER}
    labels:
      foo: bar
      quux: 17
    access:
      - domain: example.com
        role: READER
      - userByEmail: ingest@example.com
        role: WRITER
      - groupByEmail: administrators@example.com
        role: OWNER

• empty_tables : LIST Create new tables, deleting them first if they already exist.

  For detailed information about table configuration parameters, see the Google BigQuery Tables Documentation.

  Examples:

  Copy

  Copied

  empty_tables:
  - foo
  - other_dataset.bar
  - other_project:yet_another_dataset.baz

  Copy

  Copied

  empty_tables:
  - foo_table_${session_date_compact}
  - id: bar_table_${session_date_compact}
    project: other_project
    dataset: other_dataset
    friendly_name: Bar dataset ${session_date_compact}
    description: Bar dataset for ${session_date}
    expiration_time: 2016-11-01-T01:02:03Z
    schema:
      fields:
        - {name: foo, type: STRING}
        - {name: bar, type: INTEGER}
    labels:
      foo: bar
      quux: 17
    access:
      - domain: example.com
        role: READER
      - userByEmail: ingest@example.com
        role: WRITER
      - groupByEmail: administrators@example.com
        role: OWNER

• delete_tables : LIST Delete tables, if they exist.

  Examples:

  Copy

  Copied

  delete_tables:
  - foo
  - other_dataset.bar
  - other_project:yet_another_dataset.baz

  Copy

  Copied

  delete_tables:
  - foo_table_${last_session_date_compact}
  - bar_table_${last_session_date_compact}