gcs_wait>: Wait for a file in Google Cloud Storage

gcs_wait> operator can be used to wait for file to appear in Google Cloud Storage.

Copy
Copied
+wait:
  gcs_wait>: my_bucket/some/file

+wait:
  gcs_wait>: gs://my_bucket/some/file

Secrets

When you don't know how to set secrets, please refer to Managing Workflow Secret

Options

  • gcs_wait> : URI | BUCKET/OBJECT

    Google Cloud Storage URI or path of the file to wait for.

    Examples:

    Copy
    Copied
    gcs_wait>: my-bucket/my-directory/my-data.gz

    Examples:

    Copy
    Copied
    gcs_wait>: gs://my-bucket/my-directory/my-data.gz
  • bucket : NAME

    The GCS bucket where the file is located. Can be used together with the object parameter instead of putting the path on the operator command line.

  • object : PATH

    The GCS path of the file. Can be used together with the bucket parameter instead of putting the path on the operator command line.

Output parameters

  • gcs_wait.last_object

    Information about the detected file.

    Copy
    Copied
    {
    "metadata": {
        "bucket": "my_bucket",
        "contentType": "text/plain",
        "crc32c": "yV/Pdw==",
        "etag": "CKjJ6/H4988CEAE=",
        "generation": 1477466841081000,
        "id": "my_bucket/some/file",
        "kind": "storage#object",
        "md5Hash": "IT4zYwc3D23HpSGe3nZ85A==",
        "mediaLink": "https://www.googleapis.com/download/storage/v1/b/my_bucket/o/some%2Ffile?generation=1477466841081000&alt=media",
        "metageneration": 1,
        "name": "some/file",
        "selfLink": "https://www.googleapis.com/storage/v1/b/my_bucket/o/some%2Ffile",
        "size": 4711,
        "storageClass": "STANDARD",
        "timeCreated": {
            "value": 1477466841070,
            "dateOnly": false,
            "timeZoneShift": 0
        },
        "updated": {
            "value": 1477466841070,
            "dateOnly": false,
            "timeZoneShift": 0
        }
    }
}

Note: The gcs_wait> operator makes use of polling with exponential backoff. As such there might be some time interval between a file being created and the gcs_wait> operator detecting it.