s3_wait>: Wait for a file in Amazon S3

s3_wait> operator waits for file to appear in Amazon S3.

Copy

Copied

+wait:
  s3_wait>: my-bucket/my-key

Secrets

When you don't know how to set secrets, please refer to Managing Workflow Secret

aws.s3.access_key_id, aws.access_key_id
The AWS Access Key ID to use when accessing S3.
aws.s3.secret_access_key, aws.secret_access_key
The AWS Secret Access Key to use when accessing S3.
aws.s3.region, aws.region
An optional explicit AWS Region in which to access S3.
aws.s3.endpoint
An optional explicit API endpoint to use when accessing S3. This overrides the region secret.
aws.s3.sse_c_key
An optional Customer-Provided Server-Side Encryption (SSE-C) key to use when accessing S3. Must be Base64 encoded.
aws.s3.sse_c_key_algorithm
An optional Customer-Provided Server-Side Encryption (SSE-C) key algorithm to use when accessing S3.
aws.s3.sse_c_key_md5
An optional MD5 digest of the Customer-Provided Server-Side Encryption (SSE-C) key to use when accessing S3. Must be Base64 encoded.

For more information about SSE-C, See the AWS S3 Documentation.

Options

s3_wait> : BUCKET/KEY
Path to the file in Amazon S3 to wait for.

Examples:
Copy

Copied
```
s3_wait>: my-bucket/my-data.gz
```
Copy

Copied
```
s3_wait>: my-bucket/file/in/a/directory
```
region : REGION
An optional explicit AWS Region in which to access S3. This may also be specified using the aws.s3.region secret.
endpoint : ENDPOINT
An optional explicit AWS Region in which to access S3. This may also be specified using the aws.s3.endpoint secret. Note: This will override the region parameter.
bucket : BUCKET
The S3 bucket where the file is located. Can be used together with the key parameter instead of putting the path on the operator line.
key : KEY
The S3 key of the file. Can be used together with the bucket parameter instead of putting the path on the operator line.
version_id : VERSION_ID
An optional object version to check for.
path_style_access : true/false
An optional flag to control whether to use path-style or virtual hosted-style access when accessing S3. Note: Enabling path_style_access also requires specifying a region.
timeout : TIMEOUT
Set timeout.

Examples: wait 120 seconds
Copy

Copied
```
timeout: 120s
```
continue_on_timeout : true/false (default:false)
If continueontimeout is set to true, the task will finish successfully on timeout. s3.lastobject is empty in this case. Empty check is required in following tasks if access to s3.lastobject.
Copy

Copied
```
+task1:
s3_wait>: bucket/object
timeout: 60s
continue_on_timeout: true
+task2:
if>: ${!!s3.last_object}
_do:
  echo>: "No timeout"
```

Output Parameters

s3.last_object

Information about the detected file.

Copy

Copied

{
  "metadata": {
    "Accept-Ranges": "bytes",
    "Access-Control-Allow-Origin": "*",
    "Content-Length": 4711,
    "Content-Type": "application/octet-stream",
    "ETag": "5eb63bbbe01eeed093cb22bb8f5acdc3",
    "Last-Modified": 1474360744000,
    "Last-Ranges": "bytes"
  },
  "user_metadata": {
    "foo": "bar",
    "baz": "quux"
  }
}

Note: The s3_wait> operator makes use of polling with exponential backoff. As such there might be some time interval between a file being created and the s3_wait> operator detecting it. This interval start at 5 seconds and increase to double each cycle, with a maximum interval of 5 minutes.