GCP Dataflow

Google Cloud Dataflow is a managed service for executing Apache Beam data processing pipelines at scale. Trigger Dataflow jobs as part of your Paradime Bolt schedules using the Paradime CLI.

​

What you can do

• Launch a Dataflow job from a classic or flex template, optionally passing template parameters and waiting for completion

​

Prerequisites

• GCP_SERVICE_ACCOUNT_KEY_FILE: path to a GCP service account JSON key file with permissions to launch Dataflow jobs
• GCP_PROJECT_ID: your GCP project ID
• GCP_LOCATION: GCP region (e.g. ‘us-central1’)

​

Trigger from Bolt

Set your credentials as environment variables on your Bolt schedule, then run the command as part of the schedule. See the full command reference, flags, and examples in the GCP Dataflow CLI documentation.