GCP Dataflow - Paradime

GCP Dataflow is a fully managed service for unified batch and stream data processing built on Apache Beam. Trigger it from a Paradime Bolt schedule with the Paradime CLI, so it runs as part of your pipeline.

What you can do

Launch a Dataflow job from a classic or Flex template
Pass template parameters per run

Prerequisites

A service account with permission to launch Dataflow jobs (for example, the Dataflow Developer role).

Set up

Paradime authenticates to GCP with a service account key. Set them as Bolt schedule environment variables:

Variable	Required	Description
`GCP_SERVICE_ACCOUNT_KEY_FILE`	Required	Path to your GCP service account JSON key file.
`GCP_PROJECT_ID`	Required	Your GCP project ID.
`GCP_LOCATION`	Required	GCP region (for example, `us-central1`).

Trigger from Bolt

Add the Run GCP Dataflow command to a schedule. For the full command reference, flags, and examples, see the GCP Dataflow CLI.

Related topics

GCP Dataflow CLI Run GCP Dataflow Integrations Commands New Bolt command integrations

⌘I