GCP Dataflow CLI

The Paradime SDK provides a CLI command to interact with Google Cloud Dataflow, allowing you to launch a Dataflow job from a classic or flex template.

🔑 Authentication RequiredYou will need a GCP service account JSON key file with permissions to launch Dataflow jobs in your project. Provide it via the --service-account-key-file flag or the GCP_SERVICE_ACCOUNT_KEY_FILE environment variable.

Launch a Dataflow Job

Launch a Dataflow job from a template (classic or flex), optionally passing template parameters and waiting for completion.

CLI Command

paradime run gcp-dataflow-trigger

Options

Flag	Type	Description
`--service-account-key-file`	Required, TEXT	Path to your GCP service account JSON key file. Can be set via `GCP_SERVICE_ACCOUNT_KEY_FILE` environment variable.
`--project`	Required, TEXT	Your GCP project ID. Can be set via `GCP_PROJECT_ID` environment variable.
`--location`	Required, TEXT	GCP region (e.g. ‘us-central1’). Can be set via `GCP_LOCATION` environment variable.
`--template-path`	Required, TEXT	GCS path to the Dataflow template (e.g. gs://bucket/templates/my-template).
`--job-name`	Required, TEXT	Name for the launched Dataflow job.
`--template-type`	Optional, CHOICE	Type of Dataflow template (classic or flex). Choices: `classic`, `flex`. Default: `classic`.
`--parameters`	Optional, TEXT	JSON string of template parameters (e.g. `{"inputFile": "gs://..."}`).
`--wait` / `--no-wait`	Optional, Flag	Wait for the Dataflow job to complete before returning. Default: `--wait`. Can be set via `GCP_DATAFLOW_TRIGGER_WAIT` environment variable.
`--timeout`	Optional, INTEGER	Maximum time to wait for completion (in minutes). Only used with `--wait`. Default: 1440. Can be set via `GCP_DATAFLOW_TRIGGER_TIMEOUT` environment variable.
`--help`	Optional, Flag	Show the help message and exit.

Recommended SetupFor security and convenience, set your GCP credentials as environment variables:

GCP_SERVICE_ACCOUNT_KEY_FILE="/path/to/service-account-key.json"
GCP_PROJECT_ID="your_project_id"
GCP_LOCATION="us-central1"

Usage Examples

# Launch a classic template job (using environment variables)
paradime run gcp-dataflow-trigger \
  --template-path "gs://bucket/templates/my-template" \
  --job-name "my-dataflow-job"

# Launch a flex template job with parameters
paradime run gcp-dataflow-trigger \
  --template-path "gs://bucket/templates/my-flex-template" \
  --job-name "my-flex-job" \
  --template-type flex \
  --parameters '{"inputFile": "gs://bucket/input.csv"}'

# With explicit credentials and no wait
paradime run gcp-dataflow-trigger \
  --service-account-key-file "/path/to/key.json" \
  --project "your_project_id" \
  --location "us-central1" \
  --template-path "gs://bucket/templates/my-template" \
  --job-name "my-dataflow-job" \
  --no-wait

Environment Variable Reference

Environment Variable	Description
`GCP_SERVICE_ACCOUNT_KEY_FILE`	Path to your GCP service account JSON key file
`GCP_PROJECT_ID`	Your GCP project ID
`GCP_LOCATION`	GCP region (e.g. ‘us-central1’)
`GCP_DATAFLOW_TRIGGER_WAIT`	Wait for the Dataflow job to complete before returning
`GCP_DATAFLOW_TRIGGER_TIMEOUT`	Maximum time to wait for completion (in minutes)

API Keys & Access

GraphQL API

Python SDK

Paradime CLI

Webhooks

Virtual Environments

Launch a Dataflow Job

CLI Command

Options

Usage Examples

Environment Variable Reference

​Launch a Dataflow Job

​CLI Command

​Options

​Usage Examples

​Environment Variable Reference

Launch a Dataflow Job

CLI Command

Options

Usage Examples

Environment Variable Reference