GCP Dataproc CLI

The Paradime SDK provides CLI commands to interact with GCP Dataproc, allowing you to submit jobs to Dataproc clusters and list available clusters with real-time monitoring capabilities.

🔑 API Access RequiredYou will need a GCP service account JSON key file with permissions to submit Dataproc jobs and list clusters. The service account key file path, project ID, and location can be provided as flags or set via environment variables.🔧 Cluster Configuration

Ensure the target Dataproc cluster exists and is running in the specified project and region.
Confirm the service account has the necessary permissions to submit jobs to the cluster.
Verify that any GCS paths referenced by --main-file or --job-file are accessible to the cluster.

Submit a Dataproc Job

Submit a job to a Dataproc cluster.

CLI Command

paradime run gcp-dataproc-trigger

Options

Flag	Type	Description
`--service-account-key-file`	Required, TEXT	Path to your GCP service account JSON key file. Can be set via `GCP_SERVICE_ACCOUNT_KEY_FILE` environment variable.
`--project`	Required, TEXT	Your GCP project ID. Can be set via `GCP_PROJECT_ID` environment variable.
`--location`	Required, TEXT	GCP region (e.g. ‘us-central1’). Can be set via `GCP_LOCATION` environment variable.
`--cluster-name`	Required, TEXT	The name of the Dataproc cluster to submit the job to.
`--job-type`	Required, CHOICE	Type of Dataproc job to submit. Choices: `pyspark`, `spark`, `hive`, `spark-sql`, `pig`, `presto`.
`--main-file`	Optional, TEXT	GCS path to the main file (required for pyspark/spark jobs).
`--main-class`	Optional, TEXT	Main class name (for spark jobs with JAR).
`--args`	Optional, TEXT (multiple)	Arguments to pass to the job.
`--job-file`	Optional, TEXT	GCS path to the query file (required for hive/spark-sql/pig/presto jobs).
`--wait` / `--no-wait`	Optional, Flag	Wait for the Dataproc job to complete before returning. Default: `--wait`. Can be set via `GCP_DATAPROC_TRIGGER_WAIT` environment variable.
`--timeout`	Optional, INTEGER	Maximum time to wait for completion (in minutes). Only used with `--wait`. Default: 1440 (24 hours). Can be set via `GCP_DATAPROC_TRIGGER_TIMEOUT` environment variable.
`--help`	Optional, Flag	Show the help message and exit.

Recommended SetupFor security and convenience, set your GCP credentials as environment variables:

GCP_SERVICE_ACCOUNT_KEY_FILE="/path/to/service-account-key.json"
GCP_PROJECT_ID="your-gcp-project-id"
GCP_LOCATION="us-central1"

Examples

# Submit a PySpark job (using environment variables)
paradime run gcp-dataproc-trigger \
  --cluster-name "my-cluster" \
  --job-type pyspark \
  --main-file "gs://my-bucket/jobs/main.py" \
  --args "--date=2024-01-15"

List Dataproc Clusters

List all Dataproc clusters in the specified project and region.

CLI Command

paradime run gcp-dataproc-list-clusters

Options

Flag	Type	Description
`--service-account-key-file`	Required, TEXT	Path to your GCP service account JSON key file. Can be set via `GCP_SERVICE_ACCOUNT_KEY_FILE` environment variable.
`--project`	Required, TEXT	Your GCP project ID. Can be set via `GCP_PROJECT_ID` environment variable.
`--location`	Required, TEXT	GCP region (e.g. ‘us-central1’). Can be set via `GCP_LOCATION` environment variable.
`--help`	Optional, Flag	Show the help message and exit.

Usage Examples

# List all clusters (using environment variables)
paradime run gcp-dataproc-list-clusters

# With explicit credentials
paradime run gcp-dataproc-list-clusters \
  --service-account-key-file "/path/to/service-account-key.json" \
  --project "your-gcp-project-id" \
  --location "us-central1"

Environment Variable Reference

Environment Variable	Description
`GCP_SERVICE_ACCOUNT_KEY_FILE`	Path to your GCP service account JSON key file
`GCP_PROJECT_ID`	Your GCP project ID
`GCP_LOCATION`	GCP region (e.g. ‘us-central1’)
`GCP_DATAPROC_TRIGGER_WAIT`	Whether to wait for job completion
`GCP_DATAPROC_TRIGGER_TIMEOUT`	Maximum time to wait for completion (minutes)

Workflow Integration

These GCP Dataproc CLI commands are designed to integrate seamlessly into your data pipeline workflows. For more information about integrating them into your Paradime workflows, see the Bolt Schedules documentation.

API Keys & Access

GraphQL API

Python SDK

Paradime CLI

Webhooks

Virtual Environments

Submit a Dataproc Job

CLI Command

Options

Examples

List Dataproc Clusters

CLI Command

Options

Usage Examples

Environment Variable Reference

Workflow Integration

​Submit a Dataproc Job

​CLI Command

​Options

​Examples

​List Dataproc Clusters

​CLI Command

​Options

​Usage Examples

​Environment Variable Reference

​Workflow Integration

Submit a Dataproc Job

CLI Command

Options

Examples

List Dataproc Clusters

CLI Command

Options

Usage Examples

Environment Variable Reference

Workflow Integration