GCP Dataproc

Google Cloud Dataproc is a managed service for running Apache Spark and Apache Hadoop workloads on GCP clusters. Trigger Dataproc jobs as part of your Paradime Bolt schedules using the Paradime CLI.

​

What you can do

• Submit a job to a Dataproc cluster (pyspark, spark, hive, spark-sql, pig, or presto), optionally waiting for completion
• List all Dataproc clusters in a project and region

​

Prerequisites

• GCP_SERVICE_ACCOUNT_KEY_FILE: path to a GCP service account JSON key file with permissions to submit Dataproc jobs and list clusters
• GCP_PROJECT_ID: your GCP project ID
• GCP_LOCATION: GCP region (e.g. ‘us-central1’)

​

Trigger from Bolt

Set your credentials as environment variables on your Bolt schedule, then run the command as part of the schedule. See the full command reference, flags, and examples in the GCP Dataproc CLI documentation.