GCP Dataproc - Paradime

GCP Dataproc is a managed service for running Apache Spark and Apache Hadoop workloads. Trigger it from a Paradime Bolt schedule with the Paradime CLI, so it runs as part of your pipeline.

What you can do

Submit a Dataproc job to a cluster
Supported job types: pyspark, spark, hive, spark-sql, pig, presto

Prerequisites

A service account with permission to submit jobs to the cluster (Dataproc Editor).

Set up

Paradime authenticates to GCP with a service account key. Set them as Bolt schedule environment variables:

Variable	Required	Description
`GCP_SERVICE_ACCOUNT_KEY_FILE`	Required	Path to your GCP service account JSON key file.
`GCP_PROJECT_ID`	Required	Your GCP project ID.
`GCP_LOCATION`	Required	GCP region (for example, `us-central1`).

Trigger from Bolt

Add the Run GCP Dataproc Job command to a schedule. For the full command reference, flags, and examples, see the GCP Dataproc CLI.

Related topics

GCP Dataproc CLI Run GCP Dataproc Job Integrations Commands New Bolt command integrations

⌘I