> ## Documentation Index
> Fetch the complete documentation index at: https://docs.paradime.io/llms.txt
> Use this file to discover all available pages before exploring further.

# GCP Dataproc CLI

> Submit Spark, PySpark, Hive, and Presto jobs to Google Cloud Dataproc clusters and list clusters from Paradime Bolt using the paradime run gcp-dataproc CLI.

The Paradime SDK provides CLI commands to interact with GCP Dataproc, allowing you to submit jobs to Dataproc clusters and list available clusters with real-time monitoring capabilities.

<Warning>
  🔑 **API Access Required**

  You will need a GCP service account JSON key file with permissions to submit Dataproc jobs and list clusters. The service account key file path, project ID, and location can be provided as flags or set via environment variables.

  **🔧 Cluster Configuration**

  * Ensure the target Dataproc cluster exists and is running in the specified project and region.
  * Confirm the service account has the necessary permissions to submit jobs to the cluster.
  * Verify that any GCS paths referenced by `--main-file` or `--job-file` are accessible to the cluster.
</Warning>

## Submit a Dataproc Job

Submit a job to a Dataproc cluster.

### CLI Command

```bash theme={"system"}
paradime run gcp-dataproc-trigger
```

#### Options

| Flag                         | Type                      | Description                                                                                                                                                              |
| ---------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--service-account-key-file` | Required, TEXT            | Path to your GCP service account JSON key file. Can be set via `GCP_SERVICE_ACCOUNT_KEY_FILE` environment variable.                                                      |
| `--project`                  | Required, TEXT            | Your GCP project ID. Can be set via `GCP_PROJECT_ID` environment variable.                                                                                               |
| `--location`                 | Required, TEXT            | GCP region (e.g. 'us-central1'). Can be set via `GCP_LOCATION` environment variable.                                                                                     |
| `--cluster-name`             | Required, TEXT            | The name of the Dataproc cluster to submit the job to.                                                                                                                   |
| `--job-type`                 | Required, CHOICE          | Type of Dataproc job to submit. Choices: `pyspark`, `spark`, `hive`, `spark-sql`, `pig`, `presto`.                                                                       |
| `--main-file`                | Optional, TEXT            | GCS path to the main file (required for pyspark/spark jobs).                                                                                                             |
| `--main-class`               | Optional, TEXT            | Main class name (for spark jobs with JAR).                                                                                                                               |
| `--args`                     | Optional, TEXT (multiple) | Arguments to pass to the job.                                                                                                                                            |
| `--job-file`                 | Optional, TEXT            | GCS path to the query file (required for hive/spark-sql/pig/presto jobs).                                                                                                |
| `--wait` / `--no-wait`       | Optional, Flag            | Wait for the Dataproc job to complete before returning. Default: `--wait`. Can be set via `GCP_DATAPROC_TRIGGER_WAIT` environment variable.                              |
| `--timeout`                  | Optional, INTEGER         | Maximum time to wait for completion (in minutes). Only used with `--wait`. Default: 1440 (24 hours). Can be set via `GCP_DATAPROC_TRIGGER_TIMEOUT` environment variable. |
| `--help`                     | Optional, Flag            | Show the help message and exit.                                                                                                                                          |

<Info>
  **Recommended Setup**

  For security and convenience, set your GCP credentials as environment variables:

  ```bash theme={"system"}
  GCP_SERVICE_ACCOUNT_KEY_FILE="/path/to/service-account-key.json"
  GCP_PROJECT_ID="your-gcp-project-id"
  GCP_LOCATION="us-central1"
  ```
</Info>

### Examples

```bash theme={"system"}
# Submit a PySpark job (using environment variables)
paradime run gcp-dataproc-trigger \
  --cluster-name "my-cluster" \
  --job-type pyspark \
  --main-file "gs://my-bucket/jobs/main.py" \
  --args "--date=2024-01-15"
```

## List Dataproc Clusters

List all Dataproc clusters in the specified project and region.

### CLI Command

```bash theme={"system"}
paradime run gcp-dataproc-list-clusters
```

#### Options

| Flag                         | Type           | Description                                                                                                         |
| ---------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------- |
| `--service-account-key-file` | Required, TEXT | Path to your GCP service account JSON key file. Can be set via `GCP_SERVICE_ACCOUNT_KEY_FILE` environment variable. |
| `--project`                  | Required, TEXT | Your GCP project ID. Can be set via `GCP_PROJECT_ID` environment variable.                                          |
| `--location`                 | Required, TEXT | GCP region (e.g. 'us-central1'). Can be set via `GCP_LOCATION` environment variable.                                |
| `--help`                     | Optional, Flag | Show the help message and exit.                                                                                     |

### Usage Examples

```bash theme={"system"}
# List all clusters (using environment variables)
paradime run gcp-dataproc-list-clusters

# With explicit credentials
paradime run gcp-dataproc-list-clusters \
  --service-account-key-file "/path/to/service-account-key.json" \
  --project "your-gcp-project-id" \
  --location "us-central1"
```

## Environment Variable Reference

| Environment Variable           | Description                                    |
| ------------------------------ | ---------------------------------------------- |
| `GCP_SERVICE_ACCOUNT_KEY_FILE` | Path to your GCP service account JSON key file |
| `GCP_PROJECT_ID`               | Your GCP project ID                            |
| `GCP_LOCATION`                 | GCP region (e.g. 'us-central1')                |
| `GCP_DATAPROC_TRIGGER_WAIT`    | Whether to wait for job completion             |
| `GCP_DATAPROC_TRIGGER_TIMEOUT` | Maximum time to wait for completion (minutes)  |

## Workflow Integration

These GCP Dataproc CLI commands are designed to integrate seamlessly into your data pipeline workflows. For more information about integrating them into your Paradime workflows, see the [Bolt Schedules documentation.](/products/bolt/index)


## Related topics

- [GCP Datastream CLI](/developers/paradime-cli/gcp-datastream-cli.md)
- [GCP Dataflow CLI](/developers/paradime-cli/gcp-dataflow-cli.md)
- [GCP Cloud Function CLI](/developers/paradime-cli/gcp-cloud-function-cli.md)
- [GCP Cloud Run CLI](/developers/paradime-cli/gcp-cloud-run-cli.md)
- [GCP BigQuery Data Transfer CLI](/developers/paradime-cli/gcp-bigquery-transfer-cli.md)