# Fivetran CLI The Paradime SDK provides CLI commands to interact with Fivetran, allowing you to trigger connector syncs and list available connectors with real-time monitoring capabilities. {% hint style="warning" %} πŸ”‘ **API Access Required** You will need a Fivetran API key and secret to use these commands. You can create them in your Fivetran account settings: [API Keys](https://fivetran.com/docs/rest-api/getting-started#generateapikeyandapisecret). See our docs [here](https://docs.paradime.io/app-help/documentation/integrations/etl/fivetran). **πŸ”§ Connector Configuration** * Ensure connector setup is complete and not in a "broken" state. * Ensure source system credentials are valid. * Verify destination warehouse permissions are correct. * Unpause any paused connectors you want to sync. {% endhint %} {% hint style="success" %} We recommend you set the Fivetran connector sync schedule to **manual**. This ensures Fivetran runs when you expect, and syncs are ochestrated only via Paradime.
How to Set a Fivetran Sync to Manual This cURL command updates a Fivetran connection to use manual scheduling instead of automatic syncs. **Request Body Fields** * **`schedule_type: "manual"`** - Sets sync schedule to manual (no automatic syncs) * **`run_setup_tests: false`** - Skips connection validation tests during update **Prerequisites** 1. **API Key & Secret**: Generate from Fivetran account settings 2. **Connector ID**: Found in Fivetran dashboard URL for the connector or using the [#list-fivetran-connectors](#list-fivetran-connectors "mention") CLI command ```python curl -X PATCH \ https://api.fivetran.com/v1/connections/ \ -u ":" \ -H "Content-Type: application/json" \ -d '{"schedule_type": "manual", "run_setup_tests": false}' ``` **Expected Response** Success (200 OK) ```json { "code": "Success", "data": { "id": "abc123", "schedule_type": "manual", "sync_frequency": 0, "paused": false, ... } } ``` **Validate in the Fivetran Dashboard UI**\ After updating your connection to manual schedule, verify the changes in the Fivetran dashboard: 1. Navigate to your Connector Dashboard 2. Locate your connector in the list 3. Check the top right corner of the connector details page You should see: * "Enabled" * "Sync Scheduleed manually" This confirms your connection is active and set to manual scheduling mode.
{% endhint %} ## Sync Fivetran Connectors Trigger syncs for one or more Fivetran connectors with real-time progress monitoring and comprehensive status reporting. ### CLI Command ```bash paradime run fivetran-sync ``` #### Options | Flag | Type | Description | | ---------------------------------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `--connector-id` | Required, TEXT (multiple) | The ID of the Fivetran connector(s) you want to sync. Can specify multiple connectors by repeating the flag. | | `--api-key` | Required, TEXT | Your Fivetran API key. Can be set via `FIVETRAN_API_KEY` environment variable. | | `--api-secret` | Required, TEXT | Your Fivetran API secret. Can be set via `FIVETRAN_API_SECRET` environment variable. | | `--force` | Optional, Flag | Force restart ongoing syncs and attempt to sync paused connectors. Default: `False`. | | `--wait-for-completion` / `--no-wait-for-completion` | Optional, Flag | Wait for sync jobs to complete before returning. Shows real-time progress and final status. Default: `--wait-for-completion`. | | `--timeout-minutes` | Optional, INTEGER | Maximum time to wait for sync completion (in minutes). Only used with `--wait-for-completion`. Default: 1440 (24 hours). | {% hint style="info" %} **Recommended Setup** For security and convenience, set your Fivetran credentials as environment variables: ```bash FIVETRAN_API_KEY="your_api_key" FIVETRAN_API_SECRET="your_api_secret" ``` {% endhint %} ## Connector Sync Examples ### Sync a single connector ```bash # Using environment variables (recommended) paradime run fivetran-sync --connector-id "connector_abc123" # Or with explicit flags paradime run fivetran-sync \ --api-key "your_api_key" \ --api-secret "your_api_secret" \ --connector-id "connector_abc123" ``` ### Sync multiple connectors in parallel ```bash # Using environment variables (recommended) paradime run fivetran-sync \ --connector-id "sales_data_connector" \ --connector-id "marketing_connector" \ --connector-id "support_tickets" # Or with explicit flags paradime run fivetran-sync \ --api-key "your_api_key" \ --api-secret "your_api_secret" \ --connector-id "sales_data_connector" \ --connector-id "marketing_connector" \ --connector-id "support_tickets" ``` ### Force sync paused connectors ```bash # Force sync paused or ongoing connectors paradime run fivetran-sync \ --connector-id "paused_connector" \ --force ``` ### Trigger sync without waiting for completion ```bash # Start sync and return immediately without monitoring paradime run fivetran-sync \ --connector-id "large_dataset_connector" \ --no-wait-for-completion ``` {% hint style="success" %} ### Sample Output ``` Starting sync for 3 Fivetran connector(s)... ============================================================ πŸš€ TRIGGERING FIVETRAN CONNECTORS ============================================================ [1/3] πŸ”Œ sales_data_connector ---------------------------------------- [2/3] πŸ”Œ marketing_connector ---------------------------------------- [3/3] πŸ”Œ support_tickets ---------------------------------------- ============================================================ ⚑ LIVE PROGRESS ============================================================ 19:15:32 πŸ” [sales_data_connector] Checking connector status... 19:15:32 πŸ” [marketing_connector] Checking connector status... 19:15:32 πŸ” [support_tickets] Checking connector status... 19:15:33 πŸ“Š [sales_data_connector] State: scheduled | Setup: connected 19:15:33 πŸš€ [sales_data_connector] Triggering sync... 19:15:33 πŸ“Š [marketing_connector] State: paused | Setup: broken 19:15:33 ⚠️ [marketing_connector] Connector is paused - use --force to override 19:15:33 πŸ“Š [support_tickets] State: scheduled | Setup: connected 19:15:33 πŸš€ [support_tickets] Triggering sync... 19:15:34 πŸ”— [sales_data_connector] Dashboard: https://fivetran.com/dashboard/connections/sales_data_connector 19:15:34 ⏳ [sales_data_connector] Monitoring sync progress... 19:15:34 πŸ”— [support_tickets] Dashboard: https://fivetran.com/dashboard/connections/support_tickets 19:15:34 ⏳ [support_tickets] Monitoring sync progress... 19:15:34 ⏳ [sales_data_connector] Waiting to start... (0m 0s elapsed) 19:15:34 ⏳ [support_tickets] Waiting to start... (0m 0s elapsed) 19:16:40 πŸ”„ [support_tickets] Sync started 19:16:45 πŸ”„ [sales_data_connector] Sync started 19:17:13 πŸ”„ [sales_data_connector] Syncing... (1m 6s elapsed) 19:17:13 πŸ”„ [support_tickets] Syncing... (1m 6s elapsed) 19:17:18 βœ… [sales_data_connector] Completed successfully (1m 11s) 19:17:40 βœ… [support_tickets] Completed successfully (1m 33s) ================================================================================ πŸ“Š SYNC RESULTS ================================================================================ CONNECTOR STATUS DASHBOARD ------------------------- ---------- --------------------------------------------- sales_data_connector βœ… SUCCESS https://fivetran.com/dashboard/connections/sales_data_connector marketing_connector ⚠️ PAUSED https://fivetran.com/dashboard/connections/marketing_connector support_tickets βœ… SUCCESS https://fivetran.com/dashboard/connections/support_tickets ================================================================================ ``` {% endhint %} ## List Fivetran Connectors List all available Fivetran connectors with their IDs, status, and configuration details. ### CLI Command ```bash paradime run fivetran-list-connectors ``` #### Options | Flag | Type | Description | | -------------- | -------------- | ---------------------------------------------------------------------------------------- | | `--api-key` | Required, TEXT | Your Fivetran API key. Can be set via `FIVETRAN_API_KEY` environment variable. | | `--api-secret` | Required, TEXT | Your Fivetran API secret. Can be set via `FIVETRAN_API_SECRET` environment variable. | | `--group-id` | Optional, TEXT | Filter connectors by group ID. If not specified, lists all connectors across all groups. | ### Usage Examples ```bash # List all connectors (using environment variables) paradime run fivetran-list-connectors # List connectors for a specific group paradime run fivetran-list-connectors --group-id "group_abc123" # With explicit credentials paradime run fivetran-list-connectors \ --api-key "your_api_key" \ --api-secret "your_api_secret" ``` {% hint style="success" %} **Sample Output** ``` πŸ” Listing all connectors ================================================================================ πŸ“‹ FOUND 5 CONNECTOR(S) ================================================================================ [1/5] πŸ”Œ sales_postgres_prod -------------------------------------------------- Service: postgres Schema: sales_data βœ… Sync State: scheduled βœ… Setup State: connected βœ… Last Success: 2024-01-15T10:30:00.000000Z πŸ”— Dashboard: https://fivetran.com/dashboard/connections/sales_postgres_prod [2/5] πŸ”Œ marketing_hubspot -------------------------------------------------- Service: hubspot Schema: marketing_hub πŸ”„ Sync State: syncing βœ… Setup State: connected βœ… Last Success: 2024-01-15T09:15:00.000000Z πŸ”— Dashboard: https://fivetran.com/dashboard/connections/marketing_hubspot [3/5] πŸ”Œ support_zendesk -------------------------------------------------- Service: zendesk Schema: support_tickets ⏸️ Sync State: paused ⚠️ Setup State: incomplete ❌ Last Failure: 2024-01-15T08:45:00.000000Z πŸ”— Dashboard: https://fivetran.com/dashboard/connections/support_zendesk ================================================================================ ``` {% endhint %} ## Connector Status Reference Understanding the status indicators in the output: ### Sync States * βœ… scheduled/rescheduled: Connector is ready for sync or next sync is scheduled * πŸ”„ syncing: Connector is currently performing a sync * ⏸️ paused: Connector is paused and won't sync automatically * ❓ unknown: Status could not be determined ### Setup States * βœ… connected: Connector is properly configured and ready * ❌ broken: Connector configuration has issues that need attention * ⚠️ incomplete: Connector setup is not finished ## Important Notes * Parallel Execution: Multiple connectors sync simultaneously for efficiency * Real-time Monitoring: Live progress updates show sync status with timestamps * Dashboard Integration: Direct links to Fivetran dashboard for each connector * Paused Connector Handling: Use `--force` flag to attempt syncing paused connectors * Long-running Operations: Default 24-hour timeout accommodates large data syncs * Connector Identification: Use the exact connector ID from your Fivetran dashboard ## Environment Variable Reference | Environment Variable | Description | | --------------------- | ------------------------ | | `FIVETRAN_API_KEY` | Your Fivetran API key | | `FIVETRAN_API_SECRET` | Your Fivetran API secret | ## Troubleshooting
Paused Connectors If connectors are paused, the sync will be skipped unless you use the `--force` flag: ```bash paradime run fivetran-sync --connector-id "paused_connector" --force ```
Broken Setup State Connectors with broken setup need manual intervention in the Fivetran dashboard before they can sync successfully.
Network Timeouts For very large sources, consider increasing the timeout: ```bash paradime run fivetran-sync --connector-id "big_data_connector" --timeout-minutes 2880 # 48 hours ```
Finding Connector IDs Use the list command to discover available connector IDs: ```bash paradime run fivetran-list-connectors ```
## Workflow Integration These Fivetran CLI commands are designed to integrate seamlessly into your data pipeline workflows. Common use cases include: 1. **Pre-dbt**β„’ **Sync**: Trigger Fivetran syncs before running dbtβ„’ transformations 2. **Scheduled Data Ingestion**: Automate regular sync cycles for critical data sources 3. **Event-driven Syncs**: Trigger syncs based on external events or schedules 4. **Multi-source Orchestration**: Coordinate syncs across multiple data connectors For more information about integrating Fivetran syncs into your Paradime workflows, see the [Bolt Schedules documentation.](https://docs.paradime.io/app-help/documentation/bolt)