Fivetran CLI

The Paradime SDK provides CLI commands to interact with Fivetran, allowing you to trigger connector syncs and list available connectors with real-time monitoring capabilities.

🔑 API Access Required

You will need a Fivetran API key and secret to use these commands. You can create them in your Fivetran account settings: API Keys. See our docs here.

🔧 Connector Configuration

Ensure connector setup is complete and not in a "broken" state.
Ensure source system credentials are valid.
Verify destination warehouse permissions are correct.
Unpause any paused connectors you want to sync.

We recommend you set the Fivetran connector sync schedule to manual. This ensures Fivetran runs when you expect, and syncs are ochestrated only via Paradime.

How to Set a Fivetran Sync to Manual

This cURL command updates a Fivetran connection to use manual scheduling instead of automatic syncs.

Request Body Fields

schedule_type: "manual" - Sets sync schedule to manual (no automatic syncs)
run_setup_tests: false - Skips connection validation tests during update

Prerequisites

API Key & Secret: Generate from Fivetran account settings
Connector ID: Found in Fivetran dashboard URL for the connector or using the List Fivetran Connectors CLI command

curl -X PATCH \
  https://api.fivetran.com/v1/connections/<CONNECTOR_ID> \
  -u "<FIVETRAN_API_KEY>:<FIVETRAN_API_SECRET>" \
  -H "Content-Type: application/json" \
  -d '{"schedule_type": "manual", "run_setup_tests": false}'

Expected Response

Success (200 OK)

{
  "code": "Success",
  "data": {
    "id": "abc123",
    "schedule_type": "manual",
    "sync_frequency": 0,
    "paused": false,
    ...
  }
}

Validate in the Fivetran Dashboard UI After updating your connection to manual schedule, verify the changes in the Fivetran dashboard:

Navigate to your Connector Dashboard
Locate your connector in the list
Check the top right corner of the connector details page

You should see:

"Enabled"
"Sync Scheduleed manually"

This confirms your connection is active and set to manual scheduling mode.

Sync Fivetran Connectors

Trigger syncs for one or more Fivetran connectors with real-time progress monitoring and comprehensive status reporting.

CLI Command

paradime run fivetran-sync

Options

Flag

Type

Description

--connector-id

Required, TEXT (multiple)

The ID of the Fivetran connector(s) you want to sync. Can specify multiple connectors by repeating the flag.

--api-key

Required, TEXT

Your Fivetran API key. Can be set via FIVETRAN_API_KEY environment variable.

--api-secret

Required, TEXT

Your Fivetran API secret. Can be set via FIVETRAN_API_SECRET environment variable.

--force

Optional, Flag

Force restart ongoing syncs and attempt to sync paused connectors. Default: False.

--wait-for-completion / --no-wait-for-completion

Optional, Flag

Wait for sync jobs to complete before returning. Shows real-time progress and final status. Default: --wait-for-completion.

--timeout-minutes

Optional, INTEGER

Maximum time to wait for sync completion (in minutes). Only used with --wait-for-completion. Default: 1440 (24 hours).

Recommended Setup

For security and convenience, set your Fivetran credentials as environment variables:

FIVETRAN_API_KEY="your_api_key"
FIVETRAN_API_SECRET="your_api_secret"

Connector Sync Examples

Sync a single connector

# Using environment variables (recommended)
paradime run fivetran-sync --connector-id "connector_abc123"

# Or with explicit flags
paradime run fivetran-sync \
  --api-key "your_api_key" \
  --api-secret "your_api_secret" \
  --connector-id "connector_abc123"

Sync multiple connectors in parallel

# Using environment variables (recommended)
paradime run fivetran-sync \
  --connector-id "sales_data_connector" \
  --connector-id "marketing_connector" \
  --connector-id "support_tickets"

# Or with explicit flags
paradime run fivetran-sync \
  --api-key "your_api_key" \
  --api-secret "your_api_secret" \
  --connector-id "sales_data_connector" \
  --connector-id "marketing_connector" \
  --connector-id "support_tickets"

Force sync paused connectors

# Force sync paused or ongoing connectors
paradime run fivetran-sync \
  --connector-id "paused_connector" \
  --force

Trigger sync without waiting for completion

# Start sync and return immediately without monitoring
paradime run fivetran-sync \
  --connector-id "large_dataset_connector" \
  --no-wait-for-completion

Sample Output

Starting sync for 3 Fivetran connector(s)...

============================================================
🚀 TRIGGERING FIVETRAN CONNECTORS
============================================================

[1/3] 🔌 sales_data_connector
----------------------------------------

[2/3] 🔌 marketing_connector
----------------------------------------

[3/3] 🔌 support_tickets
----------------------------------------

============================================================
⚡ LIVE PROGRESS
============================================================
19:15:32 🔍 [sales_data_connector] Checking connector status...
19:15:32 🔍 [marketing_connector] Checking connector status...
19:15:32 🔍 [support_tickets] Checking connector status...
19:15:33 📊 [sales_data_connector] State: scheduled | Setup: connected
19:15:33 🚀 [sales_data_connector] Triggering sync...
19:15:33 📊 [marketing_connector] State: paused | Setup: broken
19:15:33 ⚠️  [marketing_connector] Connector is paused - use --force to override
19:15:33 📊 [support_tickets] State: scheduled | Setup: connected
19:15:33 🚀 [support_tickets] Triggering sync...
19:15:34 🔗 [sales_data_connector] Dashboard: https://fivetran.com/dashboard/connections/sales_data_connector
19:15:34 ⏳ [sales_data_connector] Monitoring sync progress...
19:15:34 🔗 [support_tickets] Dashboard: https://fivetran.com/dashboard/connections/support_tickets
19:15:34 ⏳ [support_tickets] Monitoring sync progress...
19:15:34 ⏳ [sales_data_connector] Waiting to start... (0m 0s elapsed)
19:15:34 ⏳ [support_tickets] Waiting to start... (0m 0s elapsed)
19:16:40 🔄 [support_tickets] Sync started
19:16:45 🔄 [sales_data_connector] Sync started
19:17:13 🔄 [sales_data_connector] Syncing... (1m 6s elapsed)
19:17:13 🔄 [support_tickets] Syncing... (1m 6s elapsed)
19:17:18 ✅ [sales_data_connector] Completed successfully (1m 11s)
19:17:40 ✅ [support_tickets] Completed successfully (1m 33s)

================================================================================
📊 SYNC RESULTS
================================================================================
CONNECTOR                 STATUS     DASHBOARD
------------------------- ---------- ---------------------------------------------
sales_data_connector      ✅ SUCCESS  https://fivetran.com/dashboard/connections/sales_data_connector
marketing_connector       ⚠️ PAUSED  https://fivetran.com/dashboard/connections/marketing_connector
support_tickets           ✅ SUCCESS  https://fivetran.com/dashboard/connections/support_tickets
================================================================================

List Fivetran Connectors

List all available Fivetran connectors with their IDs, status, and configuration details.

CLI Command

paradime run fivetran-list-connectors

Options

Flag

Type

Description

--api-key

Required, TEXT

Your Fivetran API key. Can be set via FIVETRAN_API_KEY environment variable.

--api-secret

Required, TEXT

Your Fivetran API secret. Can be set via FIVETRAN_API_SECRET environment variable.

--group-id

Optional, TEXT

Filter connectors by group ID. If not specified, lists all connectors across all groups.

Usage Examples

# List all connectors (using environment variables)
paradime run fivetran-list-connectors

# List connectors for a specific group
paradime run fivetran-list-connectors --group-id "group_abc123"

# With explicit credentials
paradime run fivetran-list-connectors \
  --api-key "your_api_key" \
  --api-secret "your_api_secret"

Sample Output

🔍 Listing all connectors

================================================================================
📋 FOUND 5 CONNECTOR(S)
================================================================================

[1/5] 🔌 sales_postgres_prod
--------------------------------------------------
   Service: postgres
   Schema: sales_data
   ✅ Sync State: scheduled
   ✅ Setup State: connected
   ✅ Last Success: 2024-01-15T10:30:00.000000Z
   🔗 Dashboard: https://fivetran.com/dashboard/connections/sales_postgres_prod

[2/5] 🔌 marketing_hubspot
--------------------------------------------------
   Service: hubspot
   Schema: marketing_hub
   🔄 Sync State: syncing
   ✅ Setup State: connected
   ✅ Last Success: 2024-01-15T09:15:00.000000Z
   🔗 Dashboard: https://fivetran.com/dashboard/connections/marketing_hubspot

[3/5] 🔌 support_zendesk
--------------------------------------------------
   Service: zendesk
   Schema: support_tickets
   ⏸️ Sync State: paused
   ⚠️ Setup State: incomplete
   ❌ Last Failure: 2024-01-15T08:45:00.000000Z
   🔗 Dashboard: https://fivetran.com/dashboard/connections/support_zendesk

================================================================================

Connector Status Reference

Understanding the status indicators in the output:

Sync States

✅ scheduled/rescheduled: Connector is ready for sync or next sync is scheduled
🔄 syncing: Connector is currently performing a sync
⏸️ paused: Connector is paused and won't sync automatically
❓ unknown: Status could not be determined

Setup States

✅ connected: Connector is properly configured and ready
❌ broken: Connector configuration has issues that need attention
⚠️ incomplete: Connector setup is not finished

Important Notes

Parallel Execution: Multiple connectors sync simultaneously for efficiency
Real-time Monitoring: Live progress updates show sync status with timestamps
Dashboard Integration: Direct links to Fivetran dashboard for each connector
Paused Connector Handling: Use --force flag to attempt syncing paused connectors
Long-running Operations: Default 24-hour timeout accommodates large data syncs
Connector Identification: Use the exact connector ID from your Fivetran dashboard

Environment Variable Reference

Environment Variable

Description

FIVETRAN_API_KEY

Your Fivetran API key

FIVETRAN_API_SECRET

Your Fivetran API secret

Troubleshooting

Paused Connectors

If connectors are paused, the sync will be skipped unless you use the --force flag:

paradime run fivetran-sync --connector-id "paused_connector" --force

Broken Setup State

Connectors with broken setup need manual intervention in the Fivetran dashboard before they can sync successfully.

Network Timeouts

For very large sources, consider increasing the timeout:

paradime run fivetran-sync --connector-id "big_data_connector" --timeout-minutes 2880  # 48 hours

Finding Connector IDs

Use the list command to discover available connector IDs:

paradime run fivetran-list-connectors

Workflow Integration

These Fivetran CLI commands are designed to integrate seamlessly into your data pipeline workflows. Common use cases include:

Pre-dbt™ Sync: Trigger Fivetran syncs before running dbt™ transformations
Scheduled Data Ingestion: Automate regular sync cycles for critical data sources
Event-driven Syncs: Trigger syncs based on external events or schedules
Multi-source Orchestration: Coordinate syncs across multiple data connectors

For more information about integrating Fivetran syncs into your Paradime workflows, see the Bolt Schedules documentation.

PreviousMicrosoft Power BI CLI NextAirbyte CLI

Last updated 2 months ago

Was this helpful?