Airbyte CLI - Paradime

The Paradime SDK provides CLI commands to interact with Airbyte, allowing you to trigger sync and reset jobs for connections with real-time monitoring capabilities. Works with both Airbyte Cloud and self-hosted Airbyte Server.

🔑 API Access RequiredYou will need Airbyte credentials to use these commands:

Airbyte Cloud: Client ID and Client Secret from your Airbyte Cloud account settings. More here
Airbyte Server: API Key and API Secret from your self-hosted Airbyte instance. More here

🔧 Connector Configuration

Ensure the connection exists and is enabled in the Airbyte dashboard.
Confirm that both the source and destination connectors have valid credentials and can connect.
Make sure the credentials used have the necessary permissions to run syncs and access resources.
Run manual tests or sample syncs in the Airbyte UI to validate configuration prior to large or automated runs.

Trigger Airbyte Sync/Reset

Trigger sync or reset jobs for one or more Airbyte connections with real-time progress monitoring and comprehensive status reporting.

CLI Command

paradime run airbyte-sync

Options

Flag	Type	Description
`--connection-id`	Required, TEXT (multiple)	The ID of the Airbyte connection(s) you want to run jobs for. Can specify multiple connections by repeating the flag.
`--job-type`	Required, CHOICE	Type of job to run. Choices: `sync`, `reset`.
`--client-id`	Required, TEXT	Your Airbyte client ID (Cloud) or API key (Server). Can be set via `AIRBYTE_CLIENT_ID` environment variable.
`--client-secret`	Required, TEXT	Your Airbyte client secret (Cloud) or API secret (Server). Can be set via `AIRBYTE_CLIENT_SECRET` environment variable.
`--base-url`	Optional, TEXT	Airbyte API base URL. Default: `https://api.airbyte.com/v1` (Cloud). Can be set via `AIRBYTE_BASE_URL` environment variable.
`--use-server-auth`	Optional, Flag	Use authentication for self-hosted Airbyte Server (instead of OAuth for Cloud). Default: `False`. Can be set via `USE_SERVER_AUTH` environment variable.
`--workspace-id`	Optional, TEXT	Optional workspace ID to scope the job.
`--wait-for-completion`	Optional, Flag	Wait for jobs to complete before returning. Shows real-time progress and final status. Default: `True`.
`--timeout-minutes`	Optional, INTEGER	Maximum time to wait for job completion (in minutes). Only used with `--wait-for-completion`. Default: 1440 (24 hours).

Recommended SetupFor security and convenience, set your Airbyte credentials as environment variables:For Airbyte Cloud:

AIRBYTE_CLIENT_ID="your_client_id"
AIRBYTE_CLIENT_SECRET="your_client_secret"

For Airbyte Server:

AIRBYTE_CLIENT_ID="your_api_key"
AIRBYTE_CLIENT_SECRET="your_api_secret"
AIRBYTE_BASE_URL="http://localhost:8001/api/v1"

Job Examples

Sync a single connection (Airbyte Cloud)

# Using environment variables (recommended)
paradime run airbyte-sync \
  --connection-id "e3b2eda2-44af-4e32-b1b9-8b8c9e2d1234" \
  --job-type sync

Sync multiple connections in parallel

# Using environment variables (recommended)
paradime run airbyte-sync \
  --connection-id "sales_connection_id" \
  --connection-id "marketing_connection_id" \
  --connection-id "support_connection_id" \
  --job-type sync

Trigger job without waiting for completion

# Start job and return immediately without monitoring
paradime run airbyte-sync \
  --connection-id "large_dataset_connection" \
  --job-type sync \
  --no-wait-for-completion

Custom workspace and timeout

# With specific workspace and extended timeout
paradime run airbyte-sync \
  --connection-id "connection_id" \
  --job-type sync \
  --workspace-id "workspace_123" \
  --timeout-minutes 2880  # 48 hours

Sample Output

Starting sync jobs for 3 Airbyte connection(s)...

============================================================
🚀 TRIGGERING AIRBYTE JOBS
============================================================

[1/3] 🔌 sales_postgres_connection
----------------------------------------

[2/3] 🔌 marketing_api_connection
----------------------------------------

[3/3] 🔌 support_webhook_connection
----------------------------------------

============================================================
⚡ LIVE PROGRESS
============================================================
19:15:32 🔍 [sales_postgres_connection] Checking connection status...
19:15:32 🔍 [marketing_api_connection] Checking connection status...
19:15:32 🔍 [support_webhook_connection] Checking connection status...
19:15:33 📊 [sales_postgres_connection] Status: active
19:15:33 🚀 [sales_postgres_connection] Triggering sync job...
19:15:33 📊 [marketing_api_connection] Status: active
19:15:33 🚀 [marketing_api_connection] Triggering sync job...
19:15:33 📊 [support_webhook_connection] Status: inactive
19:15:33 ⚠️  [support_webhook_connection] Warning: Connection is inactive
19:15:34 ✅ [sales_postgres_connection] Job triggered successfully (ID: job_abc123)
19:15:34 ⏳ [sales_postgres_connection] Monitoring job progress...
19:15:34 ✅ [marketing_api_connection] Job triggered successfully (ID: job_def456)
19:15:34 ⏳ [marketing_api_connection] Monitoring job progress...
19:15:35 ⏳ [sales_postgres_connection] Job pending... (0m 1s elapsed)
19:15:35 ⏳ [marketing_api_connection] Job pending... (0m 1s elapsed)
19:16:40 🔄 [marketing_api_connection] Job running... (1m 5s elapsed)
19:16:45 🔄 [sales_postgres_connection] Job running... (1m 10s elapsed)
19:17:18 ✅ [sales_postgres_connection] Job completed successfully (1m 43s)
19:17:40 ✅ [marketing_api_connection] Job completed successfully (2m 5s)

================================================================================
📊 JOB RESULTS
================================================================================
CONNECTION                STATUS     JOB TYPE
------------------------- ---------- ----------
sales_postgres_connection ✅ SUCCESS  SYNC
marketing_api_connection  ✅ SUCCESS  SYNC
support_webhook_connection⚠️ INCOMPLETE SYNC
================================================================================

List Airbyte Connections

List all available Airbyte connections with their IDs, status, and configuration details.

CLI Command

paradime run airbyte-list-connections

Options

Flag	Type	Description
`--client-id`	Required, TEXT	Your Airbyte client ID (Cloud) or API key (Server). Can be set via `AIRBYTE_CLIENT_ID` environment variable.
`--client-secret`	Required, TEXT	Your Airbyte client secret (Cloud) or API secret (Server). Can be set via `AIRBYTE_CLIENT_SECRET` environment variable.
`--base-url`	Optional, TEXT	Airbyte API base URL. Default: `https://api.airbyte.com/v1` (Cloud). Can be set via `AIRBYTE_BASE_URL` environment variable.
`--use-server-auth`	Optional, Flag	Use authentication for self-hosted Airbyte Server (instead of OAuth for Cloud). Default: `False`.
`--workspace-id`	Optional, TEXT	Filter connections by workspace ID. If not specified, lists all connections across all workspaces.

Usage Examples

# List all connections (Airbyte Cloud, using environment variables)
paradime run airbyte-list-connections

# List connections for a specific workspace
paradime run airbyte-list-connections --workspace-id "workspace_abc123"

Sample Output

🔍 Listing all connections

================================================================================
📋 FOUND 5 CONNECTION(S)
================================================================================

[1/5] 🔌 e3b2eda2-44af-4e32-b1b9-8b8c9e2d1234
--------------------------------------------------
   Name: Sales PostgreSQL
   ✅ Status: active
   Source ID: source_postgres_abc123
   Destination ID: dest_warehouse_def456

[2/5] 🔌 f4c3feb3-55bg-5e43-c2c0-9c9d0f3e2345
--------------------------------------------------
   Name: Marketing API
   ✅ Status: active
   Source ID: source_api_ghi789
   Destination ID: dest_warehouse_def456

[3/5] 🔌 a1b2cda1-33de-4d21-a1a8-7a7b8d1c1234
--------------------------------------------------
   Name: Support Webhooks
   ⏸️ Status: inactive
   Source ID: source_webhook_jkl012
   Destination ID: dest_warehouse_def456

[4/5] 🔌 b2c3edb2-44ef-5e32-b2b9-8c8d9f2d2345
--------------------------------------------------
   Name: Analytics Events
   ✅ Status: active
   Source ID: source_events_mno345
   Destination ID: dest_warehouse_def456

[5/5] 🔌 c3d4fec3-55fg-6f43-c3c0-9d9e0g3f3456
--------------------------------------------------
   Name: Customer Data
   ❌ Status: deprecated
   Source ID: source_crm_pqr678
   Destination ID: dest_warehouse_def456

================================================================================

Job Status Reference

Understanding the status indicators in the output:

Job Status

✅ succeeded: Job completed successfully
❌ failed: Job failed and needs attention
🚫 cancelled: Job was cancelled before completion
⚠️ incomplete: Job finished but may be incomplete
🔄 running: Job is currently executing
⏳ pending: Job is queued and waiting to start

Connection Status

✅ active: Connection is enabled and ready for jobs
⏸️ inactive: Connection is disabled
❌ deprecated: Connection is deprecated and should be updated

Platform Support

Airbyte Cloud

Authentication: OAuth 2.0 with client credentials
Base URL: https://api.airbyte.com/v1
Setup: Get credentials from Airbyte Cloud Settings

Airbyte Server (Self-hosted)

Authentication: API key and secret
Base URL: Custom (e.g., http://localhost:8001/api/v1)
Setup: Generate API credentials in your Airbyte Server admin panel

Important Notes

Parallel Execution: Multiple connections process jobs simultaneously for efficiency
Real-time Monitoring: Live progress updates show job status with timestamps
Platform Flexibility: Works with both Cloud and self-hosted deployments
Job Types: Supports both sync (incremental) and reset (full refresh) operations
Long-running Operations: Default 24-hour timeout accommodates large data syncs
Connection Identification: Use exact connection IDs from your Airbyte dashboard

Environment Variable Reference

Environment Variable	Description	Platform
`AIRBYTE_CLIENT_ID`	Your client ID or API key	Both
`AIRBYTE_CLIENT_SECRET`	Your client secret or API secret	Both
`AIRBYTE_BASE_URL`	Custom API base URL	Server

Troubleshooting Common Issues

Inactive Connections

If connections are inactive, jobs may fail. Check connection status in your Airbyte dashboard and ensure sources and destinations are properly configured.

Authentication Errors

For authentication issues:

Cloud: Verify client ID and secret in Airbyte Cloud settings
Server: Verify API key and secret in your server admin panel

Network Timeouts

For very large datasets, consider increasing the timeout:

paradime run airbyte-sync \
  --connection-id "big_data_connection" \
  --job-type sync \
  --timeout-minutes 2880  # 48 hours

Finding Connection IDs

Use the list command to discover available connection IDs:

paradime run airbyte-list-connections

Platform-specific Issues

Airbyte Cloud:

Ensure you’re using OAuth client credentials, not personal access tokens
Check your Airbyte Cloud subscription limits

Airbyte Server:

Verify the base URL is accessible from your environment
Ensure API access is enabled in your server configuration
Check server logs for detailed error messages

Workflow Integration

These Fivetran CLI commands are designed to integrate seamlessly into your data pipeline workflows. Common use cases include:

Pre-dbt™ Sync: Trigger Airbyte syncs before running dbt™ or other transformations.
Scheduled Data Ingestion: Automate regular sync cycles for critical data sources.
Event-driven Syncs: Trigger syncs based on external events or schedules
Multi-source Orchestration: Coordinate syncs across multiple data connectors

For more information about integrating Airbyte syncs into your Paradime workflows, see the Bolt Schedules documentation.

API Rate Limits

Be aware of API rate limits when running multiple concurrent jobs:

Airbyte Cloud: Respects standard API rate limits
Airbyte Server: Depends on your server configuration and resources

The CLI includes automatic retry logic with exponential backoff to handle temporary rate limit issues.

​Trigger Airbyte Sync/Reset

​CLI Command

​Options

​Job Examples

​Sync a single connection (Airbyte Cloud)

​Sync multiple connections in parallel

​Trigger job without waiting for completion

​Custom workspace and timeout

​List Airbyte Connections

​CLI Command

​Options

​Usage Examples

​Job Status Reference

​Job Status

​Connection Status

​Platform Support

​Airbyte Cloud

​Airbyte Server (Self-hosted)

​Important Notes

​Environment Variable Reference

​Troubleshooting Common Issues

​Inactive Connections

​Authentication Errors

​Network Timeouts

​Finding Connection IDs

​Platform-specific Issues

​Workflow Integration

​API Rate Limits

Trigger Airbyte Sync/Reset

CLI Command

Options

Job Examples

Sync a single connection (Airbyte Cloud)

Sync multiple connections in parallel

Trigger job without waiting for completion

Custom workspace and timeout

List Airbyte Connections

CLI Command

Options

Usage Examples

Job Status Reference

Job Status

Connection Status

Platform Support

Airbyte Cloud

Airbyte Server (Self-hosted)

Important Notes

Environment Variable Reference

Troubleshooting Common Issues

Inactive Connections

Authentication Errors

Network Timeouts

Finding Connection IDs

Platform-specific Issues

Workflow Integration

API Rate Limits