dlt Source and Destination Credentials via Environment Variables

Setting Environment Variables in dlt

In this guide, you’ll learn how to configure credentials and secrets for your dlt pipelines using environment variables. By the end of this 5-minute tutorial, you’ll understand the naming convention, how to translate your secrets.toml keys into environment variable names, and how to apply this across common use cases.

How dlt Reads Configuration

dlt retrieves configuration and secrets from multiple locations in the following order of priority:

Environment Variables — highest priority; if a value is found here, dlt stops searching
secrets.toml — for secrets like API keys and passwords
config.toml — for non-sensitive configuration values
Default values — defined in your pipeline code

Naming Convention

Environment variables follow a specific naming convention that maps directly to the structure of your secrets.toml or config.toml files. The rules are simple:

All letters are capitalized
Nested sections (dots . in TOML) are replaced with double underscores __

For example, the following secrets.toml entry:

[destination.bigquery.credentials]
project_id = "my_project"
private_key = "my_key"
client_email = "my_email@project.iam.gserviceaccount.com"

Translates to these environment variables:

DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="my_key"
DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="my_email@project.iam.gserviceaccount.com"

Quick rule: Replace every . with __ and capitalize everything. sources.pipedrive.pipedrive_api_key becomes SOURCES__PIPEDRIVE__PIPEDRIVE_API_KEY.

Setting Environment Variables

On Paradime

In Paradime, navigate to your environment variable settings in your workspace and set the key-value pair for your Code IDE and Bolt sections.

On Linux / macOS

Export variables directly in your terminal session:

export DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
export DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="my_key"
export DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="my_email@project.iam.gserviceaccount.com"

On Windows (Command Prompt)

set DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID=my_project

In a `.env` File (Local Development)

For local development, create a .env file in your project root and use python-dotenv to load it automatically:

# .env
DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="my_key"
DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="my_email@project.iam.gserviceaccount.com"

Then load it in your pipeline script:

from dotenv import load_dotenv
load_dotenv()

import dlt
# Your pipeline code here

Never commit your .env file to version control. Add it to your .gitignore to keep secrets out of your repository.

Examples

Example 1: Source Credentials (Pipedrive)

The equivalent secrets.toml entry:

[sources.pipedrive]
pipedrive_api_key = "abc123"

As an environment variable:

export SOURCES__PIPEDRIVE__PIPEDRIVE_API_KEY="abc123"

Example 2: Destination Credentials (BigQuery)

The equivalent secrets.toml entry:

[destination.bigquery.credentials]
project_id = "my_project"
private_key = "-----BEGIN RSA PRIVATE KEY-----\n..."
client_email = "service@project.iam.gserviceaccount.com"

As environment variables:

export DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
export DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n..."
export DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="service@project.iam.gserviceaccount.com"

Example 3: Mixing Environment Variables with Existing Variables

If you already have credentials stored under different environment variable names (for example, from another tool), you can map them to the dlt naming convention in your pipeline script:

import dlt
import os

# Map existing environment variables to dlt's expected naming convention
os.environ["DESTINATION__CREDENTIALS__CLIENT_EMAIL"] = os.environ.get("BIGQUERY_CLIENT_EMAIL")
os.environ["DESTINATION__CREDENTIALS__PRIVATE_KEY"] = os.environ.get("BIGQUERY_PRIVATE_KEY")
os.environ["DESTINATION__CREDENTIALS__PROJECT_ID"] = os.environ.get("BIGQUERY_PROJECT_ID")

# For source credentials
dlt.secrets["sources.credentials.client_email"] = os.environ.get("SHEETS_CLIENT_EMAIL")
dlt.secrets["sources.credentials.private_key"] = os.environ.get("SHEETS_PRIVATE_KEY")
dlt.secrets["sources.credentials.project_id"] = os.environ.get("SHEETS_PROJECT_ID")

Example 4: Filesystem Source (AWS S3)

The equivalent secrets.toml entry:

[sources.filesystem.credentials]
aws_access_key_id = "AKIAIOSFODNN7EXAMPLE"
aws_secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

As environment variables:

export SOURCES__FILESYSTEM__CREDENTIALS__AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export SOURCES__FILESYSTEM__CREDENTIALS__AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

How dlt Searches for a Value

When a configuration value is missing, dlt logs the exact lookup path it tried. This makes it easy to diagnose issues. For example, if a password field is missing for a Postgres destination, dlt will report something like:

In Environment Variables key DESTINATION__POSTGRES__CREDENTIALS__PASSWORD was not found.
In Environment Variables key DESTINATION__CREDENTIALS__PASSWORD was not found.
In Environment Variables key CREDENTIALS__PASSWORD was not found.
In secrets.toml key destination.postgres.credentials.password was not found.

Use these messages to confirm the exact environment variable name your pipeline expects.

Best Practices

Use environment variables in CI/CD and production to avoid storing secrets in files that could be accidentally committed to version control.
Rely on dlt’s error messages when debugging missing credentials — they show the exact keys and providers that were checked.
Don’t mix providers unnecessarily — define each credential in one place to avoid confusion over which value dlt is picking up.

​Setting Environment Variables in dlt

​How dlt Reads Configuration

​Naming Convention

​Setting Environment Variables

​On Paradime

​On Linux / macOS

​On Windows (Command Prompt)

​In a .env File (Local Development)

​Examples

​Example 1: Source Credentials (Pipedrive)

​Example 2: Destination Credentials (BigQuery)

​Example 3: Mixing Environment Variables with Existing Variables

​Example 4: Filesystem Source (AWS S3)

​How dlt Searches for a Value

​Best Practices

Setting Environment Variables in dlt

How dlt Reads Configuration

Naming Convention

Setting Environment Variables

On Paradime

On Linux / macOS

On Windows (Command Prompt)

In a `.env` File (Local Development)

Examples

Example 1: Source Credentials (Pipedrive)

Example 2: Destination Credentials (BigQuery)

Example 3: Mixing Environment Variables with Existing Variables

Example 4: Filesystem Source (AWS S3)

How dlt Searches for a Value

Best Practices