dlt Source and Destination Credentials via Environment Variables

Setting Environment Variables in dlt

In this guide, you'll learn how to configure credentials and secrets for your dlt pipelines using environment variables. By the end of this 5-minute tutorial, you'll understand the naming convention, how to translate your secrets.toml keys into environment variable names, and how to apply this across common use cases.

---

How dlt Reads Configuration

dlt retrieves configuration and secrets from multiple locations in the following order of priority:

1. Environment Variables — highest priority; if a value is found here, dlt stops searching

2. secrets.toml — for secrets like API keys and passwords

3. config.toml — for non-sensitive configuration values

4. Default values — defined in your pipeline code

---

Naming Convention

Environment variables follow a specific naming convention that maps directly to the structure of your secrets.toml or config.toml files.

The rules are simple:

• All letters are capitalized

• Nested sections (dots . in TOML) are replaced with double underscores __

For example, the following secrets.toml entry:

[destination.bigquery.credentials]
project_id = "my_project"
private_key = "my_key"
client_email = "[email protected]"

Translates to these environment variables:

DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="my_key"
DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="[email protected]"

Quick rule: Replace every . with __ and capitalize everything. sources.pipedrive.pipedrive_api_key becomes SOURCES__PIPEDRIVE__PIPEDRIVE_API_KEY.

---

Setting Environment Variables

On Paradime

In Paradime, navigate to your environment variable settings in your workspace and set the key-value pair for your Code IDE and Bolt sections.

On Linux / macOS

Export variables directly in your terminal session:

export DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
export DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="my_key"
export DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="[email protected]"

On Windows (Command Prompt)

set DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID=my_project

In a .env File (Local Development)

For local development, create a .env file in your project root and use python-dotenv to load it automatically:

# .env
DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="my_key"
DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="[email protected]"

Then load it in your pipeline script:

from dotenv import load_dotenv
load_dotenv()

import dlt
# Your pipeline code here

{% hint style="warning" %} Never commit your .env file to version control. Add it to your .gitignore to keep secrets out of your repository. {% endhint %}

---

Examples

Example 1: Source Credentials (Pipedrive)

The equivalent secrets.toml entry:

[sources.pipedrive]
pipedrive_api_key = "abc123"

As an environment variable:

export SOURCES__PIPEDRIVE__PIPEDRIVE_API_KEY="abc123"

---

Example 2: Destination Credentials (BigQuery)

The equivalent secrets.toml entry:

[destination.bigquery.credentials]
project_id = "my_project"
private_key = "-----BEGIN RSA PRIVATE KEY-----\n..."
client_email = "[email protected]"

As environment variables:

export DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
export DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n..."
export DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="[email protected]"

---

Example 3: Mixing Environment Variables with Existing Variables

If you already have credentials stored under different environment variable names (for example, from another tool), you can map them to the dlt naming convention in your pipeline script:

import dlt
import os

# Map existing environment variables to dlt's expected naming convention
os.environ["DESTINATION__CREDENTIALS__CLIENT_EMAIL"] = os.environ.get("BIGQUERY_CLIENT_EMAIL")
os.environ["DESTINATION__CREDENTIALS__PRIVATE_KEY"] = os.environ.get("BIGQUERY_PRIVATE_KEY")
os.environ["DESTINATION__CREDENTIALS__PROJECT_ID"] = os.environ.get("BIGQUERY_PROJECT_ID")

# For source credentials
dlt.secrets["sources.credentials.client_email"] = os.environ.get("SHEETS_CLIENT_EMAIL")
dlt.secrets["sources.credentials.private_key"] = os.environ.get("SHEETS_PRIVATE_KEY")
dlt.secrets["sources.credentials.project_id"] = os.environ.get("SHEETS_PROJECT_ID")

---

Example 4: Filesystem Source (AWS S3)

The equivalent secrets.toml entry:

[sources.filesystem.credentials]
aws_access_key_id = "AKIAIOSFODNN7EXAMPLE"
aws_secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

As environment variables:

export SOURCES__FILESYSTEM__CREDENTIALS__AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export SOURCES__FILESYSTEM__CREDENTIALS__AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

---

How dlt Searches for a Value

When a configuration value is missing, dlt logs the exact lookup path it tried. This makes it easy to diagnose issues. For example, if a password field is missing for a Postgres destination, dlt will report something like:

In Environment Variables key DESTINATION__POSTGRES__CREDENTIALS__PASSWORD was not found.
In Environment Variables key DESTINATION__CREDENTIALS__PASSWORD was not found.
In Environment Variables key CREDENTIALS__PASSWORD was not found.
In secrets.toml key destination.postgres.credentials.password was not found.

Use these messages to confirm the exact environment variable name your pipeline expects.

---

Best Practices

• Use environment variables in CI/CD and production to avoid storing secrets in files that could be accidentally committed to version control.

• Rely on dlt's error messages when debugging missing credentials — they show the exact keys and providers that were checked.

• Don't mix providers unnecessarily — define each credential in one place to avoid confusion over which value dlt is picking up.