# dlt Source and Destination Credentials via Environment Variables

## Setting Environment Variables in dlt

In this guide, you'll learn how to configure credentials and secrets for your dlt pipelines using environment variables. By the end of this 5-minute tutorial, you'll understand the naming convention, how to translate your `secrets.toml` keys into environment variable names, and how to apply this across common use cases.

***

### How dlt Reads Configuration

dlt retrieves configuration and secrets from multiple locations in the following order of priority:

1. **Environment Variables** — highest priority; if a value is found here, dlt stops searching
2. **`secrets.toml`** — for secrets like API keys and passwords
3. **`config.toml`** — for non-sensitive configuration values
4. **Default values** — defined in your pipeline code

***

### Naming Convention

Environment variables follow a specific naming convention that maps directly to the structure of your `secrets.toml` or `config.toml` files.

The rules are simple:

* All letters are **capitalized**
* Nested sections (dots `.` in TOML) are replaced with **double underscores `__`**

For example, the following `secrets.toml` entry:

```toml
[destination.bigquery.credentials]
project_id = "my_project"
private_key = "my_key"
client_email = "my_email@project.iam.gserviceaccount.com"
```

Translates to these environment variables:

```bash
DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="my_key"
DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="my_email@project.iam.gserviceaccount.com"
```

{% hint style="info" %}
**Quick rule:** Replace every `.` with `__` and capitalize everything. `sources.pipedrive.pipedrive_api_key` becomes `SOURCES__PIPEDRIVE__PIPEDRIVE_API_KEY`.
{% endhint %}

***

## Setting Environment Variables

#### On Paradime

In Paradime, navigate to your [environment variable settings](/app-help/documentation/settings/environment-variables.md) in your workspace and set the key-value pair for your **Code IDE** and **Bolt** sections.

#### On Linux / macOS

Export variables directly in your terminal session:

```bash
export DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
export DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="my_key"
export DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="my_email@project.iam.gserviceaccount.com"
```

#### On Windows (Command Prompt)

```cmd
set DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID=my_project
```

#### In a `.env` File (Local Development)

For local development, create a `.env` file in your project root and use `python-dotenv` to load it automatically:

```bash
# .env
DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="my_key"
DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="my_email@project.iam.gserviceaccount.com"
```

Then load it in your pipeline script:

```python
from dotenv import load_dotenv
load_dotenv()

import dlt
# Your pipeline code here
```

{% hint style="warning" %} **Never commit your `.env` file to version control.** Add it to your `.gitignore` to keep secrets out of your repository. {% endhint %}

***

### Examples

#### Example 1: Source Credentials (Pipedrive)

The equivalent `secrets.toml` entry:

```toml
[sources.pipedrive]
pipedrive_api_key = "abc123"
```

As an environment variable:

```bash
export SOURCES__PIPEDRIVE__PIPEDRIVE_API_KEY="abc123"
```

***

#### Example 2: Destination Credentials (BigQuery)

The equivalent `secrets.toml` entry:

```toml
[destination.bigquery.credentials]
project_id = "my_project"
private_key = "-----BEGIN RSA PRIVATE KEY-----\n..."
client_email = "service@project.iam.gserviceaccount.com"
```

As environment variables:

```bash
export DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID="my_project"
export DESTINATION__BIGQUERY__CREDENTIALS__PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n..."
export DESTINATION__BIGQUERY__CREDENTIALS__CLIENT_EMAIL="service@project.iam.gserviceaccount.com"
```

***

#### Example 3: Mixing Environment Variables with Existing Variables

If you already have credentials stored under different environment variable names (for example, from another tool), you can map them to the dlt naming convention in your pipeline script:

```python
import dlt
import os

# Map existing environment variables to dlt's expected naming convention
os.environ["DESTINATION__CREDENTIALS__CLIENT_EMAIL"] = os.environ.get("BIGQUERY_CLIENT_EMAIL")
os.environ["DESTINATION__CREDENTIALS__PRIVATE_KEY"] = os.environ.get("BIGQUERY_PRIVATE_KEY")
os.environ["DESTINATION__CREDENTIALS__PROJECT_ID"] = os.environ.get("BIGQUERY_PROJECT_ID")

# For source credentials
dlt.secrets["sources.credentials.client_email"] = os.environ.get("SHEETS_CLIENT_EMAIL")
dlt.secrets["sources.credentials.private_key"] = os.environ.get("SHEETS_PRIVATE_KEY")
dlt.secrets["sources.credentials.project_id"] = os.environ.get("SHEETS_PROJECT_ID")
```

***

#### Example 4: Filesystem Source (AWS S3)

The equivalent `secrets.toml` entry:

```toml
[sources.filesystem.credentials]
aws_access_key_id = "AKIAIOSFODNN7EXAMPLE"
aws_secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
```

As environment variables:

```bash
export SOURCES__FILESYSTEM__CREDENTIALS__AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export SOURCES__FILESYSTEM__CREDENTIALS__AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
```

***

### How dlt Searches for a Value

When a configuration value is missing, dlt logs the exact lookup path it tried. This makes it easy to diagnose issues. For example, if a `password` field is missing for a Postgres destination, dlt will report something like:

```
In Environment Variables key DESTINATION__POSTGRES__CREDENTIALS__PASSWORD was not found.
In Environment Variables key DESTINATION__CREDENTIALS__PASSWORD was not found.
In Environment Variables key CREDENTIALS__PASSWORD was not found.
In secrets.toml key destination.postgres.credentials.password was not found.
```

Use these messages to confirm the exact environment variable name your pipeline expects.

***

### Best Practices

* **Use environment variables in CI/CD and production** to avoid storing secrets in files that could be accidentally committed to version control.
* **Rely on dlt's error messages** when debugging missing credentials — they show the exact keys and providers that were checked.
* **Don't mix providers unnecessarily** — define each credential in one place to avoid confusion over which value dlt is picking up.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.paradime.io/app-help/guides/using-dlt-data-load-tool-pipelines-in-paradime/dlt-source-and-destination-credentials-via-environment-variables.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.