Python Scripts

Execute Python scripts in Paradime with Poetry dependency management, environment variables for secrets, and support for data libraries like Pandas and PySpark for custom processing.

Overview

Paradime provides support for executing Python scripts as part of your data pipeline. Python scripts offer flexibility and extensibility, allowing you to perform custom data processing, transformation, and analysis tasks.

Python scripts in Paradime allow you to harness the power of Python to tackle complex data operations that may not be easily achievable using built-in Paradime commands. With Python scripts, you can:

• Perform data transformation and manipulation using libraries like Pandas, NumPy, or PySpark.

• Integrate with external APIs or services to enrich your data pipeline.

• Implement custom algorithms or machine learning models.

• Perform data validation, cleansing, and quality checks.

• Generate reports, visualizations, or custom output formats.

Paradime seamlessly integrates Python scripts into your data pipeline, enabling you to leverage the vast ecosystem of Python libraries and tools to meet your specific data processing requirements.

Dependency Management

Paradime supports dependency management for Python scripts using Poetry. Poetry is a modern Python dependency management and packaging tool that simplifies the process of managing project dependencies and creating reproducible environments.

You can check our guide on how to setup Poetry environments in Paradime.

Important

If you are using Poetry, when configuring a Paradime Bolt schedule, ensure that the first command in the schedule is poetry install. This command installs the specified dependencies and creates the virtual environment for your Python script.

Storing Secrets in Environment Variables

When working with Python scripts in Paradime, you may need to handle sensitive information such as API keys, database credentials, or other secrets. It's important to store these secrets securely and avoid hardcoding them directly in your Python scripts.

Paradime provides two ways to manage environment variables for Python scripts:

• Code IDE Environment Variables: For developing and testing your Python code in the development environment

• Bolt Schedule Environment Variables: For running your Python code in production using Bolt

For detailed instructions on setting up environment variables, see:

• Code IDE Environment Variables

• Bolt Schedule Environment Variables

Accessing environment variables

In your Python script, access the secrets using the os module to retrieve the values of the environment variables. For example:

import os

api_key = os.environ['API_KEY']
db_password = os.environ['DB_PASSWORD']

The os.environ dictionary allows you to access the values of the environment variables set in your Paradime configuration.

---

Now, let's learn how to configure the Trigger Types of a Bolt Schedule.