Custom Integration

Overview

Prerequisites:

This feature is available with the Paradime Enterprise pack.
Your API keys must have the Custom Integrations Admin capability.

These examples authenticate with an account API key (api_secret="prdm_cmp_..." plus workspace_uid), which requires paradime-io 6.0.0 or later. Legacy workspace API keys (api_key + api_secret) are still supported. See Getting Started.

The Customer Integration API empowers users to seamlessly extend Paradime’s lineage and catalog capabilities with external applications. This powerful interface enables you to:

Create custom integrations tailored to your specific needs
Ingest and manage nodes from various data sources and applications
Upload node information for Paradime to incorporate into its lineage and catalog
Enhance data visibility by connecting Paradime with your entire data ecosystem

By leveraging this module, you can create a comprehensive, interconnected view of your data landscape, improving data governance, traceability, and insights across your organization. You can find examples and a template on how to get started here

Create a custom integration and upload nodes using JSON

This example uses as inputs the node_types.json and the nodes.json.

node_types.json
nodes.json

The below example is where we define our node types and the related attributes.

Defining color for a node:The color can be one defined from one of the provided color palettes:

VIOLET #827be6
ORANGE #fb982e
MANDY #ef6292
TEAL #33a9a9
GREEN #27ae60
CORAL #FF8559

Defining icon for a node:

icon_name can be chosen from Blueprint.js icons library

Example node_type.json

[
    {
        "node_type": "Datasource",
        "icon_name": "database",
        "color": "ORANGE"
    },
    {
        "node_type": "Chart",
        "icon_name": "pie-chart",
        "color": "TEAL"
    },
    {
        "node_type": "Dashboard",
        "icon_name": "dashboard",
        "color": "CORAL"
    }
]

The below example is where we define the attributes for each node and the related upstream/downstream dependencies. For all types and attributes refer to types here.

Example nodes.json

[
    {
        "name": "Finance Mart",
        "node_type": "Datasource",
        "attributes": {
            "description": "This is my first datasource"
        },
        "lineage": {
            "upstream_dependencies": [
                {
                    "table_name": "order_items"
                }
            ]
        }
    },
    {
        "name": "Daily Revenue",
        "node_type": "Chart",
        "attributes": {
            "description": "This is my first chart"
        },
        "lineage": {
            "upstream_dependencies": [
                {
                    "integration_name": "MyParadimeIntegration",
                    "node_type": "Datasource",
                    "node_name": "Finance Mart"
                }
            ],
            "downstream_dependencies": [
                {
                    "integration_name": "MyParadimeIntegration",
                    "node_type": "Dashboard",
                    "node_name": "Finance Daily Report"
                }
            ]
        }
    },
    {
        "name": "Finance Daily Report",
        "node_type": "Dashboard",
        "attributes": {
            "description": "This is my first dashboard"
        },
        "lineage": {
            "upstream_dependencies": [],
            "downstream_dependencies": []
        }
    }
]

# First party modules
import json
from pathlib import Path
from typing import List

from paradime import Paradime
from paradime.apis.custom_integration.types import Node, NodeType
from paradime.tools.pydantic import parse_obj_as

# Create a Paradime client with your API credentials
paradime = Paradime(api_endpoint="API_ENDPOINT", api_secret="prdm_cmp_...", workspace_uid="WORKSPACE_UID")

# Load node types and nodes from JSON files
node_types = parse_obj_as(List[NodeType], json.loads(Path("node_types.json").read_text()))
nodes = parse_obj_as(List[Node], json.loads(Path("nodes.json").read_text()))

# Create a custom integration or update it if it already exists
my_integration = paradime.custom_integration.upsert(
    name="MyParadimeIntegration",
    logo_url="https://example.com/logo.png", # Optional, replace with the logo URL of the integration, or remove this line.
    node_types=node_types,
)

# Add nodes to the custom integration.
paradime.custom_integration.add_nodes(
    integration_uid=my_integration.uid,
    nodes=nodes,
)

​Overview

​Create a custom integration and upload nodes using JSON

Overview

Create a custom integration and upload nodes using JSON