Custom Integration

Overview

• This feature is available with the Paradime Enterprise pack.

• Your API keys must have the Custom Integrations Admin capability.

The Customer Integration API empowers users to seamlessly extend Paradime's lineage and catalog capabilities with external applications. This powerful interface enables you to:

• Create custom integrations tailored to your specific needs

• Ingest and manage nodes from various data sources and applications

• Upload node information for Paradime to incorporate into its lineage and catalog

• Enhance data visibility by connecting Paradime with your entire data ecosystem

By leveraging this module, you can create a comprehensive, interconnected view of your data landscape, improving data governance, traceability, and insights across your organization.

Create a custom integration and upload nodes using JSON

This example uses as inputs the node_types.json and the nodes.json.

node_types.json

The below example is where we define our node types and the related attributes.

Defining color for a node:

The color can be one defined from one of the provided color palettes:

• VIOLET #827be6

• ORANGE #fb982e

• MANDY #ef6292

• TEAL #33a9a9

• GREEN #27ae60

• CORAL #FF8559

Defining icon for a node:

• icon_name can be chosen from Blueprint.js icons library

Example node_type.json

[
    {
        "node_type": "Datasource",
        "icon_name": "database",
        "color": "ORANGE"
    },
    {
        "node_type": "Chart",
        "icon_name": "pie-chart",
        "color": "TEAL"
    },
    {
        "node_type": "Dashboard",
        "icon_name": "dashboard",
        "color": "CORAL"
    }
]

nodes.json

The below example is where we define the attributes for each node and the related upstream/downstream dependencies. For all types and attributes refer to types here.

Example nodes.json

[
    {
        "name": "Finance Mart",
        "node_type": "Datasource",
        "attributes": {
            "description": "This is my first datasource"
        },
        "lineage": {
            "upstream_dependencies": [
                {
                    "table_name": "order_items"
                }
            ]
        }
    },
    {
        "name": "Daily Revenue",
        "node_type": "Chart",
        "attributes": {
            "description": "This is my first chart"
        },
        "lineage": {
            "upstream_dependencies": [
                {
                    "integration_name": "MyParadimeIntegration",
                    "node_type": "Datasource",
                    "node_name": "Finance Mart"
                }
            ],
            "downstream_dependencies": [
                {
                    "integration_name": "MyParadimeIntegration",
                    "node_type": "Dashboard",
                    "node_name": "Finance Daily Report"
                }
            ]
        }
    },
    {
        "name": "Finance Daily Report",
        "node_type": "Dashboard",
        "attributes": {
            "description": "This is my first dashboard"
        },
        "lineage": {
            "upstream_dependencies": [],
            "downstream_dependencies": []
        }
    }
]

# First party modules
import json
from pathlib import Path
from typing import List

from paradime import Paradime
from paradime.apis.custom_integration.types import Node, NodeType
from paradime.tools.pydantic import parse_obj_as

# Create a Paradime client with your API credentials
paradime = Paradime(api_endpoint="API_ENDPOINT", api_key="API_KEY", api_secret="API_SECRET")

# Load node types and nodes from JSON files
node_types = parse_obj_as(List[NodeType], json.loads(Path("node_types.json").read_text()))
nodes = parse_obj_as(List[Node], json.loads(Path("nodes.json").read_text()))

# Create a custom integration or update it if it already exists
my_integration = paradime.custom_integration.upsert(
    name="MyParadimeIntegration",
    logo_url="https://example.com/logo.png", # Optional, replace with the logo URL of the integration, or remove this line.
    node_types=node_types,
)

# Add nodes to the custom integration.
paradime.custom_integration.add_nodes(
    integration_uid=my_integration.uid,
    nodes=nodes,
)