Custom Integration

Overview

Prerequisites:

This feature is available with the Paradime Enterprise pack.
Your API keys must have the Custom Integrations Admin capability.

These examples authenticate with an account API key (api_secret="prdm_cmp_..." plus workspace_uid), which requires paradime-io 6.0.0 or later. Legacy workspace API keys (api_key + api_secret) are still supported. See Getting Started.

The Custom Integration API empowers users to seamlessly extend Paradime’s lineage and catalog capabilities with external applications. This powerful interface enables you to:

Create custom integrations tailored to your specific needs
Ingest and manage nodes from various data sources and applications
Upload node information for Paradime to incorporate into its lineage and catalog
Enhance data visibility by connecting Paradime with your entire data ecosystem

By leveraging this module, you can create a comprehensive, interconnected view of your data landscape, improving data governance, traceability, and insights across your organization. You can find examples and a template on how to get started here

Create a custom integration and upload nodes using JSON

This example uses as inputs the node_types.json and the nodes.json.

node_types.json
nodes.json

The below example is where we define our node types and the related attributes.

Defining color for a node:The color can be one defined from one of the provided color palettes:

VIOLET #827be6
ORANGE #fb982e
MANDY #ef6292
TEAL #33a9a9
GREEN #27ae60
CORAL #FF8559
LEAF
CYAN

Defining icon for a node:

icon_name can be chosen from Blueprint.js icons library

Example node_type.json

[
    {
        "node_type": "Datasource",
        "icon_name": "database",
        "color": "ORANGE"
    },
    {
        "node_type": "Chart",
        "icon_name": "pie-chart",
        "color": "TEAL"
    },
    {
        "node_type": "Dashboard",
        "icon_name": "dashboard",
        "color": "CORAL"
    }
]

The below example is where we define the attributes for each node and the related upstream/downstream dependencies. For all types and attributes refer to types here.

Example nodes.json

[
    {
        "name": "Finance Mart",
        "node_type": "Datasource",
        "attributes": {
            "description": "This is my first datasource"
        },
        "lineage": {
            "upstream_dependencies": [
                {
                    "table_name": "order_items"
                }
            ]
        }
    },
    {
        "name": "Daily Revenue",
        "node_type": "Chart",
        "attributes": {
            "description": "This is my first chart"
        },
        "lineage": {
            "upstream_dependencies": [
                {
                    "integration_name": "MyParadimeIntegration",
                    "node_type": "Datasource",
                    "node_name": "Finance Mart"
                }
            ],
            "downstream_dependencies": [
                {
                    "integration_name": "MyParadimeIntegration",
                    "node_type": "Dashboard",
                    "node_name": "Finance Daily Report"
                }
            ]
        }
    },
    {
        "name": "Finance Daily Report",
        "node_type": "Dashboard",
        "attributes": {
            "description": "This is my first dashboard"
        },
        "lineage": {
            "upstream_dependencies": [],
            "downstream_dependencies": []
        }
    }
]

# First party modules
import json
from pathlib import Path
from typing import List

from paradime import Paradime
from paradime.apis.custom_integration.types import Node, NodeType
from paradime.tools.pydantic import parse_obj_as

# Create a Paradime client with your API credentials
paradime = Paradime(api_endpoint="API_ENDPOINT", api_secret="prdm_cmp_...", workspace_uid="WORKSPACE_UID")

# Load node types and nodes from JSON files
node_types = parse_obj_as(List[NodeType], json.loads(Path("node_types.json").read_text()))
nodes = parse_obj_as(List[Node], json.loads(Path("nodes.json").read_text()))

# Create a custom integration or update it if it already exists
my_integration = paradime.custom_integration.upsert(
    name="MyParadimeIntegration",
    logo_url="https://example.com/logo.png", # Optional, replace with the logo URL of the integration, or remove this line.
    node_types=node_types,
)

# Add nodes to the custom integration.
paradime.custom_integration.add_nodes(
    integration_uid=my_integration.uid,
    nodes=nodes,
)

Methods

All methods are called on paradime.custom_integration.

create

Creates a custom integration with the specified name, logo URL, and node types.

str

required

The name of the custom integration.

Optional[str]

required

The URL of the logo for the custom integration. If not provided (None), a default logo is used.

List[NodeType]

required

A list of NodeType objects representing the node types for the custom integration.

str

The integration UID of the created custom integration.

All arguments are keyword-only.

update

Updates a custom integration with the specified parameters. Only the parameters that are not None are updated.

str

required

The unique identifier of the integration.

Optional[str]

default:"None"

The new name of the integration.

Optional[str]

default:"None"

The new logo URL of the integration.

Optional[List[NodeType]]

default:"None"

The new list of node types for the integration. Overrides the existing node types.

Optional[bool]

default:"None"

Whether the integration should be active.

None

Returns nothing.

All arguments are keyword-only.

upsert

Upserts an integration by either updating an existing integration with the given name, or creating a new integration if it doesn’t exist.

str

required

The name of the integration.

Optional[str]

default:"None"

The URL of the integration’s logo.

List[NodeType]

required

A list of node types associated with the integration.

Integration

The upserted integration.

All arguments are keyword-only.

list_all

Retrieves a list of all custom integrations, including both active and inactive integrations. Takes no arguments.

List[Integration]

A list of Integration objects representing the custom integrations.

list_active

Retrieves a list of all active custom integrations. Takes no arguments.

List[Integration]

A list of Integration objects representing the active custom integrations.

get

Retrieves an active integration with the specified UID.

str

required

The UID of the integration to retrieve. Positional argument.

Optional[Integration]

The Integration object if found, otherwise None.

get_by_name

Retrieves an active custom integration with the specified name.

str

required

The name of the integration to retrieve. Positional argument.

Optional[Integration]

The Integration object if found, otherwise None.

add_nodes

Adds all nodes to a new snapshot in the custom integration. Internally batches the nodes across multiple requests. To add nodes in a streaming fashion, use add_nodes_to_snapshot.

str

required

The unique identifier of the integration.

List[Node]

required

The list of all nodes to be added to the integration.

None

Returns nothing.

All arguments are keyword-only.

add_nodes_to_snapshot

Adds nodes to a snapshot in the custom integration. A snapshot is a collection of nodes added to the custom integration. Nodes are added in batches, and the snapshot ID keeps track of the nodes added to the snapshot. This method is useful when adding a large number of nodes in batches.

str

required

The UID of the custom integration.

List[Node]

required

The list of nodes to be added to the snapshot.

bool

required

Indicates whether the snapshot has more nodes. Set to False when adding the last batch of nodes.

Optional[int]

default:"None"

The ID of the snapshot. If not provided, a new snapshot is created. If provided, the nodes are added to the existing snapshot.

int

The ID of the snapshot after adding the nodes.

All arguments are keyword-only.

API Keys & Access

GraphQL API

Python SDK

Paradime CLI

Webhooks

Virtual Environments

Custom Integration

Overview

Create a custom integration and upload nodes using JSON

Methods

create

update

upsert

list_all

list_active

get

get_by_name

add_nodes

add_nodes_to_snapshot

​Overview

​Create a custom integration and upload nodes using JSON

​Methods

​create

​update

​upsert

​list_all

​list_active

​get

​get_by_name

​add_nodes

​add_nodes_to_snapshot

Overview

Create a custom integration and upload nodes using JSON

Methods

create

update

upsert

list_all

list_active

get

get_by_name

add_nodes

add_nodes_to_snapshot