Snowflake cost connection

IP RESTRICTIONSMake sure to allow traffic from one of the Paradime IPs in your firewall depending on the data location selected.See also: Paradime IP addresses.

1. Snowflake setup guide

The Radar connection form has no role or warehouse field. Paradime signs in as paradime_cost_user and uses that user’s default role and default warehouse. If the default role isn’t paradime_cost_role, the connection still saves, but Snowflake runs every Radar query as PUBLIC and no cost data arrives. Create a Snowflake role and user sets both defaults, and Check the Paradime user checks them.

Create a Snowflake role and user

Generate a key pair for the Paradime user, and replace <your_public_key_here> with the public key, without the BEGIN and END lines. See Snowflake’s guide to generating an encrypted or unencrypted key pair.

-- 1.### create role and user

use role useradmin;

create role if not exists paradime_cost_role comment = 'Used by paradime.io for cost analytics';
grant role paradime_cost_role to role sysadmin;

create user if not exists paradime_cost_user
  rsa_public_key = '<your_public_key_here>'
  comment = 'Used by paradime.io for cost analytics';

grant role paradime_cost_role to user paradime_cost_user;

-- set the defaults with ALTER USER, so they also apply if the user already exists
alter user paradime_cost_user set
  default_role = paradime_cost_role
  default_warehouse = paradime_warehouse;

If you use an existing user instead of paradime_cost_user, run the grant role and alter user statements for that user. create user if not exists changes nothing on a user that already exists.

Create a Snowflake warehouse

-- 2.### create warehouse

use role sysadmin;
create warehouse if not exists paradime_warehouse
  warehouse_size=xsmall
  auto_suspend=60
  initially_suspended=true
  comment = 'Used by paradime.io for cost analytics';

grant monitor, operate, usage, modify
  on warehouse paradime_warehouse to role paradime_cost_role;

Create a new database and schema for Paradime

-- 3.### create paradime db and schemas and grant access

use role accountadmin;

create database if not exists PARADIME_COST_ANALYTICS;

create schema if not exists PARADIME_COST_ANALYTICS.CORE;

grant usage on database PARADIME_COST_ANALYTICS to role paradime_cost_role;

grant usage on schema PARADIME_COST_ANALYTICS.CORE to role paradime_cost_role;

Create S3 integration

Before running this command, you will need to update STORAGE_AWS_ROLE_ARN and STORAGE_ALLOWED_LOCATIONS with your own region and company token.You can find the region and company token in the Paradime workspace settings page. company-and-workspace-token.md

-- 4.### Create S3 integration

use role accountadmin;

create or replace storage integration PARADIME_S3_STORAGE_INTEGRATION
    type = external_stage
    storage_provider = 'S3'
    enabled = true
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::817727247863:role/paradime-snowflake-role-<your_paradime_region>-<your_company_token>'  -- replace company token and region before running this command
    STORAGE_ALLOWED_LOCATIONS = ('s3://paradime-s3-<your_company_token>-<your_paradime_region>/snowflake/');  -- replace company token and region before running this command

grant usage on integration PARADIME_S3_STORAGE_INTEGRATION to role paradime_cost_role;

If you have set PREVENT_UNLOAD_TO_INLINE_URL to true, you can optionally use a named stage. To set this up run the below script.

Before running this command, you will need to update URL with your own region and company token. You can find the region and company token in the Paradime workspace settings page. company-and-workspace-token.md

-- 4.### Optional when using a named stage

use role accountadmin;

create stage PARADIME_COST_ANALYTICS.CORE.PARADIME_EXPORT
    url='s3://paradime-s3-<your_company_token>-<your_paradime_region>/snowflake/' -- replace company token and region before running this command
    storage_integration = PARADIME_S3_STORAGE_INTEGRATION;

grant usage on stage PARADIME_COST_ANALYTICS.CORE.PARADIME_EXPORT to role paradime_cost_role;

Create a new Snowflake file format

-- 5.### Create fileformat and grant privileges

use role accountadmin;

create or replace file format PARADIME_COST_ANALYTICS.CORE.PARADIME_JSON_FORMAT
    type = JSON
    NULL_IF = ();

grant all privileges on file format PARADIME_COST_ANALYTICS.CORE.PARADIME_JSON_FORMAT to role paradime_cost_role;

Grant access to Snowflake metadata to the Paradime user

-- 6.### grant access to Snowflake metadata

use role accountadmin;

grant imported privileges on database snowflake to role paradime_cost_role;

Enable the Paradime user to manage and monitor Snowflake Warehouses

-- 7.### grant access to monitor warehouses

use role accountadmin;

grant monitor usage on account to role paradime_cost_role;

--- ### grant paradime the access to manage warehouses
use role accountadmin;

grant manage warehouses on account to role paradime_cost_role;

--- ### grant paradime the access to manage tasks
use role accountadmin;

grant create TASK on schema PARADIME_COST_ANALYTICS.CORE to role paradime_cost_role;

grant execute TASK on ACCOUNT TO ROLE paradime_cost_role;

Check the Paradime user

-- 8.### check the user's defaults and roles

use role accountadmin;

desc user paradime_cost_user;   -- DEFAULT_ROLE must be PARADIME_COST_ROLE, DEFAULT_WAREHOUSE must be PARADIME_WAREHOUSE

show grants to user paradime_cost_user;   -- the list must include PARADIME_COST_ROLE

If DEFAULT_ROLE is empty or shows a different role, run the alter user statement from Create a Snowflake role and user again. If PARADIME_COST_ROLE isn’t in the grants, run grant role paradime_cost_role to user paradime_cost_user;.

Ensure your account has ORGADMIN access

Paradime relies on views from the snowflake.organization_usage such as the snowflake.organization_usage.rate_sheet_daily view is used to retrieve your daily rate for each Snowflake service (example the cost per credit for compute resources and cost per terabyte for storage). To verify if these views are populated in your account, please execute the following SQL statement:

--- ### check the rate_sheet_daily view contains data
use role accountadmin;

select * from snowflake.organization_usage.rate_sheet_daily;

If records are returned, you’re all set 🙌

If the SQL statement returns no records, it likely means that the ORGADMIN role is not enabled for the account. To enable it, follow these steps:

Log into a Snowflake account within your organization that has the ORGADMIN role enabled (at least one account should have this role).
Navigate to the sidebar and click on “Admin” and then “Accounts”.
Locate the account you want to connect to Paradime enable the ORGADMIN role for that account.

💡 It can take up to 1 day for these views become populated with data after running this statement

Purchased your Snowflake credits through a reseller?

If you purchased your Snowflake contract or credits through a reseller, such as the AWS Marketplace, you will not have access to the organization_usage views even after executing the steps mentioned above to enable the ORGADMIN role.

Share a copy of your latest Snowflake invoice with the Paradime team (via chat or by emailing support@paradime.io so that we can manually input the applicable rates for your account.

Full script here 👇

Snowflake setup guide script

lineNumbers

-- 1.### create role and user

use role useradmin;

create role if not exists paradime_cost_role comment = 'Used by paradime.io for cost analytics';
grant role paradime_cost_role to role sysadmin;

create user if not exists paradime_cost_user
  rsa_public_key = '<your_public_key_here>'
  comment = 'Used by paradime.io for cost analytics';

grant role paradime_cost_role to user paradime_cost_user;

-- set the defaults with ALTER USER, so they also apply if the user already exists
alter user paradime_cost_user set
  default_role = paradime_cost_role
  default_warehouse = paradime_warehouse;

-- 2.### create warehouse

use role sysadmin;
create warehouse if not exists paradime_warehouse
  warehouse_size=xsmall
  auto_suspend=60
  initially_suspended=true
  comment = 'Used by paradime.io for cost analytics';

grant monitor, operate, usage, modify
  on warehouse paradime_warehouse to role paradime_cost_role;

-- 3.### create paradime db and schemas and grant access

use role accountadmin;

create database if not exists PARADIME_COST_ANALYTICS;

create schema if not exists PARADIME_COST_ANALYTICS.CORE;

grant usage on database PARADIME_COST_ANALYTICS to role paradime_cost_role;

grant usage on schema PARADIME_COST_ANALYTICS.CORE to role paradime_cost_role;

-- 4.### Create S3 integration

use role accountadmin;

create or replace storage integration PARADIME_S3_STORAGE_INTEGRATION
    type = external_stage
    storage_provider = 'S3'
    enabled = true
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::817727247863:role/paradime-snowflake-role-<your_paradime_region>-<your_company_token>'  -- replace company token and region before running this command
    STORAGE_ALLOWED_LOCATIONS = ('s3://paradime-s3-<your_company_token>-<your_paradime_region>/snowflake/');  -- replace company token and region before running this command

grant usage on integration PARADIME_S3_STORAGE_INTEGRATION to role paradime_cost_role;

-- 5.### Create fileformat and grant privileges

use role accountadmin;

create or replace file format PARADIME_COST_ANALYTICS.CORE.PARADIME_JSON_FORMAT
    type = JSON
    NULL_IF = ();

grant all privileges on file format PARADIME_COST_ANALYTICS.CORE.PARADIME_JSON_FORMAT to role paradime_cost_role;

-- 6.### grant access to Snowflake metadata

use role accountadmin;

grant imported privileges on database snowflake to role paradime_cost_role;

-- 7.### grant access to monitor warehouses

use role accountadmin;

grant monitor usage on account to role paradime_cost_role;


--- ### grant paradime the access to manage warehouses
use role accountadmin;

grant manage warehouses on account to role paradime_cost_role;

--- ### grant paradime the access to manage tasks
use role accountadmin;

grant create TASK on schema PARADIME_COST_ANALYTICS.CORE to role paradime_cost_role;

grant execute TASK on ACCOUNT TO ROLE paradime_cost_role;

-- 8.### check the user's defaults and roles

use role accountadmin;

desc user paradime_cost_user;   -- DEFAULT_ROLE must be PARADIME_COST_ROLE, DEFAULT_WAREHOUSE must be PARADIME_WAREHOUSE

show grants to user paradime_cost_user;   -- the list must include PARADIME_COST_ROLE

--- ### check the rate_sheet_daily view contains data
use role accountadmin;

select * from snowflake.organization_usage.rate_sheet_daily;

2. dbt™️ project setup guide

Create a macro to manage query comments

To enable Paradime to enrich your Snowflake queries with additional metadata you will need to create a new dbt™️ macro called get_query_comment.sql in the macros folder of your project.

get_query_comment.sql dbt™️ macro

get_query_comment.sql

{% macro get_query_comment(node, extra = {}) %}
    {%- set comment_dict = extra -%}
    {%- do comment_dict.update(
        app='dbt',
        dbt_version=dbt_version,
        project_name=project_name,
        target_name=target.name,
        target_database=target.database,
        target_schema=target.schema,
        invocation_id=invocation_id
    ) -%}

    {%- if node is not none -%}
        {%- do comment_dict.update(
            node_name=node.name,
            node_alias=node.alias,
            node_package_name=node.package_name,
            node_original_file_path=node.original_file_path,
            node_database=node.database,
            node_schema=node.schema,
            node_id=node.unique_id,
            node_resource_type=node.resource_type,
            node_meta=node.config.meta,
            node_tags=node.tags,
            full_refresh=flags.FULL_REFRESH,
            which=flags.WHICH,
        ) -%}

        {%- if flags.INVOCATION_COMMAND -%}
            {%- do comment_dict.update(
                invocation_command=flags.INVOCATION_COMMAND
            ) -%}
        {%- endif -%}

        {%- if node.resource_type != ('seed') -%} {# Otherwise this throws an error saying 'Seeds cannot depend on other nodes.' #}
            {%- if node.refs is defined -%}
                {% set refs = [] %}
                {% for ref in node.refs %}
                    {%- if (dbt_version.split('.')[0] | int > 1)
                       or (dbt_version.split('.')[0] | int == 1 and dbt_version.split('.')[1] | int >= 5) -%}
                        {%- do refs.append(ref.name) -%}
                    {%- else -%}
                        {%- do refs.append(ref[0]) -%}
                    {%- endif -%}
                {% endfor %}
                {%- do comment_dict.update(
                    node_refs=refs | unique | list
                ) -%}
            {%- endif -%}
        {%- endif -%}
        {%- if node.resource_type == 'model' -%}
            {%- do comment_dict.update(
                materialized=node.config.materialized,
            ) -%}
        {%- endif -%}
    {%- endif -%}

--- only if using dbt cloud

    {%- if env_var('DBT_CLOUD_PROJECT_ID', False) -%}
    {%- do comment_dict.update(
        dbt_cloud_project_id=env_var('DBT_CLOUD_PROJECT_ID')
    ) -%}
    {%- endif -%}

    {%- if env_var('DBT_CLOUD_JOB_ID', False) -%}
    {%- do comment_dict.update(
        dbt_cloud_job_id=env_var('DBT_CLOUD_JOB_ID')
    ) -%}
    {%- endif -%}

    {%- if env_var('DBT_CLOUD_RUN_ID', False) -%}
    {%- do comment_dict.update(
        dbt_cloud_run_id=env_var('DBT_CLOUD_RUN_ID')
    ) -%}
    {%- endif -%}

    {%- if env_var('DBT_CLOUD_RUN_REASON_CATEGORY', False) -%}
    {%- do comment_dict.update(
        dbt_cloud_run_reason_category=env_var('DBT_CLOUD_RUN_REASON_CATEGORY')
    ) -%}
    {%- endif -%}

    {%- if env_var('DBT_CLOUD_RUN_REASON', False) -%}
    {%- do comment_dict.update(
        dbt_cloud_run_reason=env_var('DBT_CLOUD_RUN_REASON')
    ) -%}
    {%- endif -%}

-- only if using bolt scheduler
    {%- if env_var('PARADIME_SCHEDULE_NAME', False) -%}
    {%- do comment_dict.update(
        paradime_schedule_name=env_var('PARADIME_SCHEDULE_NAME')
    ) -%}
    {%- endif -%}

    {%- if env_var('PARADIME_SCHEDULE_RUN_ID', False) -%}
    {%- do comment_dict.update(
        paradime_schedule_run_id=env_var('PARADIME_SCHEDULE_RUN_ID')
    ) -%}
    {%- endif -%}

    {%- if env_var('PARADIME_SCHEDULE_RUN_START_DTTM', False) -%}
    {%- do comment_dict.update(
        paradime_schedule_run_start_dttm=env_var('PARADIME_SCHEDULE_RUN_START_DTTM')
    ) -%}
    {%- endif -%}

    {%- if env_var('PARADIME_SCHEDULE_TRIGGER', False) -%}
    {%- do comment_dict.update(
        paradime_schedule_trigger=env_var('PARADIME_SCHEDULE_TRIGGER')
    ) -%}
    {%- endif -%}

    {%- if env_var('PARADIME_SCHEDULE_GIT_SHA', False) -%}
    {%- do comment_dict.update(
        paradime_schedule_git_sha=env_var('PARADIME_SCHEDULE_GIT_SHA')
    ) -%}
    {%- endif -%}

    {{ return(tojson(comment_dict)) }}
{% endmacro %}

Update your dbt_project.yml file

This step ensures that with each dbt™️ run, the query comment is appended to the query running in Snowflake.

dbt_project.yml

query-comment:
  comment: '{{ get_query_comment(node) }}'
  append: true

3. Configure the cost connection in Paradime

Step 1: Add a Snowflake cost connection

Open Workspace Settings from the top menu bar of the Paradime interface.
Click Environments in the left sidebar.
Click “Add New” next to the Radar.
Select “Snowflake” as the connection type.

Step 2: Enter the required credentials

After selecting Snowflake, provide the following details:

Snowflake Account Identifier
Paradime Cost Database: PARADIME_COST_ANALYTICS

Key-Pair
Username & Password

If using the Key-Pair auth method, enter:

Username: paradime_cost_user
Private Key: The private key that matches the public key you set on the user in Create a Snowflake role and user. Include the BEGIN and END lines.
Passphrase (Optional): The passphrase you set when you generated an encrypted private key.

Snowflake requires multi-factor authentication for users who sign in with a password, and blocks password sign-in for service users. A password connection can stop working. Use Key-Pair. See Multi-factor authentication is required for this account.

If using the Username & Password auth method, enter:

Username: paradime_cost_user
Password: The password of the Snowflake user. To give the user a password, run alter user paradime_cost_user set password = '<your_generated_password_here>';.

The connection saves when Paradime can sign in and read the storage integration. This doesn’t check the user’s default role. If Radar shows no cost data after you save the connection, run the checks in Check the Paradime user.

​1. Snowflake setup guide

​Create a Snowflake role and user

​Create a Snowflake warehouse

​Create a new database and schema for Paradime

​Create S3 integration

​Create a new Snowflake file format

​Grant access to Snowflake metadata to the Paradime user

​Enable the Paradime user to manage and monitor Snowflake Warehouses

​Check the Paradime user

​Ensure your account has ORGADMIN access

​Purchased your Snowflake credits through a reseller?

​Full script here 👇

​2. dbt™️ project setup guide

​Create a macro to manage query comments

​Update your dbt_project.yml file

​3. Configure the cost connection in Paradime

​Step 1: Add a Snowflake cost connection

​Step 2: Enter the required credentials

1. Snowflake setup guide

Create a Snowflake role and user

Create a Snowflake warehouse

Create a new database and schema for Paradime

Create S3 integration

Create a new Snowflake file format

Grant access to Snowflake metadata to the Paradime user

Enable the Paradime user to manage and monitor Snowflake Warehouses

Check the Paradime user

Ensure your account has ORGADMIN access

Purchased your Snowflake credits through a reseller?

Full script here 👇

2. dbt™️ project setup guide

Create a macro to manage query comments

Update your dbt_project.yml file

3. Configure the cost connection in Paradime

Step 1: Add a Snowflake cost connection

Step 2: Enter the required credentials