Amazon S3

Paradime supports using AWS S3 custom buckets in the customer's VPC to store logs and run artifacts generated in each dbt™️ command when running schedules using Bolt.

pageRunning dbt™️ in production

How to set up the integration

To complete setting up the AWS S3 integration, reach out to the paradime team at [e-mail] to get the Paradime IAM ARN and ExternalD.

1. Create your own AWS S3 bucket

In your AWS account navigate to Amazon S3 > Buckets > Create Bucket. Enter the custom bucket name and select the region where you want to create your bucket in.

The bucket region should be the same as the region where your Paradime instance is hosted. You can find this in workspace settings.

2. Create IAM policy

Next create the IAM policy that provides the minimum required permissions for Paradime to access the Amazon S3 bucket.

Sign in to your AWS account and open the IAM Management Console and in the navigation pane, choose Policies, then Create policy.

Create IAM policy

Select the JSON tab and copy the permissions JSON as shown below to create a new IAM policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::<your-bucket-name>/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::<your-bucket-name>"
        }
    ]
}

3. Create IAM Role and attach the previously created IAM policy

Next create a new IAM role and assign the IAM policy created in the previous step to this role. In the IAM Management Console in the navigation pane, choose Roles, then Create role.

• Choose the AWS account and select the option Require external ID and add the ID the Paradime team provided and click on Next.

• Search for the IAM policy created in step 2 of this guide, select and click Next.

• Define a name for the new role and click on Create role

4. Update Trust Policy for the IAM role

Finally update the trust policy for the newly created IAM role.

• Select the tab Trust relationships and click on Edit trust policy

• Use the JSON as shown below, replace the Paradime IAM ARN and External ID with those provided by the Paradime team.

• Click on Update policy to apply the changes

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<paradime-arn>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<external-id-provided-by-paradime>"
        }
      }
    }
  ]
}

5. Send the details to the Paradime team

Once the custom S3 bucket and roles are created, reach out to the Paradime team on [e-mail] and securely share the details below via a password manager like Dashlane / 1Password.

• Bucket Name

• Bucket region

• Role ARN

Once the integration is setup we will create a dummy file paradime-empty-test-file in the customer's S3 bucket to test that Paradime can connect and write to it.

From this point onwards, dbt™️ run logs and artifacts will be stored in the custom S3 bucket.

FAQ: How the dbt™️ artifacts are stored in the S3 buckets?

In the S3 bucket you will see the Bolt folder which will contain all the artifacts by schedule name using the below folder structure.

Example

bolt 
└─ run/
   ├─ bi_sales_hourly/
   ├─ bi_sales_daily/
   └─ daily_general/
      ├─ current # in this folder we will add the latest target folder for each command of the schedule
      │  ├─ 47531_dbt_seed/ 
      │  └─ 47532_dbt_run/ 
      │     └─ target/
      │         ├─ compiled/
      │         ├─ run/
      │         ├─ graph.gpickle
      │         ├─ manifest.json
      │         ├─ partial_parse.msgpack
      │         └─ run_results.json
      │   
      └─ 2022/ 
         └─ 09/
            ├─ 16/
            ├─ 17/
            └─ 18/
               └─ 58602_1663718545.541192/
                  ├─ 47531_dbt_seed/ 
                  └─ 47532_dbt_run/ 
                     └─ target/
                        ├─ compiled/
                        ├─ run/
                        ├─ graph.gpickle
                        ├─ manifest.json
                        ├─ partial_parse.msgpack
                        └─ run_results.json