Ataccama ZIP Ingestion

Import lineage data to Ataccama ONE from a ZIP file stored in your Amazon S3 bucket, Azure Data Lake Storage (ADLS) Gen2 account, or local filesystem where you have your standalone lineage scanner.

How the Ataccama ZIP ingestion works

ONE reads a ZIP file from the configured storage location, extracts it, and imports the lineage data into the platform.

Take note of the following:

ONE never writes to your storage. The ZIP ingestion is read-only; ONE never creates, copies, deletes, or uploads objects to your bucket, container, or directory.
When multiple ZIP files are present, the newest is uploaded. ONE processes the most recently modified file.

Prerequisites

Before you configure the ZIP ingestion, make sure you have the necessary cloud storage set up and the appropriate credentials and permissions.

Amazon S3

To use Amazon S3 as your storage, you need:

An Amazon S3 bucket containing the lineage ZIP file.
The bucket name, the AWS Region the bucket is in, and the prefix where the ZIP file is stored.
Permission to create an AWS Identity and Access Management (IAM) policy and either a role or access keys for that bucket. See S3 permissions.

Azure ADLS Gen2

To use Azure Data Lake Storage Gen2 as your storage, you need:

An Azure storage account with ADLS Gen2 enabled.
A container within that storage account containing the lineage ZIP file.
The storage account name, the container name, and the prefix where the ZIP file is stored.
Either a storage account key or a Shared Access Signature (SAS) token with read and list permissions on the container.

Local filesystem

The local filesystem option is intended for use with the standalone lineage scanner.

To use a local filesystem path as your storage, you need:

A directory accessible from within the standalone scanner container containing the lineage ZIP file.
The container-side absolute path to that directory. By default, the user-data directory is mapped into the lineage scanner container (lss) at /opt/ataccama/lineage-scanning/user-data.

To use a directory outside this path, add a custom volume mount to the lineage scanner service (lss) in your Docker Compose configuration and use the container-side path as inputDir.

ZIP ingestion configuration

All fields marked with an asterisk (*) are mandatory.

Top-level configuration

Property Description

Property	Description
`sourceType*`	Identifies the ingestion type. Must be set to `ATACCAMA_ZIP`.
`name*`	A name for the ingestion configuration.
`description`	A human-readable description of the ingestion.
`type (storage)*`	The storage backend type. Possible values: `S3`, `ADLS_GEN2`, or `LOCAL`.

sourceType*

Identifies the ingestion type. Must be set to ATACCAMA_ZIP.

name*

A name for the ingestion configuration.

description

A human-readable description of the ingestion.

type (storage)*

The storage backend type. Possible values: S3, ADLS_GEN2, or LOCAL.

Amazon S3 storage

When using Amazon S3 as your storage, configure the following fields:

Property Description

Property	Description
`bucket*`	The name of the S3 bucket containing the lineage ZIP file.
`prefix`	The prefix that holds the ZIP file. When multiple ZIP files are present, ONE processes the most recently modified file. Defaults to an empty string (bucket root).
`region*`	The AWS Region the bucket is in, for example `eu-north-1`.
`type (credentials)*`	The authentication method. Possible values: `ASSUME_ROLE` for IAM role authentication. `ACCESS_KEY` for a static access key and secret key.

bucket*

The name of the S3 bucket containing the lineage ZIP file.

prefix

The prefix that holds the ZIP file. When multiple ZIP files are present, ONE processes the most recently modified file.

Defaults to an empty string (bucket root).

region*

The AWS Region the bucket is in, for example eu-north-1.

type (credentials)*

The authentication method. Possible values:

ASSUME_ROLE for IAM role authentication.
ACCESS_KEY for a static access key and secret key.

S3 permissions

The ZIP ingestion requires read-only access with the following permissions:

s3:ListBucket on the bucket, restricted to the configured prefix.
s3:GetObject on the objects under that prefix.

If the bucket uses server-side encryption with AWS Key Management Service (SSE-KMS) and a customer-managed key, ONE also needs kms:Decrypt on that key. No write, delete, or tagging permissions are required.

The following policy grants exactly these permissions. Replace the bucket name, prefix, Region, and key identifier with your own values. Omit the kms:Decrypt statement if the bucket does not use SSE-KMS.

For step-by-step instructions, see Create IAM policies in the AWS documentation.

Read-only IAM policy for the Ataccama ZIP bucket

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowListUnderPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::ataccama-zip-bucket",
      "Condition": {
        "StringLike": { "s3:prefix": "ataccama-lineage/*" }
      }
    },
    {
      "Sid": "AllowGetUnderPrefix",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::ataccama-zip-bucket/ataccama-lineage/*"
    },
    {
      "Sid": "AllowDecryptForSseKms",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:eu-north-1:<ACCOUNT_ID>:key/<KMS_KEY_ID>"
    }
  ]
}

S3 authentication

Use IAM role authentication to avoid long-lived credentials. Attach the policy to a role and configure ONE to assume it.

Alternatively, you can use a static access key and secret key. Provide them as scan plan secrets and reference them from the scan plan with the @@ref:ata:[<secret_name>] placeholder to avoid storing them directly in the plan.

Assume Role credentials

Property Description

Property	Description
`roleArn*`	The Amazon Resource Name (ARN) of the AWS role ONE assumes. Required when authentication method (`type`) is `ASSUME_ROLE`.
`sessionName*`	A name for the assumed-role session, used to identify it in AWS logs (CloudTrail). Required when `type` is `ASSUME_ROLE`.
`externalId`	The external ID required by the role’s trust policy, if one is configured. This is an AWS mechanism that helps prevent the confused deputy problem when a third party assumes a role in your account. Optional and only used when `type` is `ASSUME_ROLE`.

roleArn*

The Amazon Resource Name (ARN) of the AWS role ONE assumes. Required when authentication method (type) is ASSUME_ROLE.

sessionName*

A name for the assumed-role session, used to identify it in AWS logs (CloudTrail). Required when type is ASSUME_ROLE.

externalId

The external ID required by the role’s trust policy, if one is configured. This is an AWS mechanism that helps prevent the confused deputy problem when a third party assumes a role in your account.

Optional and only used when type is ASSUME_ROLE.

Access key credentials

Property Description

Property	Description
`accessKeyId*`	The AWS access key ID. Required when `type` is `ACCESS_KEY`.
`secretAccessKey*`	The AWS secret access key. Required when `type` is `ACCESS_KEY`.

accessKeyId*

The AWS access key ID. Required when type is ACCESS_KEY.

secretAccessKey*

The AWS secret access key. Required when type is ACCESS_KEY.

Azure ADLS Gen2 storage

When using Azure Data Lake Storage Gen2 as your storage, configure the following fields:

Property Description

Property	Description
`accountName*`	The name of your Azure storage account, without the `.dfs.core.windows.net` suffix. For example, if your account URL is `mystorageaccount.dfs.core.windows.net`, the account name is `mystorageaccount`.
`filesystem*`	The name of the container within your storage account containing the lineage ZIP file.
`prefix`	The prefix that holds the ZIP file. When multiple ZIP files are present, ONE processes the most recently modified file. Defaults to an empty string (container root).
`type (credentials)*`	The authentication method. Possible values: `ACCOUNT_KEY` for a storage account key. `SAS_TOKEN` for a Shared Access Signature token.

accountName*

The name of your Azure storage account, without the .dfs.core.windows.net suffix. For example, if your account URL is mystorageaccount.dfs.core.windows.net, the account name is mystorageaccount.

filesystem*

The name of the container within your storage account containing the lineage ZIP file.

prefix

The prefix that holds the ZIP file. When multiple ZIP files are present, ONE processes the most recently modified file.

Defaults to an empty string (container root).

type (credentials)*

The authentication method. Possible values:

ACCOUNT_KEY for a storage account key.
SAS_TOKEN for a Shared Access Signature token.

ADLS Gen2 permissions

The ZIP ingestion requires the following permissions on the container:

List the container contents.
Read ZIP files within the configured prefix.

If you are using a storage account key, it has full permissions by default. If you are using a Shared Access Signature (SAS) token, ensure it includes the following permissions:

List (l): To enumerate files under the prefix.
Read (r): To download the ZIP file.

The token must be scoped to the container or to files under the configured prefix.

For instructions on creating SAS tokens, see Create a user delegation SAS for a blob in the Azure documentation.

ADLS Gen2 authentication

Account key credentials

Property Description

Property	Description
`accountKey*`	The primary or secondary access key for your storage account. Required when authentication method (`type`) is `ACCOUNT_KEY`. Store the account key as a scan plan secret and reference it from the scan plan with the `@@ref:ata:[<secret_name>]` placeholder to avoid storing it directly in the plan.

accountKey*

The primary or secondary access key for your storage account. Required when authentication method (type) is ACCOUNT_KEY.

Store the account key as a scan plan secret and reference it from the scan plan with the @@ref:ata:[<secret_name>] placeholder to avoid storing it directly in the plan.

Shared Access Signature token credentials

Property Description

Property	Description
`sasToken*`	The Shared Access Signature (SAS) token with read and list permissions on the container. Required when authentication method (`type`) is `SAS_TOKEN`. Store the SAS token as a scan plan secret and reference it from the scan plan with the `@@ref:ata:[<secret_name>]` placeholder to avoid storing it directly in the plan.

sasToken*

The Shared Access Signature (SAS) token with read and list permissions on the container. Required when authentication method (type) is SAS_TOKEN.

Store the SAS token as a scan plan secret and reference it from the scan plan with the @@ref:ata:[<secret_name>] placeholder to avoid storing it directly in the plan.

Local filesystem storage

When using a local filesystem as your storage with lineage standalone scanner, configure the following field:

Property Description

Property	Description
`inputDir*`	The path to the local directory containing the ZIP file. The path must be reachable from within the Docker container.

inputDir*

The path to the local directory containing the ZIP file. The path must be reachable from within the Docker container.

Scan plan examples

The following examples show how to configure scan plans for all supported credential types.

S3 with IAM role authentication

The following example uses Amazon S3 with IAM role authentication (ASSUME_ROLE).

Ataccama ZIP ingestion scan plan - S3 with IAM role

{
  "scannerConfigs": [
    {
      "sourceType": "ATACCAMA_ZIP",
      "name": "ataccama-zip-s3-role",
      "description": "Ataccama ZIP ingestion with S3 and IAM role",
      "storage": {
        "type": "S3",
        "bucket": "my-lineage-bucket",
        "prefix": "ataccama-lineage/",
        "region": "eu-west-1",
        "credentials": {
          "type": "ASSUME_ROLE",
          "roleArn": "arn:aws:iam::123456789012:role/AtaccamaZipScanner",
          "sessionName": "ataccama-zip-scan",
          "externalId": "optional-external-id"
        }
      }
    }
  ]
}

S3 with static access key

The following example uses Amazon S3 with static access key authentication (ACCESS_KEY). The access key and secret key are stored as scan plan secrets and referenced using the @@ref:ata:[<secret_name>] placeholders.

Ataccama ZIP ingestion scan plan - S3 with static access key

{
  "scannerConfigs": [
    {
      "sourceType": "ATACCAMA_ZIP",
      "name": "ataccama-zip-s3-key",
      "description": "Ataccama ZIP ingestion with S3 and static key",
      "storage": {
        "type": "S3",
        "bucket": "my-lineage-bucket",
        "prefix": "ataccama-lineage/",
        "region": "eu-west-1",
        "credentials": {
          "type": "ACCESS_KEY",
          "accessKeyId": "@@ref:ata:[ATACCAMA_ACCESS_KEY_ID]",
          "secretAccessKey": "@@ref:ata:[ATACCAMA_SECRET_ACCESS_KEY]"
        }
      }
    }
  ]
}

Azure ADLS Gen2 with storage account key

The following example uses Azure Data Lake Storage Gen2 with storage account key authentication (ACCOUNT_KEY). The account key is stored as a scan plan secret and referenced using the @@ref:ata:[<secret_name>] placeholder.

Ataccama ZIP ingestion scan plan - Azure ADLS Gen2 with account key

{
  "scannerConfigs": [
    {
      "sourceType": "ATACCAMA_ZIP",
      "name": "ataccama-zip-adls-key",
      "description": "Ataccama ZIP ingestion with Azure ADLS Gen2 and account key",
      "storage": {
        "type": "ADLS_GEN2",
        "accountName": "mystorageaccount",
        "filesystem": "ataccama-lineage",
        "prefix": "ataccama-zip/",
        "credentials": {
          "type": "ACCOUNT_KEY",
          "accountKey": "@@ref:ata:[ATACCAMA_ACCOUNT_KEY]"
        }
      }
    }
  ]
}

Azure ADLS Gen2 with SAS token

The following example uses Azure Data Lake Storage Gen2 with Shared Access Signature (SAS) token authentication (SAS_TOKEN). The SAS token is stored as a scan plan secret and referenced using the @@ref:ata:[<secret_name>] placeholder.

Ataccama ZIP ingestion scan plan - Azure ADLS Gen2 with SAS token

{
  "scannerConfigs": [
    {
      "sourceType": "ATACCAMA_ZIP",
      "name": "ataccama-zip-adls-sas",
      "description": "Ataccama ZIP ingestion with Azure ADLS Gen2 and SAS token",
      "storage": {
        "type": "ADLS_GEN2",
        "accountName": "mystorageaccount",
        "filesystem": "ataccama-lineage",
        "prefix": "ataccama-zip/",
        "credentials": {
          "type": "SAS_TOKEN",
          "sasToken": "@@ref:ata:[ATACCAMA_SAS_TOKEN]"
        }
      }
    }
  ]
}

Local filesystem

The following example uses a local filesystem directory.

Ataccama ZIP ingestion scan plan - Local filesystem

{
  "scannerConfigs": [
    {
      "sourceType": "ATACCAMA_ZIP",
      "name": "ataccama-zip-local",
      "description": "Ataccama ZIP ingestion from local filesystem",
      "storage": {
        "type": "LOCAL",
        "inputDir": "/opt/ataccama/lineage-scanning/user-data/ataccama-lineage"
      }
    }
  ]
}

Scan schedule

The ZIP ingestion runs on a schedule that you set on the scan plan.

When configuring the schedule, take into account how often the lineage ZIP file in your storage is updated. Running the ingestion more often than the ZIP is updated only reprocesses the same data.

For instructions about how to schedule scans, see Schedule scan plan.

Troubleshooting

No ZIP file found under the configured prefix

ONE logs an error when no ZIP file exists under the configured prefix.

Verify that a ZIP file exists under the configured prefix in your storage.
Confirm the prefix path in the scan plan matches the location where the ZIP file is stored.

Storage authentication or permission failure

The ZIP ingestion fails with an authentication or permission error when credentials are missing, invalid, or lack required permissions.

For Amazon S3:

Verify that the IAM role or access keys are valid and current.
Confirm that the policy grants the necessary permissions, see S3 permissions.
Check that the policy is restricted to the configured prefix.

For Azure ADLS Gen2:

Verify that the storage account key is current (rotate if needed) or that the SAS token has not expired.
Confirm that the token includes correct permissions on the container, see ADLS Gen2 permissions.
Check that the token is scoped to the container or to the configured prefix.

Cannot reach the storage endpoint

The ZIP ingestion fails when the network cannot reach your cloud storage endpoint.

Verify network connectivity to the S3 endpoint (AWS) or the Azure storage endpoint (.dfs.core.windows.net).
Confirm that the bucket name and Region (S3) or the storage account name (Azure) are correct.
For network diagnostics, download the debug archive and check the log files.

Was this page useful?