Lineage Edge Processing

Edge processing in Ataccama Lineage allows you to extract lineage information from data sources that cannot be reached directly from your Ataccama Cloud environment, typically due to security constraints. This approach enables you to process data from sensitive or highly regulated data sources while also maintaining the convenience of centralized management of lineage metadata in Ataccama ONE.

In this setup, a standalone lineage scanner is deployed as a remote agent close to your data source. The scanner receives scan workloads from the cloud, processes them locally, and returns results back to ONE without manual intervention.

Compared to using a standalone lineage scanner without edge processing, where scan results have to be manually uploaded to ONE after every scanning, edge processing is more efficient, less error-prone, and easier to maintain in the long term. This makes edge processing particularly valuable for organizations operating in strictly regulated industries, as it enables compliance with data governance requirements while providing a seamless, automated experience once the initial setup is complete.

Before setting up edge processing, install a standalone lineage scanner at each edge location. For installation instructions, see Standalone Lineage Scanner.

The standalone scanner (in this context: agent) must be registered and paired with Ataccama ONE in the cloud. This creates a communication channel between the scanner at the edge and the platform.

To register the scanner, use the following command.

The edge instance name will later be used when configuring scan plans for this scanner. We recommend using a clear and descriptive name.

If the registration was successful, the expected output is as follows:

Create a valid scan plan for your data source. For detailed instructions, see Built-In Lineage Scanner and the documentation section Scanner Configuration > Configuration by Technology).

To run the scan on the scanner with edge processing, add the name of your scanner to the name of the scan plan in the format: [scan plan name] [edge instance name]. The edge instance name is the same one you used to register the scanner.

Once your scan plan is ready, you can run it to extract lineage metadata from your source.

To use Azure Key Vault to retrieve secrest in a scan plan, the connection to Azure Key Vault is established from the edge.

For details about how to configure Azure Key Vault at the edge, see Standalone Lineage Scanner > Configure Azure Key Vault.