Azure Data Factory Lineage Scanner

Scanned and supported objects

Resources scanned

The lineage information from Azure Data Factory is available on attribute level and the metadata extracted includes the following Azure Factory resources:

Factories
Linked services
Pipelines
Datasets
Dataflows

Supported source technologies

The scanner supports the following Azure Data Factories linked services:

AzurePostgreSqlLinkedService
AzureSqlDatabaseLinkedService
SqlServerLinkedService
SnowflakeV2LinkedService

Supported dataflow transformations

Source
Aggregate
Select
Filter
Join
Union
Sink

Supported pipeline activities

Copy activity

Limitations

Azure Data Factory enables 12,500 requests per hour. For more details, see the official Microsoft documentation.

Supported connectivity

The scanner connects to Azure Data Factory using the following connection method:

Connector type: API integration using the Microsoft Azure SDK.
Authentication method: Azure Service Principal.

Azure Data Factory permissions

Azure Data Factory uses Azure Role-Based Access Control (RBAC) to manage permissions. To enable access to Data Factory resources using a Service Principal, assign the Data Factory Contributor role to the corresponding app registration (service principal) in Azure Active Directory.

Assign roles in Azure Data Factory

To assign a role to an App Registration (Service Principal) in Azure Data Factory, you must have one of the following roles at the appropriate scope:

User Access Administrator — Allows managing role assignments without granting full resource control.
Owner — Grants full permissions, including the ability to manage access.
1. In Azure Portal, navigate to your Azure Data Factory.
2. Open Access control (IAM).
3. Select Add > Add permissions.
4. Select the Data Factory Contributor role and on the Members tab, assign it to the relevant user or group of users.

Scanner configuration

All fields marked with an asterisk (*) are mandatory.

Property Description

Property	Description
`name*`	Unique name for the scanner job.
`sourceType*`	Specifies the source type to be scanned. Must contain `ADF`.
`description*`	A human-readable description of the scan.
`oneConnections`	List of Ataccama ONE connection names for future automatic pairing.
`inputDataCatalogFilePath`	Path to Data Factory Manager.
`factoryName*`	Name of the factory to be processed.
`dataflowFolders*`	List of dataflow folders to be processed.
`tenantId*`	To find your organization `tenantID`, in your Azure Portal, go to Azure Active Directory > Properties.
`subscriptionId*`	To find your `subscriptionId`, in your Azure Portal, go to Subscriptions > Select your subscription.
`clientID*`	Microsoft OAuth 2.0 credentials client ID. Make sure to first register a Microsoft Entra application in Azure. The application establishes permissions for SDK resources and allows access to the Azure Data Factory SDK data.
`clientSecret*`	Microsoft OAuth 2.0 credentials client secret.
`resourceGroup*`	Name of the container that holds related resources for an Azure solution.

name*

Unique name for the scanner job.

sourceType*

Specifies the source type to be scanned. Must contain ADF.

description*

A human-readable description of the scan.

oneConnections

List of Ataccama ONE connection names for future automatic pairing.

inputDataCatalogFilePath

Path to Data Factory Manager.

factoryName*

Name of the factory to be processed.

dataflowFolders*

List of dataflow folders to be processed.

tenantId*

To find your organization tenantID, in your Azure Portal, go to Azure Active Directory > Properties.

subscriptionId*

To find your subscriptionId, in your Azure Portal, go to Subscriptions > Select your subscription.

clientID*

Microsoft OAuth 2.0 credentials client ID.

Make sure to first register a Microsoft Entra application in Azure. The application establishes permissions for SDK resources and allows access to the Azure Data Factory SDK data.

clientSecret*

Microsoft OAuth 2.0 credentials client secret.

resourceGroup*

Name of the container that holds related resources for an Azure solution.

Azure Data Factory scanner example configuration

{
   "scannerConfigs":[
      {
         "name":"ADFJob1",
         "sourceType":"ADF",
         "description":"Scan Azure Data Factory",
         "oneConnections":[

         ],
         "inputDataCatalogFilePath":null,
         "factoryName": "myfactoryName",
         "dataflowFolders": ["folder1", "folder2"],
         "tenantId":"tenant-id",
         "subscriptionId":"your-subscription-id",
         "clientId":"your-client-id",
         "clientSecret":"@@ref:ata:[ADF_CLIENT_SECRET]",
         "resourceGroup":"MetadataExtractionGroup"
      }
   ]
}

Was this page useful?