PowerBI Lineage Scanner

This scanner is using REST API to extract raw metadata from PowerBI workspaces. The metadata is then further processed, such as PowerQuery and DAX expressions are parsed and lineage metadata is created.

Metadata extraction details

The lineage information is available on attribute level. The metadata extracted includes the following objects:

Workspaces
Reports
Dashboards
Datasets
Columns
Measures

PowerBI scanning prerequisites

Installing PowerQuery parser

The PowerQuery Parser is an essential tool for importing, combining, and transforming data within PowerBI. It enables users to parse queries written in M-language, the official Power Query language.

To integrate the PowerQuery Parser into your workflow:

Use the official PowerQuery parser provided by us (see links below), capable of processing any queries written in M-language.
Standalone execution files of the parser are prepared, tailored for Microsoft, Apple, and Linux environments.
To integrate the PowerQuery Parser into your toolkit:
- Copy the parser execution file into a folder adjacent to your toolkit or any accessible directory.
- Update the configuration file of your toolkit to specify the path to the parser execution file based on your environment.

Available parsers for different operating systems:

Permission settings

PowerBI REST API permissions to call Admin - WorkspaceInfo GetScanResult

The user account used for lineage scanning must have administrator rights (such as Microsoft 365 Global Administrator or PowerBI Service Administrator), or authenticate using a service principal.

When running under service principal authentication, the service principal app must not have any admin-consent required permissions for PowerBI set on it in the Azure portal.

For more details, see the Microsoft documentation.

Access to PowerBI workspaces

The user account needs to have access rights to a given workspace in PowerBI.

In order to manage access for the PowerBI workspace, locate the workspace to be scanned in PowerBI
Click on "Manage Access" and setup ADMIN rights for the Azure PowerBI scan Entity

Enable tenant settings for metadata scanning

Two tenant settings, that control metadata scanning, needs to be enabled:

Enhance admin APIs responses with detailed metadata: This setting turns on Model caching and enhances API responses with low-level semantic model metadata (for example, name and description) for tables, columns, and measures.

Enhance admin APIs responses with detailed metadata

Enhance admin APIs responses with DAX and mashup expressions: This setting allows the API response to include DAX expressions and Mashup queries. This setting can only be enabled if the first setting is also enabled. To enable these settings, go to Admin portal > Tenant settings > Admin API settings.

Enhance admin APIs responses with DAX and mashup expressions

Toolkit configuration

Property Description and Example Values

Property	Description and Example Values
`*name`	Unique name of the scanner job
`*sourceType`	Must contain `POWER_BI`
`*description`	Human-readable description
`oneConnections`	List of Ataccama ONE connection names for future automatic pairing
`*powerQueryParserPath`	Path to the PowerQueryParser execution file
`*apiUrl`	URL of the PowerBI REST API endpoint
`*tokenUrl`	REST endpoint for Microsoft OAUTH 2.0, in the URL, we use the tenantID of your organization. To find your organization tenantID, go to Portal.azure.com > Azure Active Directory > Properties.
`*clientID`	Microsoft OAUTH 2.0 credentials client id. Register a Microsoft Entra application in Azure. The Microsoft Entra app establishes permissions for Power BI REST resources, and allows access to the Power BI REST APIs.
`*clientSecret`	Microsoft OAUTH 2.0 credentials client secret
`workspaces_disabled`	List of workspace IDs to be omitted from the scan.
`workspaces`	List of workspace IDs to be scanned. If empty, toolkit scans everything.

*name

Unique name of the scanner job

*sourceType

Must contain POWER_BI

*description

Human-readable description

oneConnections

List of Ataccama ONE connection names for future automatic pairing

*powerQueryParserPath

Path to the PowerQueryParser execution file

*apiUrl

URL of the PowerBI REST API endpoint

*tokenUrl

REST endpoint for Microsoft OAUTH 2.0, in the URL, we use the tenantID of your organization. To find your organization tenantID, go to Portal.azure.com > Azure Active Directory > Properties.

*clientID

Microsoft OAUTH 2.0 credentials client id. Register a Microsoft Entra application in Azure. The Microsoft Entra app establishes permissions for Power BI REST resources, and allows access to the Power BI REST APIs.

*clientSecret

Microsoft OAUTH 2.0 credentials client secret

workspaces_disabled

List of workspace IDs to be omitted from the scan.

workspaces

List of workspace IDs to be scanned. If empty, toolkit scans everything.

Legend: *mandatory

Identifying tenantID in Azure:

Identifying clientID in Azure:

Example configuration

{
  "scannerConfigs": [
    {
      "name": "PowerBIJob1",
      "sourceType": "POWER_BI",
      "description": "Scan Power BI platform",
      "oneConnections": [],
      "inputDataCatalogFilePath": null,
      "powerQueryParserPath" : "@@ref:env:[POWERQUERY_PARSER]",
      "tokenUrl": "https://login.microsoftonline.com/[tenantID]/oauth2/v2.0/token",
      "apiUrl": "https://api.powerbi.com/v1.0/myorg",
      "clientId": "your-client-id",
      "clientSecret": "@@ref:ata:[POWER_BI_CLIENT_SECRET]",
      "workspaces": [
        "workspace-id1-to-include",
        "workspace-id2-to-include"
      ]
    }
  ]
}

Supported PowerBI source technologies

PowerBI scanning limitations

PowerBI enables only 500 requests per hour. For more details, see Microsoft documentation on request limitations.
To obtain attribute level lineage from Reports, the following conditions need to be met as per Microsoft documentation:

FAQ

Questions frequently asked regarding the PowerBI scanner:

MacOS was not allowing me to run the PowerQuery parser because it "can’t be opened because Apple cannot check it for malicious software." How can I solve this?

This is a common issue with applications downloaded from the internet which are not recognized by Apple’s security protocols. To allow the PowerQuery parser to run, you can remove the quarantine attribute using the following command in the terminal:

xattr -d com.apple.quarantine parser-macos

Was this page useful?