Dremio Lineage Scanner

Scanned and supported objects

Views

Supported sources

Cross lineage is supported for the following Dremio sources:

Oracle
MS SQL
Snowflake

Supported statement types and SQL syntax

Supported statement types are as follows:

CREATE VIEW

Supported database objects for Design Time Lineage are as follows:

Views

Limitations

The following SQL constructs are currently not supported. This means that statements containing these SQL constructs will not be included in the lineage diagram. Keep in mind that this list is not exhaustive, and other SQL constructs might not be supported as well.

Table functions
UNNEST clause
AT SNAPSHOT clause

Supported connectivity

Connector type: JDBC.
Authentication method: Username and password.

Dremio permissions

The user connecting to the scanner database needs to be able to query these Data Dictionary views:

TABLES
VIEWS
COLUMNS

Scanner configuration

All fields marked with an asterisk (*) are mandatory.

Property Description

Property	Description
`name*`	Unique name for the scanner job.
`sourceType*`	Specifies the source type to be scanned. Must contain `DREMIO`.
`description*`	A human-readable description of the scan.
`arrowFlightJdbcUrl*`	Arrow Flight JDBC URL.
`username*`	Dremio username.
`password*`	Dremio password. Can be encrypted.
`apiHost*`	Dremio host. Required for REST API calls.
`apiPort*`	Dremio port. Required for REST API calls
`useSSLApiCalls`	If set to `true`, SSL API requests are enabled. Default value: `true`. For development purposes or Docker deployments, you can set it to `false`.
`acceptSelfSignedSSLCertificate`	If set to `true`, self-signed SSL certificates are accepted. Default value: `false`. For development purposes or Docker deployments, you can set it to `true`. Used only when `useSSLApiCalls` is set to `true`.
`includeSpaces`	List of spaces to include in lineage extraction. The filter is case insensitive and supports SQL-like wildcards, such as the percent sign (`%`) and underscore (`_`). For example, to include all schemas ending with `PROD`, add the filter `%PROD`.
`excludeSpaces`	List of spaces to exclude from lineage extraction. Similarly to `includeSpaces`, this filter is case insensitive and supports SQL wildcards.

name*

Unique name for the scanner job.

sourceType*

Specifies the source type to be scanned. Must contain DREMIO.

description*

A human-readable description of the scan.

arrowFlightJdbcUrl*

Arrow Flight JDBC URL.

username*

Dremio username.

password*

Dremio password. Can be encrypted.

apiHost*

Dremio host. Required for REST API calls.

apiPort*

Dremio port. Required for REST API calls

useSSLApiCalls

If set to true, SSL API requests are enabled. Default value: true.

For development purposes or Docker deployments, you can set it to false.

acceptSelfSignedSSLCertificate

If set to true, self-signed SSL certificates are accepted. Default value: false.

For development purposes or Docker deployments, you can set it to true. Used only when useSSLApiCalls is set to true.

includeSpaces

List of spaces to include in lineage extraction. The filter is case insensitive and supports SQL-like wildcards, such as the percent sign (%) and underscore (_). For example, to include all schemas ending with PROD, add the filter %PROD.

excludeSpaces

List of spaces to exclude from lineage extraction. Similarly to includeSpaces, this filter is case insensitive and supports SQL wildcards.

Dremio Docker deployment scanner example configuration

{
   "scannerConfigs": [
      {
         "name": "dremio-localhost",
         "sourceType": "DREMIO",
         "description": "Scan dremio export files",
         "includeSpaces": [],
         "excludeSpaces": [
            "@admin",
            "DevSpace%"
         ],
         "connection": {
            "arrowFlightJdbcUrl": "jdbc:arrow-flight-sql://localhost:32010/?useEncryption=false",
            "username": "admin",
            "password": "some_password",
            "apiHost": "localhost",
            "apiPort": 9047
         }
      }
   ]
}

Additional features

The Dremio scanner supports the same SQL anomaly detection checks as the Snowflake scanner. See SQL DQ (anomaly) detection for more details.

Was this page useful?