Dremio Lineage Scanner
Scanned and supported objects
-
Views
Supported sources
Cross lineage is supported for the following Dremio sources:
-
Oracle
-
MS SQL
-
Snowflake
Supported statement types and SQL syntax
Supported statement types are as follows:
-
CREATE VIEW
Supported database objects for Design Time Lineage are as follows:
-
Views
Limitations
The following SQL constructs are currently not supported. This means that statements containing these SQL constructs will not be included in the lineage diagram. Keep in mind that this list is not exhaustive, and other SQL constructs might not be supported as well.
-
Table functions
-
UNNEST
clause -
AT SNAPSHOT
clause
Dremio permissions
The user connecting to the scanner database needs to be able to query these Data Dictionary views:
-
TABLES
-
VIEWS
-
COLUMNS
Scanner configuration
All fields marked with an asterisk (*
) are mandatory.
Property | Description |
---|---|
|
Unique name for the scanner job. |
|
Specifies the source type to be scanned.
Must contain |
|
A human-readable description of the scan. |
|
Arrow Flight JDBC URL. |
|
Dremio username. |
|
Dremio password. Can be encrypted. |
|
Dremio host. Required for REST API calls. |
|
Dremio port. Required for REST API calls |
|
If set to For development purposes or Docker deployments, you can set it to |
|
If set to For development purposes or Docker deployments, you can set it to |
|
List of spaces to include in lineage extraction.
The filter is case insensitive and supports SQL-like wildcards, such as the percent sign ( |
|
List of spaces to exclude from lineage extraction.
Similarly to |
{
"scannerConfigs": [
{
"name": "dremio-localhost",
"sourceType": "DREMIO",
"description": "Scan dremio export files",
"includeSpaces": [],
"excludeSpaces": [
"@admin",
"DevSpace%"
],
"connection": {
"arrowFlightJdbcUrl": "jdbc:arrow-flight-sql://localhost:32010/?useEncryption=false",
"username": "admin",
"password": "some_password",
"apiHost": "localhost",
"apiPort": 9047
}
}
]
}
Additional features
The Dremio scanner supports the same SQL anomaly detection checks as the Snowflake scanner. See SQL DQ (anomaly) detection for more details.
Was this page useful?