Azure Databricks

This page describes how to configure Ataccama Data Processing Engine (DPE) for the Spark processing integration with Azure Databricks, including the connections between DPE, your Databricks cluster, and Azure Data Lake Storage (ADLS) Gen2. The Spark processing integration is a legacy option for existing deployments and for data processing plans that can’t run in SQL pushdown.

For new deployments, use the Databricks JDBC connection with SQL pushdown processing instead.

This configuration applies to hybrid deployments, self-managed environments, and Custom Ataccama Cloud deployments. After configuring DPE, add the connection in ONE (see Configure the connection in ONE).

Prerequisites

Cluster configuration

The following property must be added to your Databricks cluster’s Spark configuration in the Azure Databricks workspace:

spark.serializer org.apache.spark.serializer.KryoSerializer

To add this property:

In your Azure Databricks workspace, go to Compute.
Select your cluster or create a new one.
In the cluster configuration, expand Advanced Options.
In the Spark tab, add the property above to the Spark Config field.
Save the cluster configuration and restart the cluster if it’s already running.

ADLS container permissions

The container must have either Storage Blob Data Owner or Storage Blob Data Contributor permissions for the blob storage and the user account or Entra ID service principal. To set up these permissions, refer to the Microsoft article on authorization errors.

Required connections

To work with the Databricks cluster, you need to establish the following connections:

Processing virtual machine (VM) with DPE to Databricks cluster
Processing VM with DPE to Ataccama dedicated ADLS Gen2 container
Databricks cluster to Ataccama dedicated ADLS Gen2 container

You are responsible for setting up the connection from the Databricks cluster to your data lake.

Step 1: Metadata configuration

Configure the metadata connection in /opt/ataccama/one/dpe/etc/application.properties. This enables Ataccama ONE to browse and profile data from your Databricks cluster.

Method 1: Token authentication

Use a Databricks personal access token for authentication:

plugin.metastoredatasource.ataccama.one.cluster.databricks.name={ CLUSTER_NAME }
plugin.metastoredatasource.ataccama.one.cluster.databricks.driver-class=com.databricks.client.jdbc.Driver
plugin.metastoredatasource.ataccama.one.cluster.databricks.driver-class-path={ ATACCAMA_ONE_HOME }/dpe/lib/jdbc/DatabricksJDBC42.jar
plugin.metastoredatasource.ataccama.one.cluster.databricks.url={ DBR_JDBC_STRING }
plugin.metastoredatasource.ataccama.one.cluster.databricks.authentication=TOKEN
plugin.metastoredatasource.ataccama.one.cluster.databricks.databricksUrl={ WORKSPACE_URL }
plugin.metastoredatasource.ataccama.one.cluster.databricks.timeout=15m
plugin.metastoredatasource.ataccama.one.cluster.databricks.profiling-sample-limit=100000
plugin.metastoredatasource.ataccama.one.cluster.databricks.full-select-query-pattern=SELECT {columns} FROM {table}
plugin.metastoredatasource.ataccama.one.cluster.databricks.preview-query-pattern=SELECT {columns} FROM {table} LIMIT {previewLimit}
plugin.metastoredatasource.ataccama.one.cluster.databricks.row-count-query-pattern=SELECT COUNT(*) FROM {table}
plugin.metastoredatasource.ataccama.one.cluster.databricks.sampling-query-pattern=SELECT {columns} FROM {table} LIMIT {limit}
plugin.metastoredatasource.ataccama.one.cluster.databricks.dsl-query-preview-query-pattern=SELECT * FROM ({dslQuery}) dslQuery LIMIT {previewLimit}
plugin.metastoredatasource.ataccama.one.cluster.databricks.dsl-query-import-metadata-query-pattern=SELECT * FROM ({dslQuery}) dslQuery LIMIT 0
spring.profiles.active=SPARK_DATABRICKS

Method 2: Service principal authentication

Use an Entra ID service principal for authentication. Your clientId is the same as your applicationId.

# Basic metadata connection settings
plugin.metastoredatasource.ataccama.one.cluster.databricks.name=Databricks Cluster
plugin.metastoredatasource.ataccama.one.cluster.databricks.url=jdbc:databricks://{ DBR_JDBC_STRING }
plugin.metastoredatasource.ataccama.one.cluster.databricks.databricksUrl={ WORKSPACE_URL }
plugin.metastoredatasource.ataccama.one.cluster.databricks.driver-class=com.databricks.client.jdbc.Driver
plugin.metastoredatasource.ataccama.one.cluster.databricks.driver-class-path=${ataccama.path.root}/lib/runtime/jdbc/databricks/*
plugin.metastoredatasource.ataccama.one.cluster.databricks.timeout=15m

# Authentication settings
plugin.metastoredatasource.ataccama.one.cluster.databricks.authentication=INTEGRATED
plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.authType=AAD_CLIENT_CREDENTIAL
plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.tenantId={ TENANT_ID }
plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.clientId={ CLIENT_ID }
plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.clientSecret={ CLIENT_SECRET }
plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default

# Query patterns
plugin.metastoredatasource.ataccama.one.cluster.databricks.full-select-query-pattern=SELECT {columns} FROM {table}
plugin.metastoredatasource.ataccama.one.cluster.databricks.preview-query-pattern=SELECT {columns} FROM {table} LIMIT {previewLimit}
plugin.metastoredatasource.ataccama.one.cluster.databricks.row-count-query-pattern=SELECT COUNT(*) FROM {table}
plugin.metastoredatasource.ataccama.one.cluster.databricks.sampling-query-pattern=SELECT {columns} FROM {table} LIMIT {limit}
plugin.metastoredatasource.ataccama.one.cluster.databricks.dsl-query-preview-query-pattern=SELECT * FROM ({dslQuery}) dslQuery LIMIT {previewLimit}
plugin.metastoredatasource.ataccama.one.cluster.databricks.dsl-query-import-metadata-query-pattern=SELECT * FROM ({dslQuery}) dslQuery LIMIT 0

# Enable Spark profile
spring.profiles.active=SPARK_DATABRICKS

Additional Unity Catalog configuration

If you are using Unity Catalog, add these properties to /opt/ataccama/one/dpe/etc/application.properties:

# Unity Catalog settings
plugin.metastoredatasource.ataccama.one.cluster.databricks.unity-catalog-enabled=true
plugin.metastoredatasource.ataccama.one.cluster.databricks.catalog-exclude-pattern=^(SAMPLES)|(samples)|(main)$

Use catalog-exclude-pattern to mark catalogs as technical and prevent them from being imported to ONE. The value is a regular expression matching the catalogs that you don’t want to import.

Unity Catalog is supported only with Dedicated access mode. See Unity Catalog.

Driver configuration

This guide assumes the use of DatabricksJDBC42.jar. If you are using SparkJDBC42.jar, you need to modify the following property values in your configurations:

Using SparkJDBC42.jar

Replace these properties in /opt/ataccama/one/dpe/etc/application.properties:

# Replace this:
plugin.metastoredatasource.ataccama.one.cluster.databricks.url=jdbc:databricks://{ DBR_JDBC_STRING }
plugin.metastoredatasource.ataccama.one.cluster.databricks.driver-class=com.databricks.client.jdbc.Driver

# With this:
plugin.metastoredatasource.ataccama.one.cluster.databricks.url=jdbc:spark://{ DBR_JDBC_STRING }
plugin.metastoredatasource.ataccama.one.cluster.databricks.driver-class=com.simba.spark.jdbc.Driver

Step 2: Spark job configuration

Configure Spark processing in /opt/ataccama/one/dpe/etc/application-SPARK_DATABRICKS.properties. This enables running data quality and processing jobs on your Databricks cluster.

Spark processing basic settings

Add the following properties to /opt/ataccama/one/dpe/etc/application-SPARK_DATABRICKS.properties:

# Cluster selection
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.cluster={ CLUSTER_NAME }

# Databricks workspace URL
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.url=https://{ YOUR_DATABRICKS_CLUSTER }.azuredatabricks.net

# Mount point for temporary storage
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.point=/ataccama/dbr-tmp
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.url=abfss://{ CONTAINER_NAME }@{ STORAGE_NAME }.dfs.core.windows.net/{ PATH_TO_FOLDER }

Libraries and classpath configuration

Add the following properties to /opt/ataccama/one/dpe/etc/application-SPARK_DATABRICKS.properties:

# File system implementation
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.adl.impl=org.apache.hadoop.fs.adl.AdlFileSystem

# Relative to temp/jobs/${jobId} from repository root directory
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.cpdelim=;
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.cp.runtime=../../../lib/runtime/*;../../../lib/jdbc/*;../../../lib/jdbc_ext/*;../../../lib/runtime/ext/*;../../../lib/runtime/extra/*
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.cp.ovr=../../../lib/ovr/*.jar
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.cp.databricks=../../../lib/runtime/databricks/*
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.cp.!exclude=!kryo-*.jar;!scala*.jar;!cif-dtdb*.jar;!dropbox*.jar;!gdrive*.jar;!onedrive*.jar;!poi-*.jar;!salesforce*.jar;!saprfc*.jar;!keycloak*.jar;!swagger*.jar;!apollo*.jar;!sso*.jar;!microsoft*.jar;!jsch*.jar;!tomcat-embed*.jar;!jboss*.jar;!janino*.jar;!dremio-jdbc*.jar;!CassandraJDBC*.jar;!GoogleBigQueryJDBC*.jar;!HiveJDBC*.jar;!cdata.jdbc.redis.jar;!ngdbc*.jar;!gcs*.jar;!gax*.jar;!google-cloud*.jar;!googleapis*.jar;!jexcelapi*.jar;!msoffice*.jar;!dppapi*;!dpm*;!cometd*;!dmm*;!jetty*;!authentication-keycloak-*.jar
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.libraries.!exclude=!grpc*;!opencensus*;!netty*;!google*;!kotlin*
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.lcp.!exclude=!guava-11*

# Hadoop 3 library path
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.lcp.ext=../../../lib/runtime/hadoop3/*

The SPARK.libraries.!exclude property keeps the listed libraries off the Databricks cluster without removing them from the classpath that DPE uses to start the job. Do not move these entries to SPARK.cp.!exclude: DPE needs the gRPC and Netty libraries locally to report job progress, and Spark jobs fail at startup without them. For details about the two exclusion properties, see Databricks Configuration > Cluster libraries. If you use a job cluster, see also the library limit in Databricks job cluster configuration.

Authentication configuration

Choose one of the following authentication methods and add the properties to /opt/ataccama/one/dpe/etc/application-SPARK_DATABRICKS.properties:

Method 1: Token authentication

# Databricks authentication
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.authType=PERSONAL_TOKEN
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.token={ DBR_TOKEN }

Method 2: Service principal authentication

# Databricks authentication
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.authType=AAD_CLIENT_CREDENTIAL
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.aad.tenantId={ TENANT_ID }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.aad.clientId={ CLIENT_ID }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.aad.clientSecret={ CLIENT_SECRET }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.aad.resource=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d

Azure configuration

Configure ADLS Gen2 access for both DPE and Databricks cluster by adding the following properties to /opt/ataccama/one/dpe/etc/application-SPARK_DATABRICKS.properties:

# Connection from Processing VM with DPE to ADLS Gen2
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.azure.account.auth.type=OAuth
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.azure.account.oauth.provider.type=org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.azure.account.oauth2.client.id={ ABFSS_CLIENT_ID }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.azure.account.oauth2.client.secret={ ABFSS_CLIENT_SECRET }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.azure.account.oauth2.client.endpoint=https://login.microsoftonline.com/{ TENANT_ID }/oauth2/token

# Connection from Databricks cluster to ADLS Gen2
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.azure.account.auth.type=OAuth
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.azure.account.oauth.provider.type=org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.azure.account.oauth2.client.id={ ABFSS_CLIENT_ID }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.azure.account.oauth2.client.secret={ ABFSS_CLIENT_SECRET }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.azure.account.oauth2.client.endpoint=https://login.microsoftonline.com/{ TENANT_ID }/oauth2/token

Replace all placeholder values in { } with your actual configuration values.

Azure Key Vault integration (optional)

You can use Azure Key Vault to manage secrets instead of entering plain text values in the configuration. Connect to Key Vault using either direct connection credentials or through Managed Service Identity (MSI).

Key Vault direct connection

Add the following properties to /opt/ataccama/one/dpe/etc/application.properties to configure direct Key Vault authentication:

plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.keyvault.authType=AAD_CLIENT_CREDENTIAL
plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.keyvault.vaultUrl=https://{ VAULT_NAME }.vault.azure.net/
plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.keyvault.tenantId={ TENANT_ID }
plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.keyvault.clientId={ CLIENT_ID }
plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.keyvault.clientSecret={ CLIENT_SECRET }

Managed Service Identity (MSI)

plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.keyvault.authType=AAD_MANAGED_IDENTITY

plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.aad.keyvault.vaultUrl=https://{ VAULT_NAME }.vault.azure.net/
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.aad.keyvault.clientId={ CLIENT_ID }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.aad.keyvault.tenantId={ TENANT_ID }
plugin.metastoredatasource.ataccama.one.cluster.databricks.aad.keyvault.resource=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d

Retrieve the client ID using:

curl 'http://169.254.169.254/metadata/identity/oauth2/token?resource=https://vault.azure.net&api-version=2018-02-01' -H "Metadata: true" | jq

Using Key Vault secrets

Reference secrets in your configuration:

# Example: Using Key Vault for ADLS credentials
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.azure.account.auth.type=OAuth
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.azure.account.oauth.provider.type=org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.azure.account.oauth2.client.id=keyvault:SECRET:{ SECRET_NAME }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.azure.account.oauth2.client.secret=keyvault:SECRET:{ SECRET_NAME }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.azure.account.oauth2.client.endpoint=https://login.microsoftonline.com/{ TENANT_ID }/oauth2/token

plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.azure.account.auth.type=OAuth
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.azure.account.oauth.provider.type=org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.azure.account.oauth2.client.id=keyvault:SECRET:{ SECRET_NAME }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.azure.account.oauth2.client.secret=keyvault:SECRET:{ SECRET_NAME }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.azure.account.oauth2.client.endpoint=https://login.microsoftonline.com/{ TENANT_ID }/oauth2/token

Step 3: Restart DPE

After completing the configuration, restart DPE:

systemctl restart dpe

Configure the connection in ONE

After DPE is configured and restarted, add the Databricks connection in ONE: create a source, add a Metastore connection with Spark enabled, add credentials, and test and publish the connection. For the full procedure, see Add the Databricks connection in ONE.

For Azure Databricks, select the credential type matching the authentication method you configured in Step 1: Metadata configuration:

Token credentials for token authentication (Method 1).
Integrated credentials for service principal authentication (Method 2).

Was this page useful?