Databricks on AWS
This page describes how to configure Ataccama Data Processing Engine (DPE) for the Spark processing integration with Databricks on AWS, including the connections between DPE, your Databricks cluster, and Amazon S3 storage. The Spark processing integration is a legacy option for existing deployments and for data processing plans that can’t run in SQL pushdown.
For new deployments, use the Databricks JDBC connection with SQL pushdown processing instead.
This configuration applies to hybrid deployments, self-managed environments, and Custom Ataccama Cloud deployments. After configuring DPE, add the connection in ONE (see Configure the connection in ONE).
Step 1: Configure access to metadata
Add the following metastore properties to /opt/ataccama/one/dpe/etc/application.properties:
# Databricks as data source configuration
plugin.metastoredatasource.ataccama.one.cluster.databricks.name={ CLUSTER_NAME }
plugin.metastoredatasource.ataccama.one.cluster.databricks.driver-class=com.databricks.client.jdbc.Driver
plugin.metastoredatasource.ataccama.one.cluster.databricks.driver-class-path={ ATACCAMA_ONE_HOME }/dpe/lib/jdbc/DatabricksJDBC42.jar
plugin.metastoredatasource.ataccama.one.cluster.databricks.url={ DBR_JDBC_STRING }
plugin.metastoredatasource.ataccama.one.cluster.databricks.authentication=TOKEN
plugin.metastoredatasource.ataccama.one.cluster.databricks.databricksUrl={ WORKSPACE_URL }
plugin.metastoredatasource.ataccama.one.cluster.databricks.timeout=15m
plugin.metastoredatasource.ataccama.one.cluster.databricks.profiling-sample-limit=100000
plugin.metastoredatasource.ataccama.one.cluster.databricks.full-select-query-pattern=SELECT {columns} FROM {table}
plugin.metastoredatasource.ataccama.one.cluster.databricks.preview-query-pattern=SELECT {columns} FROM {table} LIMIT {previewLimit}
plugin.metastoredatasource.ataccama.one.cluster.databricks.row-count-query-pattern=SELECT COUNT(*) FROM {table}
plugin.metastoredatasource.ataccama.one.cluster.databricks.sampling-query-pattern=SELECT {columns} FROM {table} LIMIT {limit}
plugin.metastoredatasource.ataccama.one.cluster.databricks.dsl-query-preview-query-pattern=SELECT * FROM ({dslQuery}) dslQuery LIMIT {previewLimit}
plugin.metastoredatasource.ataccama.one.cluster.databricks.dsl-query-import-metadata-query-pattern=SELECT * FROM ({dslQuery}) dslQuery LIMIT 0
Replace all placeholder values in { } with your actual configuration values.
|
Additional Unity Catalog configuration
If you are using Unity Catalog, add the following properties to the metastore configuration in /opt/ataccama/one/dpe/etc/application.properties:
# Unity Catalog settings
plugin.metastoredatasource.ataccama.one.cluster.databricks.unity-catalog-enabled=true
plugin.metastoredatasource.ataccama.one.cluster.databricks.catalog-exclude-pattern=^(SAMPLES)|(samples)|(main)$Use catalog-exclude-pattern to mark catalogs as technical and prevent them from being imported to ONE.
The value is a regular expression matching the catalogs that you don’t want to import.
Unity Catalog is supported only with Dedicated access mode. See Unity Catalog.
Driver configuration
This guide assumes the use of DatabricksJDBC42.jar.
If you are using SparkJDBC42.jar, you need to modify the following property values in your configurations:
Using SparkJDBC42.jar
Replace these properties in /opt/ataccama/one/dpe/etc/application.properties:
# Replace this:
plugin.metastoredatasource.ataccama.one.cluster.databricks.url=jdbc:databricks://{ DBR_JDBC_STRING }
plugin.metastoredatasource.ataccama.one.cluster.databricks.driver-class=com.databricks.client.jdbc.Driver
# With this:
plugin.metastoredatasource.ataccama.one.cluster.databricks.url=jdbc:spark://{ DBR_JDBC_STRING }
plugin.metastoredatasource.ataccama.one.cluster.databricks.driver-class=com.simba.spark.jdbc.DriverStep 2: Configure job submission to Databricks cluster
Spark processing basic settings
Add the following properties to /opt/ataccama/one/dpe/etc/application.properties:
# Setup launch script for Spark (specify .sh script for Linux, .bat script for Windows)
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.exec=bin/databricks/exec_databricks.sh
# Enable debugging if needed:
#plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.debug=trueLibraries and classpath configuration
Add the following properties to /opt/ataccama/one/dpe/etc/application.properties:
# Relative to temp/jobs/${jobId} from repository root directory
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.cpdelim=;
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.cp.runtime=../../../lib/runtime/*;../../../lib/jdbc/*;../../../lib/jdbc_ext/*;../../../lib/runtime/ext/*;../../../lib/runtime/extra/*
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.cp.ovr=../../../lib/ovr/*.jar
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.cp.databricks=../../../lib/runtime/databricks/*
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.cp.!exclude=!kryo-*.jar;!scala*.jar;!cif-dtdb*.jar;!dropbox*.jar;!gdrive*.jar;!onedrive*.jar;!poi-*.jar;!salesforce*.jar;!saprfc*.jar;!keycloak*.jar;!swagger*.jar;!apollo*.jar;!sso*.jar;!microsoft*.jar;!jsch*.jar;!tomcat-embed*.jar;!jboss*.jar;!janino*.jar;!dremio-jdbc*.jar;!CassandraJDBC*.jar;!GoogleBigQueryJDBC*.jar;!HiveJDBC*.jar;!cdata.jdbc.redis.jar;!ngdbc*.jar;!gcs*.jar;!gax*.jar;!google-cloud*.jar;!googleapis*.jar;!jexcelapi*.jar;!msoffice*.jar;!dppapi*;!dpm*;!cometd*;!dmm*;!jetty*;!authentication-keycloak-*.jar
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.libraries.!exclude=!grpc*;!opencensus*;!netty*;!google*;!kotlin*
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.lcp.!exclude=!guava-11*
# Hadoop 3 library path
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.lcp.ext=../../../lib/runtime/hadoop3/*The SPARK.libraries.!exclude property keeps the listed libraries off the Databricks cluster without removing them from the classpath that DPE uses to start the job.
Do not move these entries to SPARK.cp.!exclude: DPE needs the gRPC and Netty libraries locally to report job progress, and Spark jobs fail at startup without them.
For details about the two exclusion properties, see Databricks Configuration > Cluster libraries.
If you use a job cluster, see also the library limit in Databricks job cluster configuration.
General cluster settings
Add the following properties to /opt/ataccama/one/dpe/etc/application.properties:
# Cluster ID or name (uses newest cluster if not specified)
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.cluster={ CLUSTER_NAME }
# Number of concurrent job runs
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.job.max_concurrent_runs=150
# Force file copying regardless of modification time
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.fsc.force=falseAuthentication configuration
Add the following properties to /opt/ataccama/one/dpe/etc/application.properties:
# Databricks platform access
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.url={ WORKSPACE_URL }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.authType=PERSONAL_TOKEN
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.dbr.token={ DBR_TOKEN }Amazon Web Services configuration
Add the following properties to /opt/ataccama/one/dpe/etc/application.properties:
# S3 folder for storing libraries and files
# Mount point in DBFS:
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.point=/mnt/ataccama
# S3 mount URL:
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.url=s3a://{ BUCKET_NAME }/{ CLUSTER_NAME }/tmp/dbr
# User account for mounting filesystem
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.s3n.awsAccessKeyId={ ACCESS_KEY_ID }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.s3n.awsSecretAccessKey={ SECRET_ACCESS_KEY }
# User account for Ataccama job configuration
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.s3a.access.key={ ACCESS_KEY_ID }
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.s3a.secret.key={ SECRET_ACCESS_KEY }
# Alternative authentication method (IAM role-based) - use in place of the four properties above:
#plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.mount.conf.fs.s3n.aws.credentials.provider=com.amazonaws.auth.InstanceProfileCredentialsProvider
#plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.s3a.aws.credentials.provider=com.amazonaws.auth.InstanceProfileCredentialsProvider
# AWS-specific optimizations
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.s3a.fast.upload=true
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.conf.fs.s3a.fast.upload.buffer=bytebuffer
# Keystore options for Spark via JAVA_OPTS
plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.env.JAVA_OPTS=-Dproperties.encryption.keystore=../../../etc/one-encryption.jceks -Dproperties.encryption.keystore.passwordFile=../../../etc/one-encryption.passwd -Dinternal.encryption.keystore=../../../etc/one-encryption.jceks -Dinternal.encryption.keystore.passwordFile=../../../etc/one-encryption.passwd
Replace all placeholder values enclosed in { } with your actual configuration values.
|
Configure the connection in ONE
After DPE is configured and restarted, add the Databricks connection in ONE: create a source, add a Metastore connection with Spark enabled, add credentials, and test and publish the connection. For the full procedure, see Add the Databricks connection in ONE.
For Databricks on AWS, select Token credentials and provide the personal access token you configured in Step 2: Configure job submission to Databricks cluster.
Was this page useful?