Databricks Configuration

Sets the script for customizing how Spark jobs are launched. For example, you can change which shell script is used to start the job. The script is located in the bin/databricks folder in the root directory of the application.

The default value for Linux is ${ataccama.path.root}/bin/databricks/exec_databricks.sh.

Points to the location of Ataccama licenses needed for Spark jobs. The property should be configured only if licenses are stored outside of the standard locations, that is the home directory of the user and the folder runtime/license_keys.

Default value: ../../../lib/runtime/license_keys.

Sets the JAVA_HOME variable for running Spark jobs. For Spark jobs, JDK version 8 is required.

Default value: /usr/java/jdk1.8.0_65.

Enables the debugging mode. The debug mode shows the driver and the executor classloaders used in Spark, which lets you detect any classpath conflicts between the Ataccama classpath and the Spark classpath in the Spark driver and in the Spark executors.

Default value: true.

Tests whether the cluster has the correct mount and libraries. Turning the cluster check off speeds up launching jobs when the cluster libraries have already been installed and there are no changes made to the runtime. In that case, checking is skipped.

Default value: true.

Enables automatic mounting of internal Azure Storage to Azure Databricks, which simplifies the installation and configuration process. This requires saving of Azure Storage credentials in configuration settings.

Default value: true.

Specifies how many jobs can be run in parallel.

Default value: 150.

If set to true, input files from the executor are synchronized with the files on the cluster at each launch or processing. If set to false, the cached versions of files and lookups are used instead, unless their respective checksums have changed.

Default value: true.

If set to true, initiated jobs are added to the queue if the maximum number of concurrent jobs is exceeded. If set to false, jobs are automatically canceled instead.

Default value: true.

The folder in the Databricks File System that is used as a mount point for the Azure Data Lake Storage Gen2 folder.

Default value: /mnt/ataccama.

The location of the ADLS Gen2 folder for storing libraries and files for processing. The folder is then mounted to the directory in the Databricks File System defined in the property mount.point.

Default value: adl://…​/tmp/dbr.

Defines where lookup files are created. If set to true, the builder creates lookup file on the driver side, using RDD.toLocalIterator. If set to false, lookup files are built on the Spark executor side. Building lookup files on the Spark executor can be beneficial when working with large volumes of data.

Default value: false.

The classpath delimiter used for libraries for Spark jobs.

Default value: ; (semicolon).

The libraries needed for Spark jobs. The files in the classpath are copied to the Databricks cluster. Spark jobs are stored as tmp/jobs/${jobId} in the root directory of the application.

For Spark processing of Azure Data Lake Storage Gen2, an additional set of libraries is required: ../../../lib/runtime/ext/*.

Default value: ../../../lib/runtime/*;../../../lib/jdbc/\*;../../../lib/jdbc_ext/*.

Points to additional override libraries that can be used for miscellaneous fixes.

Default value: ../lib/ovr/*.jar.

Points to additional Databricks libraries.

Default value: ../lib/runtime/databricks/*.

Excludes certain Java libraries and drivers that are in the same classpath but are not needed for Spark processing. Each library needs to be prefixed by an exclamation mark (!). Libraries are separated by the delimiter defined in the cpdelim property.

Default value: !atc-hive-jdbc*;!hive-jdbc*;!slf4j-api-*.jar;!kryo-*.jar;!scala*.jar;!commons-lang3*.jar;!cif-dtdb*.jar.

Excludes certain Java libraries and drivers that are in the local classpath. Each library needs to be prefixed by an exclamation mark (!). Libraries are separated by the delimiter defined in the cpdelim property.

Default value: !guava-11*;!hive-exec*.

Points to the local external libraries.

Default value: ../lib/runtime/hadoop3/*.

Excludes certain Databricks libraries from file synchronization between the ONE runtime engine and the libraries installed in Databricks.

Libraries that are installed within the cluster and matched by this property are not uninstalled before the job run even if they are not part of ONE runtime. Other libraries (that is, those that are not part of ONE runtime and not matched by the property) are uninstalled before the job start.

The property can also be set to a valid regular expression.

Default value: dbfs:/FileStore/job-jars/.*.

Optional list of credential providers. Should be specified as a comma-separated list.

If unspecified, a default list of credential provider classes is queried in sequence.

The access key identifier for the Amazon S3 user account. Used when mounting the file system.

The secret access key for the Amazon S3 user account. Used when mounting the file system.

The access key identifier for the Amazon S3 user account. Used when configuring Ataccama jobs.

The secret access key for the Amazon S3 user account. Used when configuring Ataccama jobs.

Enables fast upload of data. For more information, see How S3A Writes Data to S3.

Default value: true.

Configures how data is buffered during fast upload. Must be set to bytebuffer for AWS. For more information, see How S3A Writes Data to S3.

Default value: bytebuffer.

The type of access token provider. Used when mounting the file system.

Default value: ClientCredential.

The token endpoint. A request is sent to this URL to refresh the access token. Used when mounting the file system.

Default value: <https://login.microsoftonline.com/…​/oauth2/token>.

The client identifier of the ADLS account. Used when mounting the file system.

The client secret of the ADLS account. Used when mounting the file system.

Enables the Azure Data Lake File System.

Default value: org.apache.hadoop.fs.adl.AdlFileSystem.

The type of access token provider. Used when running Ataccama jobs.

Default value: ClientCredential.

The token endpoint used for refreshing the access token. Used when running Ataccama jobs.

Default value: <https://login.microsoftonline.com/…​/oauth2/token>.

The client identifier of the ADLS account. Used when running Ataccama jobs.

The client secret of the ADLS account. Used when running Ataccama jobs.

Perpetually running generic cluster (with auto-sleep) that is configured once and can run multiple Spark jobs at once.

The Databricks job scheduler creates a new job cluster whenever a job needs to be run and terminates the cluster after the job is completed.

To enable the job cluster, set the property to true.

Default value: false.

Checks whether the configuration has changed since the last job cluster was created.

Default value: false.

Defines the suffix of jobs run on the cluster. For example, if jobSuffix=jobCluster, the jobs name could be DbrRunnerMain-jobCluster.

Default value: jobCluster.

Defines the Spark version used on the cluster.

plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.job-cluster.resources.node_type_id=Standard_D3_v2

Or

plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.job-cluster.resources.instance_pool_id=0426-110823-bided16-pool-x2cinbmv

Sets the driver and the worker type. This property does not have a default value.

Must be set to node_type_id or, if you are using a pooling cluster, an instance_pool_id.

If you are using an instance_pool_id, it is used for both the driver and worker nodes.

Defines the number of workers If you use autoscaling, there is no need to define this property.

Default value: 3.

Enables autoscaling. If set to true, the number of workers is scaled automatically depending on the resources needed at the time.

Default value: false.

Defines the minimum number of workers when using autoscaling. Only applies if autoscaling is enabled.

Default value: 1.

Defines the maximum number of workers when using autoscaling. Only applies if autoscaling is enabled.

Default value: 2.

When enabled, this cluster dynamically acquires additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly. For more information, see the official Databricks documentation.

SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

An object containing a set of tags for cluster resources. Databricks tags all cluster resources (such as AWS instances and EBS volumes) with these tags in addition to default_tags.

DBFS destination. For example, dbfs:/my/path.

S3 location of cluster log. destination and either region or warehouse must be provided. For example, { "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } }.

The max price for AWS spot instances, as a percentage of the corresponding instance type’s on-demand price. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose max price percentage matches this field will be considered. For safety, this field is limited to be no more than 10000.

The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. This value does not affect cluster size and cannot be mutated over the lifetime of a cluster.

Availability type used for all subsequent nodes past the first_on_demand ones. If first_on_demand is zero, this availability type will be used for the entire cluster. Examples are SPOT, ON_DEMAND, and SPOT_WITH_FALLBACK.

Workspace location of init script. Destination must be provided. For example, { "workspace" : { "destination" : "/Users/someone@domain.com/init_script.sh" } }.

S3 location of init script. Destination and either region or warehouse must be provided. For example, { "s3": { "destination" : "s3://init_script_bucket/prefix", "region" : "us-west-2" } }.

(Deprecated) DBFS location of init script. Destination must be provided. For example, { "dbfs" : { "destination" : "dbfs:/home/init_script" } }.

An object containing a set of optional, user-specified environment variable key-value pairs. Key-value pair of the form (X,Y) are exported as is (i.e., export X='Y') while launching the driver and workers. For example,: {"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}.