DPE Configuration

Enables debug logging mode. If set to true, job logs are shown in the server logs and temporary files are not cleaned immediately from the job workspace. The scheduler removes temporary job files later depending on the settings.

Default value: false.

Used to send information to Data Processing Manager (DPM) about the location of DPE. If DPM cannot reach DPE by the machine hostname, this property overrides that hostname.

If not set, the hostname is determined by trying to resolve the DNS records for the localhost address.

Default value: localhost.

Used to send information to DPM about the location of DPE. If DPM cannot reach DPE by the gRPC port, this property overrides that port.

Default value: 8532.

The HTTP server port for DPE.

Default value: 8034.

A comma-separated list of environments in which this instance of DPE can be used. In the current release, this only affects user restrictions when accessing the file system data sources.

A meaningful name for DPE that is used in DPM Admin Console. This is especially useful in firewall-friendly mode as it makes it easier to identify DPE instances. We recommend using alphanumeric characters without spaces.

If not set, the default value matches the hostname or the URL and the port of the DPE server. In the firewall-friendly mode, the default value is 0.0.0.0 unless modified.

When set to true , alerts are sent to Ataccama Support when an on-premise DPE disconnects from ONE.

Default value: false.

The URL of the server where Keycloak is running.

Default value: http://localhost:8080/auth.

The name of the Keycloak realm.

Default value: ataccamaone.

The client identifier used to verify the admin user’s authorization token.

Default value: dpe-admin-client.

The secret key of the client identifier for the admin account. Secret keys can be generated using Keycloak.

Default value: dpe-admin-client-s3cret.

The client identifier. Used to verify a user’s authorization token and to log in a user.

Default value: dpe-token-client.

The secret key of the client. Secret keys can be generated using Keycloak.

Default value: dpe-token-client-s3cret.

Specifies the issuer of the JWT token. Typically, Keycloak uses the URL of the realm as the token issuer.

Default value: ${ataccama.authentication.keycloak.server-url}/realms/${ataccama.authentication.keycloak.realm}.

The type of client token authentication. Possible values: BASIC, SIGNED_JWT, SECRET_JWT.

Default value: BASIC.

Points to the keystore file used for SIGNED_JWT authentication.

The type of the keystore used for SIGNED_JWT authentication. Possible values: JKS, PKCS12.

Default value: JKS.

The password of the keystore used for SIGNED_JWT authentication. Used if the keystore is encrypted.

The private key name specified in the keystore used for SIGNED_JWT authentication.

The default value is the client identifier.

The password for the private key. Used if the private key is encrypted.

The default value is the keystore password.

Specifies for how long the JWT token used for authentication in Keycloak remains valid. Used for SIGNED_JWT and SECRET_JWT authentication strategies.

Default value: 15s. For a full list of accepted units, see Duration units.

Defines how often DPE checks the connection to DPM. If there is no reply from DPM, DPE tries to register again with DPM. Expressed in milliseconds.

Default value: 5000.

The name of the gRPC channel that DPE uses to communicate with DPM.

Default value: dpm.

The IP address or the URL of the server where DPM is running.

Default value: localhost.

The port where the gRPC server is running.

Default value: 8531.

Enables TLS for the connection to DPM.

Default value: false.

Defines whether mutual TLS authentication is enabled. Possible values: NONE, OPTIONAL, REQUIRED.

When set to OPTIONAL, if the server receives an mTLS request, it attempts to authenticate the request using mTLS.

Disabled by default.

Specifies whether the gRPC client should verify the certificate of the server with which they communicate. Used if mTLS is enabled.

The full path to the TLS certificate, for example, file:/path/to/client.crt.

The full path to the private key of the certificate, for example, file:/path/to/client.key.

The full path to the public certificate of the root certificate authority, for example, file:/path/to/rootCA.crt.

The full path to a truststore file that contains public keys and certificates against which the client verifies the certificates from the server, for example, file:path/to/trust/cert/trust-store.pfx. Used only when tls.trust-all is disabled.

The password for the truststore. Used if the truststore is encrypted.

The type of truststore. Possible values: PKCS12, JCEKS.

If set to true, an HTTP client is created for communicating with MDM.

Default value: true.

Enables TLS authentication when communicating with MDM.

The full path to the truststore, for example, file:/path/to/truststore.

The password for the truststore.

The location of the plugins folder.

Default value: ./plugin.

Points to the folder containing the JDBC drivers used by DPE.

Default value: ${ataccama.path.root}/lib/jdbc.

The connection entity name used in the metadata model (MMD) for local file systems.

The connection entity name used in the metadata model (MMD) for ONE MDM data source.

The connection entity name used in the metadata model (MMD) for ONE RDM data source.

The connection entity name used in the metadata model (MMD) for S3 data source.

The number of columns processed at once when fetching basic statistics. This property is irrelevant if mergedStatistics=true.

Default value: 1.

The number of columns processed at once when fetching frequencies or merged statistics.

Default value: 1.

Snowflake uses different defaults for tables and views. See Snowflake query pushdown processing.

Get both basic statistics and frequencies at once (for a group of columns).

Default value: true.

The number of queries processed at once. This setting limits the number of queries sent from DPE to the data source for profiling. It is applied globally for the whole DPE and the number of allowed queries is shared between all pushdown profiling jobs for that data source.

If the connection pool is exhausted, DPE waits for a connection to become available instead of failing the query.

Default value: 32.

The maximum length of string values stored as samples in profiling results (for example, in frequency analysis or quantiles and other value distributions). This doesn’t affect calculations such as length statistics, masks, or patterns, which use the full string to compute.

When a string exceeds this limit, it is truncated as follows: the beginning of the string is preserved in the length specific, the number of omitted characters is reported, and the rest of the string is displayed as a hash value. The hash ensures uniqueness of truncated values.

Default value: 10000.

For non-pushdown profiling, see com.ataccama.profiling.maxStringLength in Profiling.

The maximum length of string values stored as samples in profiling results (for example, in frequency analysis or quantiles and other value distributions). This doesn’t affect calculations such as length statistics, masks, or patterns, which use the full string to compute.

When a string exceeds this limit, it is truncated as follows: the beginning of the string is preserved in the length specified, the number of omitted characters is reported, and the rest of the string is displayed as a hash value. The hash ensures uniqueness of truncated values.

Default value: 10000.

The maximum number of threads that can be dedicated to closing the clients that have been evicted.

Default value: 10.

If set to true, the cache statistics are recorded, which can be useful for performance monitoring. To turn it off, set the property to false.

Default value: true.

The maximum size of the DataConnect cache for all data sources. The value refers to the number of cached client instances, where each instance corresponds to a combined definition of a data source and one of its credentials.

To disable caching, set the value to 0.

Default value: 50.

The maximum number of browse query results that are cached for each DataSource client.

Default value: 100.

Specifies for how long the entries are stored in the cache.

Default value: 450s. For a full list of accepted units, see Duration units.

Defines for how long the DataSource client cache stores items for browsing. Starting from 14.1.0, the cache is cleared after this period expires.

Default value: 300s. For a full list of accepted units, see Duration units.

The root path of the default mounted file system.

It is possible to set up multiple file systems in the same DPE. These folders can be used for profiling and browsing data in MMM.

To add another file system, replace the name of the file system (default) in the property name with the name of the new file system and specify its location, for example:

Restricts which roles are allowed to work with a particular file system. To set limitations for another file system, make sure to provide the correct file system name and the environment name, for example, ataccama.one.platform.local-file-systems.default.dev.allowed-roles=LFS_PROD.

The sample file size for items loaded from local file systems. Used to detect the content type, such as application/octet stream, and so on.

Default value: 100KB. For a full list of accepted units, see Size units.

The number of sheet rows, including the header row, that are sampled to detect the data types of columns when metadata is imported for Microsoft Excel (XLS and XLSX) files. Applies to Excel files from all file-based sources, not only local file systems.

To sample all rows of the sheet, set the property to 0 or a negative value.

Regardless of the row limit size, the full file is read at each import. A larger sample makes incorrect type detection less likely but takes longer on large sheets and uses more processing capacity.

Default value: 50.

If set to true, in case an error occurs while writing objects to ONE Object Storage, all previously stored objects are removed during cleanup.

Default value: true.

Defines how often the object storage is checked for objects that need to be removed following a storage failure. Expressed in milliseconds.

Default value: 600000.

The type of data source for persistent storage. If you are using the default embedded database, set the value to EMBEDDED (default). If you want to use another data source, set the value to CUSTOM.

Default value: EMBEDDED.

Points to the folder storing the persisted data. Persisted data includes all application data, files for processing, and processing results.

Default value: ${ataccama.path.storage}.

The classpath delimiter used for libraries for local jobs.

Default value: ; (semicolon).

The libraries needed for local jobs. Jobs are stored as tmp/jobs/${jobId} in the root directory of the module.

Starting from 13.4.0, all JDBC libraries need to be specified here in order to use the driverClass in Global runtime configuration.

You can specify:

Default value: ../../../lib/runtime/*;../../../lib/jdbc_ext/*.

The directory that holds temporary work files for local jobs. A folder is created for each job within this directory, with the name containing the jobId.

If not specified, the system environment variable tmp is used. If the system environment variable is not set up, the temporary job folder is created in the root directory of the machine.

After each job finishes, the corresponding work files are automatically deleted from this directory.

References the script for customizing how local jobs are launched. For example, you can change which shell script is used to start the job. The script is located in the bin/local folder in the root directory of the module.

The default value for Linux is local/exec_local.sh.

(Optional) Excludes certain Java libraries and drivers that are in the same classpath but are not needed for processing.

Each library needs to be prefixed by an exclamation mark (!). Libraries are separated by the delimiter defined in the cpdelim property.

Example: !hadoop*.jar.

Points to the location of Ataccama licenses needed for local jobs. Configure it only if licenses are stored outside of the standard locations, that is, the user home directory and the folder dpe/lib/runtime/license_keys.

Starting from 17.2.0, engines that connect to Ataccama Cloud, including hybrid DPEs on your own infrastructure, don’t need a license file as DPM provides a signed token with each job instead. The property is relevant only for self-managed deployments, where DPM also runs on your side.

Default value: ../../../lib/runtime/license_keys.

Sets the JAVA_HOME variable for running local jobs, for example, /usr/java/jdk1.8.0_65. The property can only be used together with the property plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.exec.

The runtime is compatible with Java 8 or later.

Configures any environment variable for running local jobs.

To set a custom environment variable, provide the name of the variable instead of the placeholder ANY_ENVIRONMENT_VARIABLE and its value.

plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.job-specific-system-properties.allowed-keys

plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.job-specific-system-properties.allowed-keys

A comma-separated list of keys identifying the system properties that can be set when submitting a job, which makes their values specific for a particular job. These job-specific system properties are then passed to each spawned runtime JVM associated with the identified launch type (LOCAL, SPARK).

If set to true, job data is removed immediately after the job finishes.

Default value: true.

Specifies the time period after which job results are deleted from DPE, starting from when the job finishes. Typically, the property serves as a backup in case job files could not be deleted after the job was completed.

This option applies regardless of how the property plugin.executor.ataccama.one.clear-job is configured.

Default value: 2h. For a full list of accepted units, see Duration units.

Defines how often DPM checks for recently completed jobs in DPE. Expressed in milliseconds.

Default value: 1800000 (30 minutes).

The number of old jobs to be cleaned at once.

Default value: 1000.

The maximum duration of each job-cleaning run. This property must be set with respect to plugin.executor.ataccama.one.job-check-interval so that expired jobs could be removed effectively.

Default value: 1m. For a full list of accepted units, see Duration units.

The maximum number of processes or threads that can be run in parallel on this engine.

Starting from 17.2.0, DPM also enforces a limit across all engines. See Concurrent job limit and license.

Default value (self-managed and hybrid deployments): 5.

Default value (Ataccama Cloud deployments): 1.

Enables notifications over gRPC from the ONE runtime server for runtime jobs.

Default value: true.

Enables notifications over gRPC from the ONE runtime server for data quality monitoring jobs.

Default value: true.

A regular expression matching the names of environment variables set for DPE that can be accessed by child processes spawned for runtime jobs.

Default value: JAVA_HOME|JAVA_OPTS|PATH|Path|LANG|LANGUAGE|LC_.*.

The required number of evaluation threads.

To set this, use JAVA_OPTS, for example:

The variable defines the number of models which are processed in parallel in the DQ engine.

Default value: 1.

Set this property to true when your Snowflake database has a large number of tables (>10000) due to query limits when using the default configuration.

To set this, use JAVA_OPTS, for example:

For more information about the query limits, see Data Sources Configuration, section Troubleshooting.

Default value: false.

Batch limit for component rules.

To set this using JAVA_OPTS, use the dqd.rules.batchSize syntax, for example:

Default value: 1000.

The type of application shutdown.

If set to graceful, after receiving the shutdown signal, the application waits for any active requests to finish before proceeding. If set to immediate, the application shuts down as soon as the shutdown signal is received.

Default value: graceful.

Defines how long a shutdown phase can last. After this time expires, the application shuts down regardless of any active requests.

Default value: 30s. For a full list of accepted units, see Duration units.

If set to true, the application waits for any running jobs to complete before continuing with graceful shutdown.

The waiting period is defined through the property shutdown.wait-jobs.

How long the application waits for running jobs to complete before shutting down gracefully. Once the timeout is reached, any remaining jobs are canceled.

Should be shorter than the value set in spring.lifecycle.timeout-per-shutdown-phase. Used only if the shutdown.kill-jobs property is set to true.

Default value: 20s. For a full list of accepted units, see Duration units.