If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

DPM Configuration

In on-premise deployments, the following properties configure Data Processing Module (DPM) and are provided in the dpm/etc/application.properties file.

In addition, the following properties can be specified for DPM as well:

Keycloak authentication

Properties Data type Description

Properties	Data type	Description
`ataccama.authentication.keycloak.base-url`	String	A helper property used for constructing other Keycloak authentication properties. Default value: `http://localhost:8080`.
`ataccama.authentication.keycloak.server-url`	String	The URL of the server where Keycloak is running. Default value: `${ataccama.authentication.keycloak.base-url}/auth`.
`ataccama.authentication.keycloak.realm`	String	The name of the Keycloak realm. Default value: `ataccamaone`.
`ataccama.authentication.keycloak.token.client-id`	String	The client identifier. Used to verify a user’s authorization token and to log in a user. Default value: `dpm-token-client`.
`ataccama.authentication.keycloak.token.secret`	String	The secret key of the client. Secret keys can be generated using Keycloak. Default value: `dpm-token-client-s3cret`.
`ataccama.authentication.keycloak.token.issuer`	String	Specifies the issuer of the JWT token. Typically, Keycloak uses the URL of the realm as the token issuer. Default value: `${ataccama.authentication.keycloak.server-url}/realms/${ataccama.authentication.keycloak.realm}`.
`ataccama.authentication.keycloak.admin.enable`	Boolean	Enables Keycloak admin access.

ataccama.authentication.keycloak.base-url

String

A helper property used for constructing other Keycloak authentication properties.

Default value: http://localhost:8080.

ataccama.authentication.keycloak.server-url

String

The URL of the server where Keycloak is running.

Default value: ${ataccama.authentication.keycloak.base-url}/auth.

ataccama.authentication.keycloak.realm

String

The name of the Keycloak realm.

Default value: ataccamaone.

ataccama.authentication.keycloak.token.client-id

String

The client identifier. Used to verify a user’s authorization token and to log in a user.

Default value: dpm-token-client.

ataccama.authentication.keycloak.token.secret

String

The secret key of the client. Secret keys can be generated using Keycloak.

Default value: dpm-token-client-s3cret.

ataccama.authentication.keycloak.token.issuer

String

Specifies the issuer of the JWT token. Typically, Keycloak uses the URL of the realm as the token issuer.

Default value: ${ataccama.authentication.keycloak.server-url}/realms/${ataccama.authentication.keycloak.realm}.

ataccama.authentication.keycloak.admin.enable

Boolean

Enables Keycloak admin access.

gRPC Server

General settings

Property Data type Description

Property	Data type	Description
`ataccama.server.grpc.port`	Number	The port where the gRPC server is running. Default value: `8531`.
`ataccama.server.grpc.max-message-size`	String	Limits the size of messages that the gRPC server can process. The message size needs to fit in the working memory. Default value: `1GB`. For a full list of accepted units, see Size units.
`ataccama.server.grpc.executor.poolSizeCore`	Number	The gRPC server request executor core pool size. Make sure this value is sufficient for the usual traffic in order to avoid creating additional connections because the current version of the gRPC library has the keep-alive for threads set to 0. Default value: `10`.
`ataccama.server.grpc.executor.poolSizeMax`	Number	The gRPC server request executor max pool size. The queue length of executor is unlimited by default, and therefore the maximum pool size is effectively ignored. However, the value should be equal to or higher than the core pool size. Default value: `10`.

ataccama.server.grpc.port

Number

The port where the gRPC server is running.

Default value: 8531.

ataccama.server.grpc.max-message-size

String

Limits the size of messages that the gRPC server can process. The message size needs to fit in the working memory.

Default value: 1GB. For a full list of accepted units, see Size units.

ataccama.server.grpc.executor.poolSizeCore

Number

The gRPC server request executor core pool size. Make sure this value is sufficient for the usual traffic in order to avoid creating additional connections because the current version of the gRPC library has the keep-alive for threads set to 0.

Default value: 10.

ataccama.server.grpc.executor.poolSizeMax

Number

The gRPC server request executor max pool size. The queue length of executor is unlimited by default, and therefore the maximum pool size is effectively ignored. However, the value should be equal to or higher than the core pool size.

Default value: 10.

Authentication

Property Data type Description

Property	Data type	Description
`ataccama.authentication.grpc.basic.enable`	Boolean	Enables basic authentication on the gRPC Server. Default value: `true`.
`ataccama.authentication.grpc.bearer.enable`	Boolean	Enables bearer authentication on the gRPC Server. Default value: `true`.
`ataccama.authentication.grpc.internal.jwt.enable`	Boolean	Enables internal JWT token authentication on the gRPC Server. Default value: `true`.
`ataccama.authentication.grpc.mtls.enable`	Boolean	Enables mTLS authentication on the gRPC Server. Default value: `false`.

ataccama.authentication.grpc.basic.enable

Boolean

Enables basic authentication on the gRPC Server.

Default value: true.

ataccama.authentication.grpc.bearer.enable

Boolean

Enables bearer authentication on the gRPC Server.

Default value: true.

ataccama.authentication.grpc.internal.jwt.enable

Boolean

Enables internal JWT token authentication on the gRPC Server.

Default value: true.

ataccama.authentication.grpc.mtls.enable

Boolean

Enables mTLS authentication on the gRPC Server.

Default value: false.

TLS/mTLS

Property Data type Description

Property	Data type	Description
`ataccama.server.grpc.tls.enabled`	Boolean	Enables TLS authentication on the gRPC server. Default value: `false`.
`ataccama.server.grpc.tls.m-tls`	String	Defines whether mutual TLS authentication is enabled. Possible values: `NONE`, `OPTIONAL`, `REQUIRED`. When set to `OPTIONAL`, if the server receives an mTLS request, it attempts to authenticate the request using mTLS. Disabled by default.
`ataccama.server.grpc.tls.cert-chain`	String	The full path to the TLS certificate, for example, `file:/path/to/server.crt`.
`ataccama.server.grpc.tls.private-key`	String	The full path to the private key of the certificate, for example, `file:/path/to/server.key`.
`ataccama.server.grpc.tls.trust-cert-collection`	String	The full path to the public certificate of the root certificate authority, for example, `file:/path/to/rootCA.crt`.

ataccama.server.grpc.tls.enabled

Boolean

Enables TLS authentication on the gRPC server.

Default value: false.

ataccama.server.grpc.tls.m-tls

String

Defines whether mutual TLS authentication is enabled. Possible values: NONE, OPTIONAL, REQUIRED.

When set to OPTIONAL, if the server receives an mTLS request, it attempts to authenticate the request using mTLS.

Disabled by default.

ataccama.server.grpc.tls.cert-chain

String

The full path to the TLS certificate, for example, file:/path/to/server.crt.

ataccama.server.grpc.tls.private-key

String

The full path to the private key of the certificate, for example, file:/path/to/server.key.

ataccama.server.grpc.tls.trust-cert-collection

String

The full path to the public certificate of the root certificate authority, for example, file:/path/to/rootCA.crt.

gRPC Client

General settings

Property Data type Description

Property	Data type	Description
`ataccama.client.grpc.properties.max-message-size`	String	Limits the size of messages that the gRPC client can process. Default value: `1GB`. For a full list of accepted units, see Size units.
`ataccama.client.grpc.properties.executor.poolSizeCore`	Number	The default gRPC client request executor core pool size (intended to be applied only to the dynamically created DPE channels). The pools must be set so that they can accommodate all requests to a single DPE (job submitting, status checks, `data-connect` queries either in pre-processing or directly from the clients). The current version of the gRPC library requires a sufficient number of threads to ensure that the status checks do not wait for a thread in the queue. If you observe DPE spontaneously disconnecting as a result of timed-out status checks, increase the pool size. Default value: `20`.
`ataccama.client.grpc.properties.executor.poolSizeMax`	Number	The default gRPC client request executor maximum pool size (intended to be applied only to the dynamically created DPE channels). The queue length of the executor is unlimited by default, and therefore the maximum pool size is effectively ignored. However, make sure that this value is equal to or higher than the core pool size. Default value: `20`.

ataccama.client.grpc.properties.max-message-size

String

Limits the size of messages that the gRPC client can process.

Default value: 1GB. For a full list of accepted units, see Size units.

ataccama.client.grpc.properties.executor.poolSizeCore

Number

The default gRPC client request executor core pool size (intended to be applied only to the dynamically created DPE channels). The pools must be set so that they can accommodate all requests to a single DPE (job submitting, status checks, data-connect queries either in pre-processing or directly from the clients).

The current version of the gRPC library requires a sufficient number of threads to ensure that the status checks do not wait for a thread in the queue. If you observe DPE spontaneously disconnecting as a result of timed-out status checks, increase the pool size.

Default value: 20.

ataccama.client.grpc.properties.executor.poolSizeMax

Number

The default gRPC client request executor maximum pool size (intended to be applied only to the dynamically created DPE channels). The queue length of the executor is unlimited by default, and therefore the maximum pool size is effectively ignored. However, make sure that this value is equal to or higher than the core pool size.

Default value: 20.

TLS/mTLS

Starting from 13.5.0, each DPE can actively request TLS-secured communication with DPM. In order to allow only TLS-secured connections with all DPE instances, use the property ataccama.one.dpm.registry.enforce-tls=true.

Property Data type Description

Property	Data type	Description
`ataccama.client.grpc.tls.enabled`	Boolean	Enables TLS authentication when communicating with the gRPC server. It also ensures that the communication with the server over gRPC is secure (encrypted instead of in plaintext) and guarantees the integrity of messages. Default value: `false`.
`ataccama.client.grpc.tls.mtls`	Boolean	Enables mutual TLS authentication between the server and the client. Default value: `false`.
`ataccama.client.grpc.tls.cert-chain`	String	The full path to the TLS certificate, for example, `file:/path/to/client.crt`.
`ataccama.client.grpc.tls.private-key`	String	The full path to the private key of the certificate, for example, `file:/path/to/client.key`.
`ataccama.client.grpc.tls.trust-cert-collection`	String	The full path to the public certificate of the root certificate authority, for example, `file:/path/to/rootCA.crt`.

ataccama.client.grpc.tls.enabled

Boolean

Enables TLS authentication when communicating with the gRPC server. It also ensures that the communication with the server over gRPC is secure (encrypted instead of in plaintext) and guarantees the integrity of messages.

Default value: false.

ataccama.client.grpc.tls.mtls

Boolean

Enables mutual TLS authentication between the server and the client.

Default value: false.

ataccama.client.grpc.tls.cert-chain

String

The full path to the TLS certificate, for example, file:/path/to/client.crt.

ataccama.client.grpc.tls.private-key

String

The full path to the private key of the certificate, for example, file:/path/to/client.key.

ataccama.client.grpc.tls.trust-cert-collection

String

The full path to the public certificate of the root certificate authority, for example, file:/path/to/rootCA.crt.

MMM Connection

The following properties are used to configure gRPC channels through which DPM connects to a Metadata Management Module (MMM) instance in order to retrieve information needed for executing ONE plans. This is typically data for MMM catalog items that are used in the executed plans.

Property Data type Description

Property	Data type	Description
`ataccama.client.connection.mmm.host`	String	The hostname of the machine where MMM is running. Default value: `localhost`.
`ataccama.client.connection.mmm.grpc.port`	Number	The gRPC port where MMM is running. Default value: `8521`.
`ataccama.client.connection.mmm.http.enabled`	Boolean	Enables the HTTP connection between DPM and MMM. Default value: `true`.
`ataccama.client.connection.mmm.http.port`	Number	The HTTP port where MMM is running. Default value: `8021`.

ataccama.client.connection.mmm.host

String

The hostname of the machine where MMM is running.

Default value: localhost.

ataccama.client.connection.mmm.grpc.port

Number

The gRPC port where MMM is running.

Default value: 8521.

ataccama.client.connection.mmm.http.enabled

Boolean

Enables the HTTP connection between DPM and MMM.

Default value: true.

ataccama.client.connection.mmm.http.port

Number

The HTTP port where MMM is running.

Default value: 8021.

MDM Connection

Property Data type Description

Property	Data type	Description
`plugin.mdm-datasource.ataccama.one.server.grpc.port`	String	The port of the gRPC interface that is used by MDM. Default value: `18581`.

plugin.mdm-datasource.ataccama.one.server.grpc.port

String

The port of the gRPC interface that is used by MDM.

Default value: 18581.

DPM settings

Property Data type Description

ataccama.one.dpm.registry.check-engine-threads

Number

The number of threads used to check the status of DPE.

Default value: 4.

ataccama.one.dpm.registry.check-connected-interval

String

Defines how often DPM checks the status of a connected DPE.

Default value: 1m. For a full list of accepted units, see Duration units.

ataccama.one.dpm.registry.check-disconnected-interval

String

Specifies the initial interval duration for checking the status of a disconnected DPE.

DPM performs the first check immediately after it detects a disconnected DPE. If the connection remains disrupted, subsequent checks are done based on this interval.

Default value: 1m. For a full list of accepted units, see Duration units.

ataccama.one.dpm.registry.check-disconnected-interval-multiplier

Number

A multiplier used to increase the interval between subsequent checks performed on a disconnected DPE. The interval becomes longer after each unsuccessful try until the maximum interval length is reached.

Default value: 2.

ataccama.one.dpm.registry.check-disconnected-interval-max

String

Determines the maximum interval duration for checking the status of a disconnected DPE. The interval multiplier cannot increase the interval duration past this value.

Default value: 10m. For a full list of accepted units, see Duration units.

ataccama.one.dpm.registry.grpc-checker-timeout

Number

Configures after which period a gRPC request times out when checking the engine status. Expressed in milliseconds.

Default value: 200.

ataccama.one.dpm.registry.channel-provider-ttl

String

Sets the threshold after which a gRPC channel provider associated with a registered DPE can be safely disposed of. This helps minimize the unnecessary use of DPM resources.

Default value: 1h. For a full list of accepted units, see Duration units.

ataccama.one.dpm.registry.inactivate-scheduler-period

Number

Specifies how often DPM’s internal scheduler is run.

During each run, the scheduler checks when each DPE has last been seen, and, if that period exceeds the value set in automatically-inactivate-engines-not-seen-for, the DPE is considered as inactive. Expressed in milliseconds.

Default value: 600000.

ataccama.one.dpm.registry.automatically-inactivate-engines-not-seen-for

String

Defines after which period a DPE is automatically inactivated. Inactivate DPEs are not allocated any jobs, however, they remain registered and their information is still persisted in DPM.

Default value: 2d. For a full list of accepted units, see Duration units.

ataccama.one.dpm.registry.manually-inactivate-engines-not-seen-for

String

Defines how long a DPE must remain inactive before it can be removed through a GraphQL request (mutation inactivateEngines). Should be higher than plugin.executor.ataccama.one.job-disconnected-timeout.

Default value: 65m. For a full list of accepted units, see Duration units.

ataccama.one.dpm.registry.delete-scheduler-period

Number

Defines how often the scheduler automatically deletes inactive DPEs. Expressed in milliseconds.

Default value: 601000.

ataccama.one.dpm.registry.automatically-delete-engines-not-seen-for

String

Defines how long a DPE must remain inactive before it is automatically deleted. Should be higher than ataccama.one.dpm.registry.automatically-inactivate-engines-not-seen-for.

Default value: 4d. For a full list of accepted units, see Duration units.

ataccama.one.dpm.registry.manually-delete-engines-not-seen-for

String

Defines how long a DPE must remain inactive before it can be deleted through a GraphQL request (mutation deleteEngines). Should be higher than ataccama.one.dpm.registry.manually-inactivate-engines-not-seen-for.

Default value: 130m. For a full list of accepted units, see Duration units.

ataccama.one.dpm.disconnected-warn-period

Number

Determines how often WARN messages about a disconnected engine are written to the log. By default, this is done once every 24 hours. Expressed in milliseconds.

Default value: 86400000.

ataccama.one.dpm.registry.enforce-tls

Boolean

This optional property enforces TLS-level security on communication of DPM and all connected DPEs. If set to true, DPM does not connect to non-TLS secured DPEs.

Default value: false.

ataccama.one.flyway.lock-retry-count

Number

Specifies how many times DPM retries to place a lock on the database in order to complete its migration. If the lock is not placed by the time the count is reached, migration fails.

DPM retries in 1s intervals. To retry indefinitely, set to -1.

Default value: 60.

ataccama.one.dpm.datasources.enabled

String

A comma-separated list of connection types that are shown when configuring a new data source connection in ONE Web Application.

Example: ADLS,AMAZON_S3,AURORA-MYSQL,AURORA-POSTGRESQL.

If this property is specified, only the connection types listed in the property value are available. Otherwise, all connection types are available.

By default, this property is not set.

The value for each connection type can be found in the web application. For the required connection type, navigate to Overview > Connection details and check the value for the Executor type property.

GraphQL and DPM Admin Console

Property Data type Description

Property	Data type	Description
`server.port`	Number	The HTTP server port for GraphQL API and monitoring endpoints. This is also where the DPM Admin Console user interface is available. Default value: `8031`.
`plugin.executor.ataccama.one.protected-files`	String	A list of files containing data that should not be made available to non-power users. The files can still be accessed in DPM Admin Console but are not downloaded to ONE Desktop. Default value: `status.xml`.
`plugin.executor.ataccama.one.graphql.job.listing.limit`	Number	Defines the maximum number of recent jobs listed in DPM Admin Console. Default value: `200`.
`plugin.dqc-support.ataccama.one.launch-model.file-names.inaccessible`	String	A comma-separated list of launch model files that are removed from the listing in DPM Admin Console, in addition to the already streamed files, such as logs. By default, job properties are not shown as they are included in the submit details and might leak sensitive information if unmasked. Default value: `job.properties`.
`graphql.servlet.mapping`	String	Configures where the GraphQL servlet is exposed. Default value: `/graphql`.
`ataccama.one.webserver.gql.allowed-roles`	String	Limits access to DPM Admin Console based on the user role. Default value: `admin,DPP_admin`.
`ataccama.one.graphql.xml-content-masking.keys`	String	A comma-separated list specifying sensitive XML content in keys that is masked when `.properties` files are published over the GraphQL API. For example, `com.ataccama.dqc.processor.bin.config.RuntimeConfiguration` masks passwords in the runtime configuration.
`ataccama.one.graphql.masking.strategy`	String	The masking strategy used to avoid displaying sensitive information. There are two strategies available: `replacement` - Keys are replaced by another string defined in the `masking.mask` property. `omitting` - Sensitive entries are not published at all. Default value: `replacement`.
`ataccama.one.graphql.masking.mask`	String	The string with which sensitive keys are replaced if the replacement masking strategy is applied. Default value: `\u2022\u2022\u2022\u2022\u2022` (corresponds to five bullet points).
`ataccama.one.graphql.masking.keys`	String	A comma-separated list of sensitive keys that should be masked when `.properties` files are published over the GraphQL API. For example, `mount.conf.fs.azure.account.oauth2.client.secret`. The list can contain duplicated values.
`ataccama.one.dpm.job.logs.max-rows`	Number	Maximum number of rows returned for log output from DPE. Default value: `8000`.
`ataccama.one.dpm-console.server.base-url`	String	The base URL of the DPM Admin Console frontend. Default value: `/`.
`ataccama.one.dpm-console.server.api-url`	String	The URL of the GraphQL endpoint that is used by DPM Admin Console for the user interface. Default value: `/graphql`.

server.port

Number

The HTTP server port for GraphQL API and monitoring endpoints. This is also where the DPM Admin Console user interface is available.

Default value: 8031.

plugin.executor.ataccama.one.protected-files

String

A list of files containing data that should not be made available to non-power users. The files can still be accessed in DPM Admin Console but are not downloaded to ONE Desktop.

Default value: status.xml.

plugin.executor.ataccama.one.graphql.job.listing.limit

Number

Defines the maximum number of recent jobs listed in DPM Admin Console.

Default value: 200.

plugin.dqc-support.ataccama.one.launch-model.file-names.inaccessible

String

A comma-separated list of launch model files that are removed from the listing in DPM Admin Console, in addition to the already streamed files, such as logs.

By default, job properties are not shown as they are included in the submit details and might leak sensitive information if unmasked.

Default value: job.properties.

graphql.servlet.mapping

String

Configures where the GraphQL servlet is exposed.

Default value: /graphql.

ataccama.one.webserver.gql.allowed-roles

String

Limits access to DPM Admin Console based on the user role.

Default value: admin,DPP_admin.

ataccama.one.graphql.xml-content-masking.keys

String

A comma-separated list specifying sensitive XML content in keys that is masked when .properties files are published over the GraphQL API. For example, com.ataccama.dqc.processor.bin.config.RuntimeConfiguration masks passwords in the runtime configuration.

ataccama.one.graphql.masking.strategy

String

The masking strategy used to avoid displaying sensitive information. There are two strategies available:

replacement - Keys are replaced by another string defined in the masking.mask property.
omitting - Sensitive entries are not published at all.

Default value: replacement.

ataccama.one.graphql.masking.mask

String

The string with which sensitive keys are replaced if the replacement masking strategy is applied.

Default value: \u2022\u2022\u2022\u2022\u2022 (corresponds to five bullet points).

ataccama.one.graphql.masking.keys

String

A comma-separated list of sensitive keys that should be masked when .properties files are published over the GraphQL API. For example, mount.conf.fs.azure.account.oauth2.client.secret. The list can contain duplicated values.

ataccama.one.dpm.job.logs.max-rows

Number

Maximum number of rows returned for log output from DPE.

Default value: 8000.

ataccama.one.dpm-console.server.base-url

String

The base URL of the DPM Admin Console frontend.

Default value: /.

ataccama.one.dpm-console.server.api-url

String

The URL of the GraphQL endpoint that is used by DPM Admin Console for the user interface.

Default value: /graphql.

Single sign-on for DPM Admin Console

Property Data type Description

Property	Data type	Description
`ataccama.one.webserver.sso.realm`	String	The name of the Keycloak realm used for SSO. Default value: `${ataccama.authentication.keycloak.realm}`.
`ataccama.one.webserver.sso.base-url`	String	The base URL where Keycloak is available. Used as a prefix for other SSO URLs. Default value: `${ataccama.authentication.keycloak.server-url}`.
`ataccama.one.webserver.sso.auth-url`	String	The URL where users are redirected to provide authentication credentials. Default value: `${ataccama.one.webserver.sso.base-url}/realms/${ataccama.one.webserver.sso.realm}/protocol/openid-connect/auth`.
`ataccama.one.webserver.sso.token-url`	String	The URL used to obtain authentication tokens from Keycloak. Default value: `${ataccama.one.webserver.sso.base-url}/realms/${ataccama.one.webserver.sso.realm}/protocol/openid-connect/token`.
`ataccama.one.webserver.sso.logout-url`	String	The URL used for logging users out. Default value: `${ataccama.one.webserver.sso.base-url}/realms/${ataccama.one.webserver.sso.realm}/protocol/openid-connect/logout`.
`ataccama.one.webserver.sso.client-id`	String	The client identifier used for verifying user authorization tokens and for logging in. Default value: `dpm-webapp-public-client`.

ataccama.one.webserver.sso.realm

String

The name of the Keycloak realm used for SSO.

Default value: ${ataccama.authentication.keycloak.realm}.

ataccama.one.webserver.sso.base-url

String

The base URL where Keycloak is available. Used as a prefix for other SSO URLs.

Default value: ${ataccama.authentication.keycloak.server-url}.

ataccama.one.webserver.sso.auth-url

String

The URL where users are redirected to provide authentication credentials.

Default value: ${ataccama.one.webserver.sso.base-url}/realms/${ataccama.one.webserver.sso.realm}/protocol/openid-connect/auth.

ataccama.one.webserver.sso.token-url

String

The URL used to obtain authentication tokens from Keycloak.

Default value: ${ataccama.one.webserver.sso.base-url}/realms/${ataccama.one.webserver.sso.realm}/protocol/openid-connect/token.

ataccama.one.webserver.sso.logout-url

String

The URL used for logging users out.

Default value: ${ataccama.one.webserver.sso.base-url}/realms/${ataccama.one.webserver.sso.realm}/protocol/openid-connect/logout.

ataccama.one.webserver.sso.client-id

String

The client identifier used for verifying user authorization tokens and for logging in.

Default value: dpm-webapp-public-client.

Plugins and JDBC drivers

Property Data type Description

Property	Data type	Description
`plugins.path`	String	The location of the `plugins` folder. Default value: `${ataccama.path.plugin}`.

plugins.path

String

The location of the plugins folder.

Default value: ${ataccama.path.plugin}.

DQ Evaluation plugin

Property Data type Description

Property	Data type	Description
`plugin.dq-eval.ataccama.one.lookup-cache-size`	Number	The disk space allocated to the lookup files cache. Expressed in megabytes. Default value: `200`.
`plugin.dq-eval.ataccama.one.lookup-cache-expiration`	Number	Configures for how long lookup files are cached. Expressed in minutes. Default value: `60`.
`plugin.dq-eval.ataccama.one.filter-combination-limit`	Number	The maximum allowed number of filter combinations. Default value: `100`.

plugin.dq-eval.ataccama.one.lookup-cache-size

Number

The disk space allocated to the lookup files cache. Expressed in megabytes.

Default value: 200.

plugin.dq-eval.ataccama.one.lookup-cache-expiration

Number

Configures for how long lookup files are cached. Expressed in minutes.

Default value: 60.

plugin.dq-eval.ataccama.one.filter-combination-limit

Number

The maximum allowed number of filter combinations.

Default value: 100.

Persistent storage

The type of data source used for persistent storage is defined through DPE. Any missing drivers need to be added to the <ataccama_home>/lib folder.

Property Data type Description

Property	Data type	Description
`spring.datasource.url`	String	A JDBC connection string pointing to the persistent storage database. Default value: `jdbc:postgresql://localhost:5432/dpm`.
`spring.datasource.username`	String	The username for the persistent storage database. Default value: `one`.
`spring.datasource.password`	String	The password for the persistent storage database. Default value: `one`.

spring.datasource.url

String

A JDBC connection string pointing to the persistent storage database.

Default value: jdbc:postgresql://localhost:5432/dpm.

spring.datasource.username

String

The username for the persistent storage database.

Default value: one.

spring.datasource.password

String

The password for the persistent storage database.

Default value: one.

Audit

Property Data type Description

Property	Data type	Description
`ataccama.audit.enabled`	Boolean	Enables auditing. Default value: `false`.
`ataccama.audit.spring.datasource.jdbcUrl`	String	A JDBC connection string pointing to the database where audit logs are stored. Default value: `jdbc:postgresql://localhost:5432/audit`.
`ataccama.audit.spring.datasource.user`	String	The username for the audit database. Default value: `one`.
`ataccama.audit.spring.datasource.password`	String	The password for the audit database. Default value: `one`.

ataccama.audit.enabled

Boolean

Enables auditing.

Default value: false.

ataccama.audit.spring.datasource.jdbcUrl

String

A JDBC connection string pointing to the database where audit logs are stored.

Default value: jdbc:postgresql://localhost:5432/audit.

ataccama.audit.spring.datasource.user

String

The username for the audit database.

Default value: one.

ataccama.audit.spring.datasource.password

String

The password for the audit database.

Default value: one.

DataConnect plugin

The DataConnect Plugin is used to query metadata in a particular data source. The retrieved information can be cached.

Property Data type Description

Property	Data type	Description
`plugin.dataconnect.ataccama.one.constraints.prioritize-cached-data-source.enabled`	Boolean	Enables prioritizing cached data sources when routing jobs and other queries. Default value: `true`.
`plugin.dataconnect.ataccama.one.grpc-timeout`	String	Sets the timeout for gRPC requests on DPE. Default value: `2h`.

plugin.dataconnect.ataccama.one.constraints.prioritize-cached-data-source.enabled

Boolean

Enables prioritizing cached data sources when routing jobs and other queries.

Default value: true.

plugin.dataconnect.ataccama.one.grpc-timeout

String

Sets the timeout for gRPC requests on DPE.

Default value: 2h.

Profiling

The executor uses up the available resources in the following way:

If the core limit of the thread pool has not been reached, the executor tries to use a free active thread from the pool.
If there are no free threads left and the number of active threads corresponds to the pool core limit, the executor keeps adding tasks until the queue is filled.
A new thread is added only when the core size and the queue size have been exhausted.
When the maximum pool size is also reached, the executor rejects any new tasks.

Property Data type Description

plugin.profiling.ataccama.one.async.pool.size.max

Number

The maximum number of threads in the thread pool used for executing asynchronous methods in the Profiling plugin.

Default value: 5.

plugin.profiling.ataccama.one.async.pool.size.core

Number

The number of threads that the thread pool must contain at all times, including both active and idle threads. This refers to the thread pool for executing asynchronous methods in the Profiling plugin.

Default value: 2.

plugin.profiling.ataccama.one.async.queue.size

Number

The number of computation threads that the executor queue can hold when executing asynchronous methods in the Profiling plugin.

Default value: 200.

plugin.profiling.ataccama.one.persist-profiles

Boolean

If set to true, profiling results are stored to the shared object storage. To disable this option, set the value to false.

Default value: false.

plugin.profiling.ataccama.one.count-records

Boolean

If enabled, this property adds a RecordCounter step between the reader component and profiling.

To see the record count in the logs, it is necessary to set the value to true and to set the JAVA_OPTS environment variable -DenableRecordCounters=true in the given DPE for LOCAL jobs.

This can be done by setting the DPE property:

plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.env.JAVA_OPTS=-DenableRecordCounters=true

If JAVA_OPTS are not set correctly, the counting of records is not active.

Default value: false.

plugin.profiling.ataccama.one.count-records-batch-size

Number

This property can be used to adjust the size of records batch size.

Default value: 100000.

plugin.profiling.ataccama.one.buffer-records

Boolean

This property can speed up the reading of data from the source. However, it might also increase disk space usage.

Default value: false.

Data source connection throttling

Starting from version 14.1.0, we recommend defining the connection throttling rules primarily through DPM Admin Console. For instructions, see DPM and DPE Configuration in DPM Admin Console.

The properties described in this section now serve only to provide the initial values, not as the primary method for defining the connection throttling rules.

In order to ensure the maximum number of connections is dedicated to the ONE platform and at the same time catalog browsing is never blocked by DPE jobs, you can now limit connections separately for DPE jobs and data-connect queries.

Use all three plugin.jdbc-datasource.ataccama.one.dpm.resource-allocation.connections properties to enable this function and specify the maximum number of connections used for DPE jobs and for data-connect queries.

All connected DPE instances also need the plugin.jdbcdatasource.ataccama.one.connections.maximum-size property set to 0 to disable caching of data source clients.

plugin.jdbc-datasource.ataccama.one.dpm.resource-allocation.connections.<name>.pattern=jdbc:oracle://<host:port>/.*
plugin.jdbc-datasource.ataccama.one.dpm.resource-allocation.connections.<name>.max-connections-jobs=4
plugin.jdbc-datasource.ataccama.one.dpm.resource-allocation.connections.<name>.max-connections-data-connect=20

JDBC patterns are logged by DPM if a configuration error occurs. Therefore, we strongly discourage inserting sensitive information into the JDBC matching pattern and using sensitive parts of the connection strings in matching patterns.

Property Data type Description

Property	Data type	Description
`plugin.jdbc-datasource.ataccama.one.dpm.resource-allocation.connections.<name>.pattern`	String	A list of regular expressions matching the data source connection strings. Example: `jdbc:oracle://183.121.078.56:1521>/.`, `jdbc:postgres:.`.
`plugin.jdbc-datasource.ataccama.one.dpm.resource-allocation.connections.<name>.max-connections-jobs`	Number	Set the maximum number of connections that can be used for DPE jobs. Unused slots are automatically allocated for `data-connect` queries if needed. If set to `0`, no threads are allocated and all jobs fail. If set to a value lower than is required by a specific job, the job fails immediately.
`plugin.jdbc-datasource.ataccama.one.dpm.resource-allocation.connections.<name>.max-connections-data-connect`	Number	Set the number of connections that can be used for `data-connect` queries. Unused slots are not available for DPE jobs. If set to `0`, `data-connect` queries use any slots available from DPE jobs to allow for catalog browsing.
`plugin.metadata-import.ataccama.one.datasource-allocation.connections-per-datasource`	Number	(Optional) This property helps DPM to estimate the number of connections allocated per each data source when importing metadata. Make sure the number set here does not exceed the sum of values set in `max-connections-jobs` and `max-connections-data-connect`.

plugin.jdbc-datasource.ataccama.one.dpm.resource-allocation.connections.<name>.pattern

String

A list of regular expressions matching the data source connection strings.

Example: jdbc:oracle://183.121.078.56:1521>/., jdbc:postgres:..

plugin.jdbc-datasource.ataccama.one.dpm.resource-allocation.connections.<name>.max-connections-jobs

Number

Set the maximum number of connections that can be used for DPE jobs. Unused slots are automatically allocated for data-connect queries if needed.

If set to 0, no threads are allocated and all jobs fail. If set to a value lower than is required by a specific job, the job fails immediately.

plugin.jdbc-datasource.ataccama.one.dpm.resource-allocation.connections.<name>.max-connections-data-connect

Number

Set the number of connections that can be used for data-connect queries. Unused slots are not available for DPE jobs.

If set to 0, data-connect queries use any slots available from DPE jobs to allow for catalog browsing.

plugin.metadata-import.ataccama.one.datasource-allocation.connections-per-datasource

Number

(Optional) This property helps DPM to estimate the number of connections allocated per each data source when importing metadata. Make sure the number set here does not exceed the sum of values set in max-connections-jobs and max-connections-data-connect.

Resource allocation of runtime steps

You can further finetune DPM resource allocation by using two advanced global runtime properties as follows:

Property Data type Description

Property	Data type	Description
`plugin.dqc-support.ataccama.one.datasource-step-allocation.steps.<name-of-step>.non-threaded`	Number	Setting this property helps DPM estimate the number of connections needed for execution of one instance of the specified type of ONE Runtime step. Non-threaded applies to processes not affected by parallelism setting that always run in one thread.
`plugin.dqc-support.ataccama.one.datasource-step-allocation.steps.<name-of-step>.threaded`	Number	Setting this property helps DPM estimate the number of connections needed for execution of one instance of the specified type of ONE Runtime step. Threaded applies to processes affected by parallelism setting. Such steps can run in the maximum of threads set in parallelism.

plugin.dqc-support.ataccama.one.datasource-step-allocation.steps.<name-of-step>.non-threaded

Number

Setting this property helps DPM estimate the number of connections needed for execution of one instance of the specified type of ONE Runtime step. Non-threaded applies to processes not affected by parallelism setting that always run in one thread.

plugin.dqc-support.ataccama.one.datasource-step-allocation.steps.<name-of-step>.threaded

Number

Setting this property helps DPM estimate the number of connections needed for execution of one instance of the specified type of ONE Runtime step. Threaded applies to processes affected by parallelism setting. Such steps can run in the maximum of threads set in parallelism.

The name of the step corresponds to the class name of the step that is displayed when opening a .plan file in an XML editor (for example, com.ataccama.dqc.tasks.io.text.write.TextFileWriter or com.ataccama.dqc.tasks.io.json.call.JsonCall). Make sure to replace the dot (.) in class names by a dash (-) in property names. For example, the previously mentioned class names should be com-ataccama-dqc-tasks-io-text-write-TextFileWriter and com-ataccama-dqc-tasks-io-json-call-JsonCall respectively.

Make sure that the result of the equation non-threaded + (parallelism * threaded) does not exceed the total number of your DPE job slots.

Executor

Starting from 13.4.0, the job-processing executor is used for job preprocessing, with the job-processing.max-pool-size property defining the job processing maximum parallelism. The queue capacity needs to be slightly higher than the maximum pool size.

The new job-submitting and job-killing executors are used for the following purposes:

job-submitting: Used for submitting jobs, which was previously done serially and included fetching the whole database of the queued jobs. Jobs are now submitted in parallel (property job-submitting.max-pool-size) and only the jobs that are waiting to be submitted are fetched from the database. The queue capacity should be slightly higher than the maximum pool size.
job-killing: A dedicated thread pool for cancelling jobs with a limited thread capacity (10000 by default). This queue is not persisted in the database.

The default thread pool sizes do not guarantee the best possible performance, which should be achieved through monitoring the system performance under the given settings and adjusting the configuration accordingly.

Property Data type Description

Property	Data type	Description
`plugin.executor.ataccama.one.job-processing.max-pool-size`	Number	The maximum number of threads in the thread pool used for pre-processing jobs and submitting them to the main queue in the Executor plugin. Default value: `5`. The same value is used if the property is not set.
`plugin.executor.ataccama.one.job-processing.core-pool-size`	Number	The minimum number of threads used for pre-processing jobs and submitting them to the main queue in the Executor plugin. Default value: `5`. The same value is used if the property is not set.
`plugin.executor.ataccama.one.job-processing.queue-capacity`	Number	The maximum number of jobs in the pre-processing queue. Subsequent jobs are retrieved from the database as the queue empties The value should be slightly higher than `job-processing.max-pool-size`. Default value: `10`.
`plugin.executor.ataccama.one.job-postprocessing.max-pool-size`	Number	The maximum number of threads in the thread pool used for post-processing jobs. Default value: `5`.
`plugin.executor.ataccama.one.job-postprocessing.core-pool-size`	Number	The minimum number of threads used for post-processing jobs. Default value: `5`.
`plugin.executor.ataccama.one.job-postprocessing.queue-capacity`	Number	The maximum number of jobs in the post-processing queue. The value should be slightly higher than job-postprocessing.max-pool-size. Default value: `10`.
`plugin.executor.ataccama.one.job-submitting.max-pool-size`	Number	The maximum number of threads in the thread pool used for submitting jobs in the Executor plugin. Default value: `5`.
`plugin.executor.ataccama.one.job-submitting.core-pool-size`	Number	The number of threads that the thread pool used for submitting jobs must contain at all times, including both active and idle threads. Default value: `5`.
`plugin.executor.ataccama.one.job-submitting.queue-capacity`	Number	The maximum number of jobs in the queue waiting to be submitted. The value should be slightly higher than `job-submitting.max-pool-size`. Default value: `10`.
`plugin.executor.ataccama.one.job-killing.max-pool-size`	Number	The maximum number of threads in the thread pool used for canceling jobs in the Executor plugin. Default value: `2`.
`plugin.executor.ataccama.one.job-killing.core-pool-size`	Number	The number of threads that the thread pool used for canceling jobs must contain at all times, including both active and idle threads. Default value: `2`.
`plugin.executor.ataccama.one.job-killing.queue-capacity`	Number	The maximum number of jobs scheduled to be canceled. Default value: `10000`.
`plugin.executor.ataccama.one.event-processing.max-pool-size`	Number	The maximum number of threads in the thread pool used for event processing in the Executor plugin. Default value: `5`. The same value is used if the property is not set.
`plugin.executor.ataccama.one.event-processing.core-pool-size`	Number	The number of threads that the thread pool must contain at all times, including both active and idle threads. This refers to the thread pool used for event processing in the Executor plugin. Default value: `5`. The same value is used if the property is not set.
`plugin.executor.ataccama.one.event-processing.queue-capacity`	Number	The maximum number of computation threads that the executor queue dedicated to event processing in the Executor plugin can hold. If the property is not set, the capacity is practically unlimited. Default value: `2147483647`.
`plugin.executor.ataccama.one.event-processing.max-batch-size`	Number	The maximum number of events to be processed in a single thread. Default value: `25`.
`plugin.executor.ataccama.one.event-emitting.max-pool-size`	Number	The maximum number of threads in the thread pool used for notifying about events in DPM. Default value: `5`.
`plugin.executor.ataccama.one.event-emitting.core-pool-size`	Number	The number of threads that the thread pool used for notifying about events must contain at all times, including both active and idle threads. Default value: `5`.
`plugin.executor.ataccama.one.event-emitting.queue-capacity`	Number	The maximum number of computation threads that the executor queue used for notifying about events in DPM can hold. If the property is not set, the capacity is practically unlimited. Default value: `2147483647`.
`plugin.executor.ataccama.one.processed-event-chunk-size`	Number	The number of events for processing each time. Default value: `100`.
`plugin.executor.ataccama.one.processed-event-lock-duration`	String	Specifies how long the event processing holds the lock. This value should be lower than `lockAtMostInterval` with a reasonable margin. If the value is greater or equal to `lockAtMostInterval`, then DPM does not start. Default value: `2s`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.processed-event-poll-interval`	String	Specifies the frequency of the event processing poll checks for new events while holding the lock. Default value: `250ms`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.job-expiration-interval`	String	Defines for how long job files and processing results are stored in DPM. Default value: `7d`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.job-check-interval`	Number	Specifies how often DPM checks for expired job files and removes them. Expressed in milliseconds. Default value: `3600000` (1 hour).
`plugin.executor.ataccama.one.job-unlock-max-retry-count`	Number	Specifies how many times to retry unlocking a job in case of a database error. Default value: `5`.
`plugin.executor.ataccama.one.job-unlock-retry-delay`	String	Specifies how long to wait between retries when unlocking a job. Default value: `200ms`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.lock-at-most-interval`	String	Determines for how long a DPE can access a particular database. Each database can be used by only one DPE at a time, which is managed by setting a scheduler lock that stores and tracks status logs in a separate table. After this interval expires, the lock is released and another DPE can connect to the same database. This value needs to be higher than `processed-event-lock-duration` with a reasonable margin. If the value is lower than or equal to `processed-event-lock-duration`, then DPM does not start. Default value: `5s`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.update-jobs-interval`	Number	Configures how often the job queue is synced with each instance of DPM. The shared job queue contains jobs that are distributed between multiple DPMs. Jobs are then transferred from DPM to a compatible DPE handled by that DPM. Expressed in milliseconds. Default value: `10000`. The same value is used if the property is not set.
`plugin.executor.ataccama.one.job-lost-timeout`	String	Specifies for how long a job must remain inactive to be considered lost. Such jobs are then resubmitted if they remain in the pre-processing state, otherwise DPM attempts to cancel them. Default value: `5s`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.subscription-job-cache-expiration`	String	Configures for how long jobs can stay in the event subscriber job cache. Default value: `5m`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.subscription-job-cache-size`	Number	The number of jobs that are kept in the event subscriber job cache. Default value: `1000`.
`plugin.executor.ataccama.one.job-input-storage-timeout`	String	Sets for how long job input files can be stored before being assigned to a job. Default value: `1h`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.job-input-storage-check-interval`	Number	Determines how often to check for expired job input files in the job input storage. Expressed in milliseconds. Default value: `1800000` (30 minutes).
`plugin.executor.ataccama.one.job-watchdog-check-interval`	String	Defines how often DPM retrieves the status of queued jobs, including their health information and use of resources in order to avoid queue starvation. This determines how often the following timeout checks are performed: `submit-timeout`, `job-disconnected-timeout`, `job-at-dpm-timeout`. Expressed in milliseconds. Default value: `5000`. The same value is used if the property is not set.
`plugin.executor.ataccama.one.submit-timeout`	String	Configures for how long a job can remain in the job queue before being submitted to a DPE. If set to `0`, jobs are not submitted to DPEs and they fail immediately. If the property is not set, jobs are kept in the queue indefinitely. Default value: `30m`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.job-disconnected-timeout`	String	Determines for how long a job can be recovered after DPE is disconnected. When the timeout is reached, the job status is set to `FAILURE`. If DPM cannot reach DPE for a while, the job status first changes to `DISCONNECTED`. In case the connection is reestablished before the timeout expires, the job status is then set according to the information provided by DPE (for example, `RUNNING`, `SUCCESS`, `FAILURE`). Default value: `1h`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.job-at-dpm-timeout`	String	Specifies the maximum amount of time a job can stay in one of the following statuses: `UNKNOWN`, `PREPROCESSING`, `POSTPROCESSING`. When the timeout is reached, the job status is set to `FAILURE`. Default value: `1h`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.job-submit-timeout`	String	Specifies the maximum amount of time a job can stay in the `SUBMITTED` status. When the timeout is reached, the job status is set to `FAILURE`. Default value: `1h`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.grpc-timeout`	String	Specifies the timeout for synchronous gRPC requests on DPE such as previewing tables and counting numbers of rows during pre-processing. Increase this timeout value if jobs fail during pre-processing phases. Default value: `60s`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.job-unlock-max-retry-count`	Number	Specifies the number of attempts when unlocking a job in case of a database error. Default value: `5`.
`plugin.executor.ataccama.one.job-unlock-retry-delay`	String	Specifies the waiting time between attempts when unlocking a job in case of a database error. Default value: `200ms`. For a full list of accepted units, see Duration units.
`plugin.executor.ataccama.one.allow-job-resubmit`	Boolean	Allows resubmitting a job lost in `PREPROCESSING` state. Default value: `false`.
`plugin.executor.ataccama.one.max-page-size`	Number	The maximum number of rows that should be fetched from the database when paginating. Default value: `100`.
`plugin.dq-eval.ataccama.one.drillthrough-page-size`	Number	The number of records that can be displayed on a single page in the shared file system.

plugin.executor.ataccama.one.job-processing.max-pool-size

Number

The maximum number of threads in the thread pool used for pre-processing jobs and submitting them to the main queue in the Executor plugin.

Default value: 5. The same value is used if the property is not set.

plugin.executor.ataccama.one.job-processing.core-pool-size

Number

The minimum number of threads used for pre-processing jobs and submitting them to the main queue in the Executor plugin.

Default value: 5. The same value is used if the property is not set.

plugin.executor.ataccama.one.job-processing.queue-capacity

Number

The maximum number of jobs in the pre-processing queue. Subsequent jobs are retrieved from the database as the queue empties

The value should be slightly higher than job-processing.max-pool-size.

Default value: 10.

plugin.executor.ataccama.one.job-postprocessing.max-pool-size

Number

The maximum number of threads in the thread pool used for post-processing jobs.

Default value: 5.

plugin.executor.ataccama.one.job-postprocessing.core-pool-size

Number

The minimum number of threads used for post-processing jobs.

Default value: 5.

plugin.executor.ataccama.one.job-postprocessing.queue-capacity

Number

The maximum number of jobs in the post-processing queue.

The value should be slightly higher than job-postprocessing.max-pool-size.

Default value: 10.

plugin.executor.ataccama.one.job-submitting.max-pool-size

Number

The maximum number of threads in the thread pool used for submitting jobs in the Executor plugin.

Default value: 5.

plugin.executor.ataccama.one.job-submitting.core-pool-size

Number

The number of threads that the thread pool used for submitting jobs must contain at all times, including both active and idle threads.

Default value: 5.

plugin.executor.ataccama.one.job-submitting.queue-capacity

Number

The maximum number of jobs in the queue waiting to be submitted.

The value should be slightly higher than job-submitting.max-pool-size.

Default value: 10.

plugin.executor.ataccama.one.job-killing.max-pool-size

Number

The maximum number of threads in the thread pool used for canceling jobs in the Executor plugin.

Default value: 2.

plugin.executor.ataccama.one.job-killing.core-pool-size

Number

The number of threads that the thread pool used for canceling jobs must contain at all times, including both active and idle threads.

Default value: 2.

plugin.executor.ataccama.one.job-killing.queue-capacity

Number

The maximum number of jobs scheduled to be canceled.

Default value: 10000.

plugin.executor.ataccama.one.event-processing.max-pool-size

Number

The maximum number of threads in the thread pool used for event processing in the Executor plugin.

Default value: 5. The same value is used if the property is not set.

plugin.executor.ataccama.one.event-processing.core-pool-size

Number

The number of threads that the thread pool must contain at all times, including both active and idle threads. This refers to the thread pool used for event processing in the Executor plugin.

Default value: 5. The same value is used if the property is not set.

plugin.executor.ataccama.one.event-processing.queue-capacity

Number

The maximum number of computation threads that the executor queue dedicated to event processing in the Executor plugin can hold. If the property is not set, the capacity is practically unlimited.

Default value: 2147483647.

plugin.executor.ataccama.one.event-processing.max-batch-size

Number

The maximum number of events to be processed in a single thread.

Default value: 25.

plugin.executor.ataccama.one.event-emitting.max-pool-size

Number

The maximum number of threads in the thread pool used for notifying about events in DPM.

Default value: 5.

plugin.executor.ataccama.one.event-emitting.core-pool-size

Number

The number of threads that the thread pool used for notifying about events must contain at all times, including both active and idle threads.

Default value: 5.

plugin.executor.ataccama.one.event-emitting.queue-capacity

Number

The maximum number of computation threads that the executor queue used for notifying about events in DPM can hold. If the property is not set, the capacity is practically unlimited.

Default value: 2147483647.

plugin.executor.ataccama.one.processed-event-chunk-size

Number

The number of events for processing each time.

Default value: 100.

plugin.executor.ataccama.one.processed-event-lock-duration

String

Specifies how long the event processing holds the lock. This value should be lower than lockAtMostInterval with a reasonable margin. If the value is greater or equal to lockAtMostInterval, then DPM does not start.

Default value: 2s. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.processed-event-poll-interval

String

Specifies the frequency of the event processing poll checks for new events while holding the lock.

Default value: 250ms. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.job-expiration-interval

String

Defines for how long job files and processing results are stored in DPM.

Default value: 7d. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.job-check-interval

Number

Specifies how often DPM checks for expired job files and removes them. Expressed in milliseconds.

Default value: 3600000 (1 hour).

plugin.executor.ataccama.one.job-unlock-max-retry-count

Number

Specifies how many times to retry unlocking a job in case of a database error.

Default value: 5.

plugin.executor.ataccama.one.job-unlock-retry-delay

String

Specifies how long to wait between retries when unlocking a job.

Default value: 200ms. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.lock-at-most-interval

String

Determines for how long a DPE can access a particular database. Each database can be used by only one DPE at a time, which is managed by setting a scheduler lock that stores and tracks status logs in a separate table. After this interval expires, the lock is released and another DPE can connect to the same database.

This value needs to be higher than processed-event-lock-duration with a reasonable margin. If the value is lower than or equal to processed-event-lock-duration, then DPM does not start.

Default value: 5s. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.update-jobs-interval

Number

Configures how often the job queue is synced with each instance of DPM. The shared job queue contains jobs that are distributed between multiple DPMs. Jobs are then transferred from DPM to a compatible DPE handled by that DPM. Expressed in milliseconds.

Default value: 10000. The same value is used if the property is not set.

plugin.executor.ataccama.one.job-lost-timeout

String

Specifies for how long a job must remain inactive to be considered lost. Such jobs are then resubmitted if they remain in the pre-processing state, otherwise DPM attempts to cancel them.

Default value: 5s. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.subscription-job-cache-expiration

String

Configures for how long jobs can stay in the event subscriber job cache.

Default value: 5m. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.subscription-job-cache-size

Number

The number of jobs that are kept in the event subscriber job cache.

Default value: 1000.

plugin.executor.ataccama.one.job-input-storage-timeout

String

Sets for how long job input files can be stored before being assigned to a job.

Default value: 1h. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.job-input-storage-check-interval

Number

Determines how often to check for expired job input files in the job input storage. Expressed in milliseconds.

Default value: 1800000 (30 minutes).

plugin.executor.ataccama.one.job-watchdog-check-interval

String

Defines how often DPM retrieves the status of queued jobs, including their health information and use of resources in order to avoid queue starvation. This determines how often the following timeout checks are performed: submit-timeout, job-disconnected-timeout, job-at-dpm-timeout. Expressed in milliseconds.

Default value: 5000. The same value is used if the property is not set.

plugin.executor.ataccama.one.submit-timeout

String

Configures for how long a job can remain in the job queue before being submitted to a DPE. If set to 0, jobs are not submitted to DPEs and they fail immediately.

If the property is not set, jobs are kept in the queue indefinitely.

Default value: 30m. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.job-disconnected-timeout

String

Determines for how long a job can be recovered after DPE is disconnected. When the timeout is reached, the job status is set to FAILURE.

If DPM cannot reach DPE for a while, the job status first changes to DISCONNECTED. In case the connection is reestablished before the timeout expires, the job status is then set according to the information provided by DPE (for example, RUNNING, SUCCESS, FAILURE).

Default value: 1h. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.job-at-dpm-timeout

String

Specifies the maximum amount of time a job can stay in one of the following statuses: UNKNOWN, PREPROCESSING, POSTPROCESSING. When the timeout is reached, the job status is set to FAILURE.

Default value: 1h. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.job-submit-timeout

String

Specifies the maximum amount of time a job can stay in the SUBMITTED status. When the timeout is reached, the job status is set to FAILURE.

Default value: 1h. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.grpc-timeout

String

Specifies the timeout for synchronous gRPC requests on DPE such as previewing tables and counting numbers of rows during pre-processing. Increase this timeout value if jobs fail during pre-processing phases.

Default value: 60s. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.job-unlock-max-retry-count

Number

Specifies the number of attempts when unlocking a job in case of a database error.

Default value: 5.

plugin.executor.ataccama.one.job-unlock-retry-delay

String

Specifies the waiting time between attempts when unlocking a job in case of a database error.

Default value: 200ms. For a full list of accepted units, see Duration units.

plugin.executor.ataccama.one.allow-job-resubmit

Boolean

Allows resubmitting a job lost in PREPROCESSING state.

Default value: false.

plugin.executor.ataccama.one.max-page-size

Number

The maximum number of rows that should be fetched from the database when paginating.

Default value: 100.

plugin.dq-eval.ataccama.one.drillthrough-page-size

Number

The number of records that can be displayed on a single page in the shared file system.

Shutdown

Property Data type Description

Property	Data type	Description
`server.shutdown`	String	The type of application shutdown. If set to `graceful`, after receiving the shutdown signal, the application waits for any active requests to finish before proceeding. If set to `immediate`, the application shuts down as soon as the shutdown signal is received. Default value: `graceful`.
`spring.lifecycle.timeout-per-shutdown-phase`	String	Defines how long a shutdown phase can last. After this time expires, the application shuts down regardless of any active requests. Default value: `30s`. For a full list of accepted units, see Duration units.

server.shutdown

String

The type of application shutdown.

If set to graceful, after receiving the shutdown signal, the application waits for any active requests to finish before proceeding. If set to immediate, the application shuts down as soon as the shutdown signal is received.

Default value: graceful.

spring.lifecycle.timeout-per-shutdown-phase

String

Defines how long a shutdown phase can last. After this time expires, the application shuts down regardless of any active requests.

Default value: 30s. For a full list of accepted units, see Duration units.

Global runtime configuration

Starting from 14.1.0, we recommend defining the global runtime configuration primarily through DPM Admin Console. For instructions, see DPM and DPE Configuration in DPM Admin Console.

The property described in this section now serves only to provide the initial value, not as the primary method for defining the global runtime configuration.

For each job, DPM generates a specific runtime configuration or retrieves it from ONE Desktop. However, it is also possible to modify this configuration by providing a global runtime configuration that is then merged with the generated one. The global runtime configuration is supplied to DPM in the form of a base64-encoded string.

If the global runtime configuration requires the use of drivers that are not included in the default installation, you need to add these drivers to the runtime class path. To do this, include the path to the driver files, such as lib/runtime/jdbc/snowflake/*, in the plugin.executor-launch-model.ataccama.one.launch-type-properties.launch-type.cp.runtime parameter.

If you need to include multiple paths in this parameter, use the plugin.executor-launch-model.ataccama.one.launch-type-properties.launch-type.cpdelim property to specify the delimiter used to separate the paths.

For example, if the plugin.executor-launch-model.ataccama.one.launch-type-properties.launch-type.cpdelim is set to a semicolon (;), you would include the paths as follows:

plugin.executor-launch-model.ataccama.one.launch-type-properties.launch-type.cp.runtime=path1;path2;path3;path4

Property Data type Description

plugin.dqc-support.ataccama.one.dqc.runtime-config.default

String

The global runtime configuration provided as a base64-encoded string.

In XML format, runtime configurations have the following structure:

Runtime configuration example - Connecting to MinIO

<?xml version='1.0' encoding='UTF-8'?>
<runtimeconfig>
  <contributedConfigs>
    <config class="com.ataccama.dqc.s3.config.S3Contributor">
      <s3Connections>
        <s3Connection bucket="lookups" s3Compatible="MINIO" endpoint="http://localhost:9000/" secretKey="crypted:AES:KogsCeZBG2bBAgUVo5mnBlvgFKDOprxHpvlDzAItQj2T6ZhMvqDI7ugA7Azbs5EB" accessKey="minio" name="minio" sseType="NONE"/>
      </s3Connections>
    </config>
  </contributedConfigs>
</runtimeconfig>

Runtime configuration example - Connecting to a JDBC data source

<?xml version='1.0' encoding='UTF-8'?>
<runtimeconfig>
  <dataSources>
    <dataSource password="crypted:AES:<password>" name="MSOUT" driverClass="com.microsoft.sqlserver.jdbc.SQLServerDriver" user="one" url="jdbc:sqlserver://ec2-54-246-154-116.eu-west-1.compute.amazonaws.com:1433;databaseName=master"/>
  </dataSources>
</runtimeconfig>

We do not recommend using this property to modify JDBC connection strings for data sources that are already configured in DPE as it could lead to unexpected issues. Additionally, if you are using multiple DPEs, all of them need to be compatible with the global runtime configuration.

Since the runtime configuration can be later used in post-processing, the JDBC data source should be specified even if it is already declared in DPE. In order for post-processing to work correctly, the following requirements need to be fulfilled:

The <datasource> element must include the driverClass attribute instead of driverName. The driverClass needs to be specified in plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.cp.runtime property of DPE.
The driver class must be in the DQC classpath.
Post-processing components referencing the data source must use the same name of the data source as specified in the runtime configuration.

When a global runtime configuration is used, all remote executions have access to its content. For instance, if a user can access and edit the relevant post-processing components, they also have full access to the database and the data source based on the credentials provided in the global runtime configuration.

For more information, see Importing and Exporting Runtime Configuration.

Accepted units

Duration

Accepted units for time duration are as follows:

ns (nanoseconds)
us (microseconds)
ms (milliseconds)
s (seconds)
m (minutes)
h (hours)
d (days)

Size

Accepted units for file or message size are as follows:

B (bytes)
KB (kilobytes)
MB (megabytes)
GB (gigabytes)
TB (terabytes)

Was this page useful?