If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Constraints Configuration

Constraints are meant to be used as helpers for routing jobs to specific Data Processing Engines (DPEs).

By default, each DPE has access to every available data source and file system and can be employed in any environment and for any type of jobs. Using constraints, you can restrict or override the individual capabilities of a DPE instance, which are usually derived from the installed plugins and other settings.

As the capabilities of each DPE are taken into account when Data Processing Module (DPM) manages requests and assigns jobs, this can ensure that DPE has the resources to successfully handle incoming requests from DPM. In addition, you can also use constraints to provide custom capabilities.

The value of a constraining property is typically a comma-separated list of one or more string values referring to capabilities, with a straight double quote (") used as a quoting character and a backslash (\) used as an escape character. When using capabilities with different value types as elements of an enumeration, start the property value with the type specifier enclosed in square brackets ([<type_specifier>]), for example, [pattern] or [string].

You can have multiple types of elements in an enumeration, but when using properties to set them, the same type is assumed for all enumeration elements, as defined by the type specifier. The following type specifiers are allowed: [string], [pattern], [long], [integer], [double], [float], [boolean]. Intervals, custom types, and intervals of custom types are currently not supported.

If a constraint is defined as an environment variable, the same value type must be used for all elements of an enumeration. Nesting enumerations is not allowed. Furthermore, constraints used as environment properties accept only primitive data types that are supported by the previously mentioned type specifiers of property values.

In case you need to express more complex constraints, consider extending DPE with a suitable plugin that would handle most of the constraints programmatically or derive the constraints from the existing configuration.

The following properties are provided either through the Configuration Service, in the DPE deployment, or in the dpe/etc/application.properties file.

Property Data type Description

constraints.com.ataccama.dpe.plugin.dataconnect.provider.type

String

Restricts the types of data sources that DPE can access. By default, each DPE can access any available data source. To view the available values for this property, navigate to the Engines tab in DPM Admin Console and select the corresponding DPE. Then, under the Constraints section, locate the com.ataccama.dpe.plugin.dataconnect.provider.type array to find the available values.

Sample value: FILESYSTEM,GDRIVE,RDM,POSTGRESQL,ORACLE.

constraints.com.ataccama.dpe.plugin.executor.provider.type

String

Defines the types of jobs that are directed to this DPE. To view the available values for this property, navigate to the Engines tab in DPM Admin Console and select the corresponding DPE. Then, under the Constraints section, locate the com.ataccama.dpe.plugin.executor.provider.type array to find the available values. Sample value: METADATA-IMPORT,DQC,DQM,PUSHDOWN-PROFILING.

constraints.com.ataccama.dpe.plugin.dataconnect.local.fs.datasource.mount

String

Determines which file systems this DPE can access.

Default value: default.

constraints.com.ataccama.dpe.plugin.dataconnect.jdbc.connectionstring

String

Specifies to which data sources DPE can connect.

The allowed connection strings are provided here in the form of a pattern. You can also provide full connection strings for each data source.

We strongly recommend not using patterns that contain potentially sensitive values or parameters.

Default value: [pattern]jdbc:oracle:.,jdbc:postgresql:.,jdbc:mysql:.,jdbc:sqlserver:.,jdbc:h2:.*.

constraints.com.ataccama.dpe.plugin.dqc.datasource.url

String

By default, DQC data source URLs do not exclude ONE jobs using their own drivers. This property restricts that behavior in order to allow using only registered DataConnect or Metastore drivers and can be used for routing jobs to a specific DPE. The value can be also set to a custom expression instead.

Default value: [pattern].*.

constraints.com.ataccama.dpe.plugin.dataconnect.adls.container

String

Determines the storage account name and the container name of the ADLS connection that should be processed by this DPE.

The value of this constraint should be structured as follows: container_name@storage_account_name.

constraints.com.ataccama.dpe.plugin.dataconnect.managed.identity

String

Defines the managed identity ID in DPE.

constraints.com.ataccama.dpe.plugin.dataconnect.aws.instance.iam

Boolean

Allows authentication to Amazon S3 server with IAM credentials specified in Metadata Management Module (MMM). DPE has to be deployed in an EC2 instance for this property to be set.

If set to true, this DPE is ready to accept requests with IAM credentials.

Default value: [boolean]false.

Was this page useful?