Cloud Portal ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
latest (16.2.0) 16.1.0 16.0.0 15.4.0 15.3.0 15.2.0 15.1.0 14.5.x 13.9.x 13.8.x 13.7.x 13.6.x 13.5.0 13.4.x 13.3.x 13.2.x
User Community Service Desk Downloads

Project-Specific Configuration

This article describes how to configure project-specific configuration files.

There are two main configuration options:

Runtime configuration

The runtime configuration file (.runtimeConfig) provides environment-specific settings used by the embedded runtime server. This file defines resources available to the MDM Server when the server is started.

File location

In self-managed deployment, the runtime configuration file must be referenced from the corresponding .serverConfig file (see Server Setup). The location of the file is defined in the ataccama.one.mdm.config.runtime-folder application property. By default, the path is set to <ataccama.one.mdm.config.config-folder>/../runtime.

In cloud deployments, the configuration file must be placed in the preconfigured runtime folder, see MDM Custom Ataccama Cloud Deployment.

Runtime resources and parameters

The following runtime resources and parameters can be configured:

The configuration file can be created in a text editor or by exporting the current settings of folder shortcuts, data sources, and configured servers in ONE Desktop. See Export and Import Runtime Configuration.

Other runtime variables need to be configured manually according to the specifications in this article. The configuration file is an XML file in the following format:

Sample runtime configuration 
<?xml version='1.0' encoding='utf-8'?>
<runtimeconfig>

  <!-- CONTRIBUTED CONFIGS -->
  <contributedConfigs>
    <config class="com.ataccama.dqc.processor.support.UrlResourceContributor">
      <urls>
        <url name="SomeConfiguredServer" user="myusername" password="crypted:DESede:p63913D4fMa175vrXECs1nOHdV1SG5sUto5HhuV6Izg=" url="localhost:22"/>
      </urls>
    </config>
    <config class="com.ataccama.dqc.jms.config.JmsContributor">
      <jmsConnections>
        <jmsConnection connectionFactory="QueueConnectionFactory" name="someJMSbroker">
          <contextParams>
            <contextParam name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
            <contextParam name="java.naming.provider.url" value="tcp://acme.com:61616"/>
          </contextParams>
        </jmsConnection>
      </jmsConnections>
    </config>
  </contributedConfigs>

  <!-- DATA SOURCES -->
  <dataSources>
    <dataSource name="name"
                driverclass="com.mysql.jdbc.Driver"
                url="jdbc:mysql://localhost/myDatabase"
                user="root"
                password="root">
      <properties>
        <property name="name" value="value" />
      </properties>
    </dataSource>
  </dataSources>

  <!--FOLDER SHORTCUTS -->
  <pathVariables>
    <pathVariable name="MyPath" value="D:/ONE/" />
  </pathVariables>

  <!-- WORKFLOWS RESOURCES FOLDER -->
  <resourcesFolder>./resources</resourcesFolder>

  <!-- WORKFLOWS RESOURCE CONFIGURATION -->
  <resources>
    <resource id="demo" units="4" name="Demo resource" />
  </resources>
</runtimeconfig>

Drivers

MDM is distributed with a number of drivers enabling integration with data sources, message queue brokers, and other third-party components.

Databases Messaging queue providers Other

Databricks

ActiveMQ

Salesforce

DB2

Kafka, Avro

MSSQL

SQS

Oracle

PostgreSQL

S3

Snowflake

Sybase (JTDS)

Teradata
In addition, Derby, H2, and Redshift are delivered with ONE Desktop (runtime/lib/jdbc), not in the MDM Server package.

If your project requires an external library not included in the standard solution, you can install and configure those drivers as needed. See Loading Additional Drivers.

Remote server connections

Remote server connections are defined in the contributed configurations section of the runtime configuration.

Contributed configurations contain connection definitions that can be referenced from various steps and configuration files. There are several kinds of such configurations, for example, JMS servers or URLs. Contributed configurations can be created in ONE Desktop in the Servers node in the File Explorer. See Connect to a Server.

URL (generic server)

  • name - Name of the URL resource.

  • url - URL address.

  • authConfig - Select the authentication type according to the server security settings. The following options are available:

    • No authentication:

      • class - Select com.ataccama.dqc.processor.bin.config.auth.NoneAuthConfig.

    • Basic authentication:

      • class - Select com.ataccama.dqc.processor.bin.config.auth.BasicAuthConfig.

      • user - Username.

      • password - User password. The password can be either in plain or encrypted form; for encryption, you can use the onlinectl utility. See Encrypt Passwords.

    • OpenID Connect authentication:

      • class - Select com.ataccama.dqc.processor.bin.config.auth.OpenIdConnectAuthConfig.

      • clientId - Client ID.

      • clientSecret - Client secret.

      • tokenEndPointUrl - URL from which the HTTP client obtains an access token. Contact your admin.

Sample URL resource definition
<contributedConfigs>
    <config class="com.ataccama.dqc.processor.support.UrlResourceContributor">
        <urls>
            <url name="SomeConfiguredServer" url="testserver.ataccama.com:8888">
                <authConfig password="crypted:AES:tjfcHC9iTJpjZmV2y/uFKaX+WZuAZMRRSAzvVFmYVwRRWG6drUKfeBEudUYoV339" class="com.ataccama.dqc.processor.bin.config.auth.BasicAuthConfig" user="test_user"/>
            </url>
      </urls>
    </config>
</contributedConfigs>

Azure Data Lake Storage Gen 2

  • name - Name of the URL resource.

  • accountFQDN - Fully qualified domain name of the account. Can be found in user settings in Azure. Account FQDN has the <account_name>.http://azuredatalakestore.net[azuredatalakestore.net] format.

  • clientId - Client ID.

  • clientKey - Client key.

  • authenticationTokenEndpoint - URL from which the HTTP client obtains an access token. Contact your admin for details.

  • authenticateUser - Enables username and password authentication. Set to false to use Azure Active Directory (AAD) authentication methods.

  • authType - Use one of the following values to enable AAD authentication:

    • AAD_CLIENT_CREDENTIAL - If using Azure AD Service Principal with a secret.

    • AAD_MANAGED_IDENTITY - If using Azure AD Managed Identities.

Azure Data Lake Storage Gen2
<contributedConfigs>
    <config class="com.ataccama.dqc.azure.config.AzureGen2Contributor">
        <azureGen2Connections>
            <azureGen2Connection
                clientId="clientID"
                authenticateUser="false"
                <!-- authType="AAD_CLIENT_CREDENTIAL" -->
                <!-- authType="AAD_MANAGED_IDENTITY" -->
                clientKey="crypted:AES:encryptedKey"
                containerName="containerName"
                name="AzurGen2"
                storageAccount="storageAccount"
                authTokenEndpoint="https://login.microsoftonline.com/<tokenID>/oauth2/token" <!-- for use with AAD Service Principal-->
                clientId="clientID"/>
        </azureGen2Connections>
    </config>
</contributedConfigs>

JMS

  • name - Name of the URL resource.

  • connectionFactory - Connection factory class name.

  • user - Username.

  • password - User password. The password can be either in plain or encrypted form; for encryption, you can use the onlinectl utility. See Encrypt Passwords.

  • contextParams (properties) - Optional array of Java properties passed to the connection.

Sample JSM contributor definition
<contributedConfigs>
    <config class="com.ataccama.dqc.jms.config.JmsContributor">
        <jmsConnections>
            <jmsConnection connectionFactory="QueueConnectionFactory" name="someJMSbroker">
                <contextParams>
                    <contextParam name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
                    <contextParam name="java.naming.provider.url" value="tcp://acme.com:61616"/>
                </contextParams>
            </jmsConnection>
        </jmsConnections>
    </config>
</contributedConfigs>

Apache Kafka

  • name - Name of the URL resource.

  • connectionString - Comma-separated list of Kafka broker servers in the <host>:<port> format. For example, kafkabroker1.domain.com:9092,kafkabroker2.domain.com:9092.

  • properties (optional) - List of Kafka properties shared by all Kafka steps using the Kafka server connection. For a list of all possible properties, see the official Kafka documentation.

    If two properties have the same name, a property defined in a Kafka step overrides the one defined in the Kafka server connection.

To use Apache Kafka with SSL authentication, provide the following properties:

Apache Kafka with SSL authentication
<contributedConfigs>
    <config class="com.ataccama.dqc.streaming.config.KafkaContributor">
        <kafkaConnections>
            <kafkaConnection name="KafkaServer" connectString="kafkabroker1.domain.com:9092,kafkabroker2.domain.com:9092">
                <properties>
                    <property name="security.protocol" value="SSL"/>
                    <property name="ssl.truststore.location" value="/some-directory/kafka/certs/kafka.client.truststore.jks"/>
                    <property name="ssl.truststore.password" value="test1234"/>
                    <property name="ssl.keystore.location" value="/some-directory/kafka.client.keystore.jks"/>
                    <property name="ssl.keystore.password" value="test1234"/>
                    <property name="ssl.truststore.password" value="test1234"/>
                </properties>
            </kafkaConnection>
        </kafkaConnections>
    </config>
</contributedConfigs>

To use Apache Kafka in AWS Managed Streaming for Kafka (AWS MSK) with IAM authentication, provide the following properties:

Apache Kafka in AWS Managed Streaming for Kafka (AWS MSK) with IAM authentication
<contributedConfigs>
<config class="com.ataccama.dqc.streaming.config.KafkaContributor">
            <kafkaConnections>
                <kafkaConnection name="AmazonMSK" schemaRegistry="" connectString="<YourConnectString>">
                    <properties>
                        <property name="ssl.truststore.location" value="/some-directory/kafka/certs/kafka.client.truststore.jks"/>
                        <property name="ssl.truststore.password" value="test1234"/>
                        <property name="security.protocol" value="SASL_SSL"/>
                        <property name="sasl.mechanism" value="AWS_MSK_IAM"/>
                        <property name="sasl.jaas.config" value="software.amazon.msk.auth.iam.IAMLoginModule required;"/>
                        <property name="sasl.client.callback.handler.class" value="software.amazon.msk.auth.iam.IAMClientCallbackHandler"/>
                    </properties>
                </kafkaConnection>
            </kafkaConnections>
        </config>
</contributedConfigs>

Amazon S3

  • clientEncryptKey - A key to encrypt the data on the client side. By default, Java limits the maximum key length for encryption to 128 bits. To remove the key length restriction, download JCE Unlimited Strength policy files to the <JAVA_HOME>/lib/security folder.

  • authType - Use one of the following values to enable S3 server authentication:

    • AWS_INSTANCE_IAM - If using IAM roles assigned to EC2 instance.

    • AWS_ACCESS_KEY - If using the access key and secret key.

    • AWS_WEB_IDENTITY_TOKEN - If using service accounts and assigning IAM roles to Kubernetes pods.

  • secretKey - Secret access key associated with the S3 account.

  • accessKey - Access key associated with the S3 account.

  • name - The server connection name.

  • sseKey - Select the encryption key from the keys generated by the server. If you leave this field empty, a default service key (generated by the server on a customer by service by region level) is used. The field is available only with SSE-KMS server encryption.

  • sseType - Specifies how the server encrypts the data. The following options are available:

    • None: No server-side data encryption. Default value.

    • SSE-S3: Encryption key is generated and selected by the S3 server.

    • SSE-KMS: Encryption key is selected by a user from the keys generated on the server.

  • url - The server URL in the s3a://<bucket>[/<directory>] format.

  • assumeRole (optional) - Allows to assume a role from a different AWS account (such as AWS Access Key, EC2 Instance Role, or Web Identity Role) instead of using the user’s known identity.

    • enabled - If true, specifies that the assume role feature is enabled.

    • roleArn - The Amazon Resource Name (ARN) of the IAM role to be assumed.

    • sessionName - The session name, used to identify the connection in AWS logs.

    • externalId - The optional external ID used in the trust relationship between accounts.

    • region - The AWS region where the Security Token Service (STS) should be used.

    • class - The class that handles the configuration for the AWS assume role feature.

Sample Amazon S3 resource contributor definition
<contributedConfigs>
    <config class="com.ataccama.dqc.s3.config.S3Contributor">
      <s3Connections>
        <s3Connection
              <!-- authType="AWS_ACCESS_KEY" -->
              <!-- authType="AWS_INSTANCE_IAM" -->
              <!-- authType="AWS_WEB_IDENTITY_TOKEN" -->
              secretKey="crypted:AES:PIJhJbDIXJbr7Gahr67XPNevfmi7X7/QnEMlkW51Ob9pSiNyAkFTplVtwofD52ZLn64h235DICo+hLKNvFkABQ=="
              accessKey="AKIAJAWAMV3F3O37TPTA"
              name="s3" sseKey="SERVER_KEY_ID"
              sseType="NONE"
              url="https://ataccama.s3.amazonaws.com">
            <assumeRole
                enabled="true"
                roleArn="arn:aws:iam::111111111111:role/cross_account_role"
                sessionName="Ataccama_One"
                externalId="ataccama_one_external_id"
                region="us-east-1"
                class="com.ataccama.dqc.aws.auth.runtime.config.AwsAssumeRoleAuthConfig"/>
        </s3Connection>
      </s3Connections>
    </config>
</contributedConfigs>

SMTP

  • host - The SMTP server host.

  • port - The connection port used by the server.

  • user - Username.

  • password - Password for the specified user. The password can be either in plain or encrypted form; for encryption, you can use the onlinectl utility. See encrypt-passwords.adoc.

Sample SMTP resource contributor definition
<contributedConfigs>
    <config class="com.ataccama.dqc.processor.support.SmtpResourceContributor">
        <smtpConnections>
            <smtpConnection password="crypted:AES:5rNM3amiDCHjOSo3PRdF4scrNEHMhzeKmMr8TlRjLbFvaoDyY18kR8SpS1TXUm/o" port="25" host="smtpserver.company.com" name="SMTPServer" user="test_user"/>
        </smtpConnections>
    </config>
</contributedConfigs>
Additional SMTP configuration

In cloud deployments, SMTP server is configured by default.

In self-managed deployments, additional configuration is required. Provide the following Java parameters to the server:

  • -Dmail.smtp.auth=true - Enables SMTP authentication. If set to true, the SMTP client must provide credentials (username and password) to authenticate with the SMTP server.

  • -Dmail.smtp.starttls.enable=true - Enables STARTTLS, which allows upgrading a plain text connection to an encrypted TLS connection.

  • -Dmail.smtp.ssl.protocols=TLSv1.2 - Defines which TLS protocol to use.

You can set these parameters in the JAVA_OPTS environment variable or using the command line.

Set JAVA_OPTS - example
set JAVA_OPTS="-Dmail.smtp.auth=true -Dmail.smtp.starttls.enable=true -Dmail.smtp.ssl.protocols=TLSv1.2"

Data source

The data source represents the information needed for a data source connection (for example, for connection to a database). Data sources can be created in ONE Desktop in the Databases node in the File Explorer. See Databases.

  • dataSource

    • name - Name of the data source.

    • driverClass - Driver used to connect to the data source.

    • url - URL address of the data source.

    • ata.jdbc.aad.authType - Authentication using Azure AD:

      • AAD_MANAGED_IDENTITY - If using AAD Managed Identity.

      • AAD_SERVICE_PRINCIPAL - If using AAD Service Principal with a secret.

    • ata.jdbc.aad.keyvault.authType - Authentication using Azure AD with Azure Key Vault:

      • AAD_MANAGED_IDENTITY - If using AAD Managed Identity with Azure Key Vault.

      • AAD_SERVICE_PRINCIPAL - If using AAD Service Principal with a secret kept in Azure Key Vault.

    • ata.jdbc.aad.keyvault.vaultUrl - URL of the Azure Key Vault service.

    • ata.jdbc.aad.keyvault.clientId - Your client ID that is used to authenticate using Azure Key Vault.

    • ata.jdbc.aad.keyvault.tenantId - Your client ID for Azure Key Vault.

    • ata.jdbc.aad.keyvault.clientSecret - We recommend storing your secret for the service principal in the <propertiesEncrypted> section although it can be stored unencrypted too.

    • user - Username.

    • password - User password. The password can be either in plain or encrypted form; for encryption, you can use the onlinectl utility. See Encrypt Passwords.

    • validationQuery - An SQL SELECT command used to validate a database connection prior to using it.

    • minSize - The minimum number of established connections that are kept in the connection pool at all times. Default value: 1.

      Example: If you start the online server with minSize set to 2, then two database connections are established automatically after the server is started.

    • maxIdleSize - The maximum number of inactive connections that are kept in the connection pool. All inactive connections exceeding maxIdleSize are disposed of automatically. Default value: 10.

    • maxAge - The maximum time (in milliseconds) an inactive connection can be (re)used in the connection pool. Default value: 0 (unlimited).

      Example: If maxAge is set to 10000, the particular connection is reused only in the time interval of 10 seconds; if there is another connection request after this interval (for example, you run a ONE plan with JDBC Reader), the mentioned connection is closed and a new connection is established and used instead.

  • properties - Properties related to the selected database engine (refer to the respective engine documentation, for example, Oracle Driver documentation or MySQL Driver documentation).

    • name - Name of the property (for example, user or defaultRowPrefetch in an Oracle database).

    • value - Value of the property.

  • propertiesEncrypted - Stores the value of the property in the AES encrypted format.

    • ata.jdbc.aad.clientSecret - Your secret for Azure AD Service Principal.

      ata.jdbc.aad.keyvault.authType can be used for Databricks, Snowflake, Microsoft SQL Server, and other connections that can be configured via the properties parameter in the runtime configuration.
Sample data source definition
<dataSources>
    <dataSource
                name="name"
                driverclass="com.mysql.jdbc.Driver"
                url="jdbc:mysql://localhost/myDatabase"
                user="root"
                password="root"
                validationQuery="select 1"
                minSize="2"
                maxIdleSize="5"
                maxAge="60000">
        <properties>
            <property name="connectTimeout" value="0" />
        </properties>
    </dataSource>
</dataSources>

Connect to Databricks

If you want to browse Databricks JDBC tables in ONE Desktop, you need to add these properties to the runtime configuration and then import them to ONE Desktop.

The authentication methods for connecting ONE Runtime Server to Databricks as a data source using JDBC (without Ataccama Data Processing modules) are described in the comments in the following example:

Sample Databricks definition
<dataSources>
    <dataSource
                name="Databricks"
                driverName="Databricks"
                url="jdbc:spark://.../;AuthMech=11;Auth_Flow=0;">
         <properties>
            <!-- Generate personal token for authentication at Databricks. -->
            <!-- <property name="ata.jdbc.authType" value="PERSONAL_TOKEN"/> -->

            <!-- Use Azure Active Directory Service Principal with a secret.
We recommend storing your secret in <propertiesEncrypted> section. -->
            ata.jdbc.authType=AAD_SERVICE_PRINCIPAL
            ata.jdbc.aad.tenantUrl="https://login.microsoftonline.com/<TENANT_ID>/oauth2/v2.0/token"
            ata.jdbc.aad.clientId="<CLIENT_ID>"
            ata.jdbc.aad.resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"

            <!-- Use Azure Active Directory Managed Identity for Azure sources.
This method works only for clusters on the same Azure VM as Azure MSI. -->
            ata.jdbc.authType="AAD_MANAGED_IDENTITY"
            ata.jdbc.tokenPropertyKey="<Auth_AccessToken>"
            ata.jdbc.aad.resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"

<!-- Databricks Resource ID at Azure is not cluster-specific and does not change.-->
        </properties>
        <propertiesEncrypted>
            ata.jdbc.aad.clientSecret="<ENCRYPTED_SECRET>"
        </propertiesEncrypted>
    </dataSource>
</dataSources>

Folder shortcuts

A path to a file can be specified as an absolute path, a relative path, or using folder shortcuts. A folder shortcut is a named path to a file or folder. Folder shortcuts can be created in the Folder Shortcuts node in the File Explorer. For detailed instructions, see Folder Shortcuts.

  • name - Name of folder shortcut.

  • value - Folder represented by this shortcut.

Sample folder shortcut definition
<pathVariables>
    <pathVariable name="MyPath" value="D:/ONE/" />
</pathVariables>

For example, if the actual path to the file is D:/ONE/MyProject/config.xml, the name could be set to MyPath (arbitrary value) and the value to D:/ONE/. In that case, when using the folder shortcut, the path would look as follows: pathvar://MyPath/MyProject/config.xml.

Resources folder and configuration for workflows

If the orchestration workflow is used, you can also configure:

Was this page useful?