If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Runtime Configuration

The runtime configuration file defines resources available to ONE Runtime Server in the batch mode and when the online server is started.

The batch mode is implemented via the runcif utility ([installation folder]/runtime/bin) where the path to the runtime configuration file is supplied by the -runtimeConfig <filename> parameter:

runcif.sh -server -serverPort 4040 -runtimeConfig example.runtimeConfig example.plan

The runtime configuration file is referenced in the Server Configuration file, so the resources defined in the runtime configuration are available to the online server too.

The default runtime configuration file is default.runtimeConfig, located in [installation folder]/runtime/server/etc/. Use this file to create your own runtime configuration.

Runtime resources and parameters

The following runtime resources and parameters can be configured:

Contributed configurations (remote server connections)
Data sources
Folder shortcuts
Runtime components
Initial parallelism level
Logging
Resources folder for workflows
Resources configuration for workflows

The configuration file can be created in a text editor or by exporting the current settings of folder shortcuts, data sources, and configured servers in ONE Desktop. See Export and Import Runtime Configuration.

Other runtime variables need to be configured manually according to the specifications in this article. The configuration file is an XML file in the following format:

Sample runtime configuration

<?xml version='1.0' encoding='utf-8'?>
<runtimeconfig>

  <!-- CONTRIBUTED CONFIGS -->
  <contributedConfigs>
    <config class="com.ataccama.dqc.processor.support.UrlResourceContributor">
      <urls>
        <url name="SomeConfiguredServer" user="myusername" password="crypted:DESede:p63913D4fMa175vrXECs1nOHdV1SG5sUto5HhuV6Izg=" url="localhost:22"/>
      </urls>
    </config>
    <config class="com.ataccama.dqc.jms.config.JmsContributor">
      <jmsConnections>
        <jmsConnection connectionFactory="QueueConnectionFactory" name="someJMSbroker">
          <contextParams>
            <contextParam name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
            <contextParam name="java.naming.provider.url" value="tcp://acme.com:61616"/>
          </contextParams>
        </jmsConnection>
      </jmsConnections>
    </config>
  </contributedConfigs>

  <!-- DATA SOURCES -->
  <dataSources>
    <dataSource name="name"
                driverclass="com.mysql.jdbc.Driver"
                url="jdbc:mysql://localhost/myDatabase"
                user="root"
                password="root">
      <properties>
        <property name="name" value="value" />
      </properties>
    </dataSource>
  </dataSources>

  <!--FOLDER SHORTCUTS -->
  <pathVariables>
    <pathVariable name="MyPath" value="D:/ONE/" />
  </pathVariables>

  <!-- RUNTIME COMPONENTS -->
  <runtimeComponents>
    <runtimeComponent class="com.ataccama.dqc.processor.monitoring.file.FileLoggerComp"
                      fileName="filename"
                      stdout="true"
                      loggingIntervalInMins="1" />
  </runtimeComponents>

  <!-- PARALLELISM LEVEL -->
  <parallelismLevel>2</parallelismLevel>

  <!-- LOGGING CONFIGURATION FILE -->
  <loggingConfig>./etc/logging.xml</loggingConfig>

  <!-- WORKFLOWS RESOURCES FOLDER -->
  <resourcesFolder>./resources</resourcesFolder>

  <!-- WORKFLOWS RESOURCE CONFIGURATION -->
  <resources>
    <resource id="demo" units="4" name="Demo resource" />
  </resources>
</runtimeconfig>

Contributed configurations

Contributed configurations contain connection definitions that can be referenced from various steps and configuration files. There are several kinds of such configurations, for example, JMS servers or URLs. Contributed configurations can be created in ONE Desktop in the Servers node in the File Explorer. See Connect to a Server.

URL (generic server)

name - Name of the URL resource.
url - URL address.
authConfig - Select the authentication type according to the server security settings. The following options are available:
- No authentication:
  - class - Select com.ataccama.dqc.processor.bin.config.auth.NoneAuthConfig.
- Basic authentication:
  - class - Select com.ataccama.dqc.processor.bin.config.auth.BasicAuthConfig.
  - user - Username.
  - password - User password. The password can be either in plain or encrypted form; for encryption, you can use the onlinectl utility. See Encrypt Passwords.
- OpenID Connect authentication:
  - class - Select com.ataccama.dqc.processor.bin.config.auth.OpenIdConnectAuthConfig.
  - clientId - Client ID.
  - clientSecret - Client secret.
  - tokenEndPointUrl - URL from which the HTTP client obtains an access token. Contact your admin.

Sample URL resource definition

<contributedConfigs>
    <config class="com.ataccama.dqc.processor.support.UrlResourceContributor">
        <urls>
            <url name="SomeConfiguredServer" url="testserver.ataccama.com:8888">
                <authConfig password="crypted:AES:tjfcHC9iTJpjZmV2y/uFKaX+WZuAZMRRSAzvVFmYVwRRWG6drUKfeBEudUYoV339" class="com.ataccama.dqc.processor.bin.config.auth.BasicAuthConfig" user="test_user"/>
            </url>
      </urls>
    </config>
</contributedConfigs>

Azure Data Lake Storage Gen 1

name - Name of the URL resource.
accountFQDN - Fully qualified domain name of the account. Can be found in user settings in Azure. Account FQDN has the <account_name>.azuredatalakestore.net format.
clientId - Client ID.
clientKey - Client key.
authenticationTokenEndpoint - URL from which the HTTP client obtains an access token. Contact your admin for details.
authenticateUser - Enables username and password authentication. Set to false (the feature is not yet supported by ADLS).

Sample Azure connection definition

<contributedConfigs>
    <config class="com.ataccama.dqc.azure.config.AzureContributor">
        <azureConnections>
            <azureConnection clientId="00000000-0000-0000-0000-000000000000" authenticateUser="false" clientKey="crypted:AES:vKYpslbnAZVGc8dKV5XB8eAJ0iDlESofid/IZtlYIJKFMVsWtXuazDeOfyK4GPVjgb3L1Frd0yniWHyGfcFYa5PpmEy+oMju6ADsDNuzkQE=" name="ADLSServer" accountFQDN="myaccount.azuredatalakestore.net" authTokenEndpoint="https://login.microsoftonline.com/11111111-1111-1111-1111-111111111111/oauth2/token"/>
        </azureConnections>
    </config>
</contributedConfigs>

Azure Data Lake Storage Gen 2

name - Name of the URL resource.
accountFQDN - Fully qualified domain name of the account. Can be found in user settings in Azure. Account FQDN has the <account_name>.http://azuredatalakestore.net[azuredatalakestore.net] format.
clientId - Client ID.
clientKey - Client key.
authenticationTokenEndpoint - URL from which the HTTP client obtains an access token. Contact your admin for details.
authenticateUser - Enables username and password authentication. Set to false to use Azure Active Directory (AAD) authentication methods.
authType - Use one of the following values to enable AAD authentication:
- AAD_CLIENT_CREDENTIAL - If using Azure AD Service Principal with a secret.
- AAD_MANAGED_IDENTITY - If using Azure AD Managed Identities.

Azure Data Lake Storage Gen2

<contributedConfigs>
    <config class="com.ataccama.dqc.azure.config.AzureGen2Contributor">
        <azureGen2Connections>
            <azureGen2Connection
                clientId="clientID"
                authenticateUser="false"
                <!-- authType="AAD_CLIENT_CREDENTIAL" -->
                <!-- authType="AAD_MANAGED_IDENTITY" -->
                clientKey="crypted:AES:encryptedKey"
                containerName="containerName"
                name="AzurGen2"
                storageAccount="storageAccount"
                authTokenEndpoint="https://login.microsoftonline.com/<tokenID>/oauth2/token" <!-- for use with AAD Service Principal-->
                clientId="clientID"/>
        </azureGen2Connections>
    </config>
</contributedConfigs>

Google Cloud Storage

bucket - The bucket URL associated with your project within the Google Cloud Platform.
keyFile - The key file (.json or .p12) location on your local hard drive.
name - The server connection name.
projectId - The project ID associated with your project within the Google Cloud Platform.

Sample Google Cloud Storage resource contributor definition

<contributedConfigs>
    <config class="com.ataccama.dqc.google.config.GoogleContributor">
        <googleConnections>
            <googleConnection bucket="ataccama_example" keyFile="C:/Users/test-qa-235308-e21d12de6ee9.json" name="GoogleCloudStorage" projectId="test-qa-235308"/>
        </googleConnections>
    </config>
</contributedConfigs>

JMS

name - Name of the URL resource.
connectionFactory - Connection factory class name.
user - Username.
password - User password. The password can be either in plain or encrypted form; for encryption, you can use the onlinectl utility. See Encrypt Passwords.
contextParams (properties) - Optional array of Java properties passed to the connection.

Sample JSM contributor definition

<contributedConfigs>
    <config class="com.ataccama.dqc.jms.config.JmsContributor">
        <jmsConnections>
            <jmsConnection connectionFactory="QueueConnectionFactory" name="someJMSbroker">
                <contextParams>
                    <contextParam name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
                    <contextParam name="java.naming.provider.url" value="tcp://acme.com:61616"/>
                </contextParams>
            </jmsConnection>
        </jmsConnections>
    </config>
</contributedConfigs>

Apache Kafka

name - Name of the URL resource.
connectionString - Comma-separated list of Kafka broker servers in the <host>:<port> format. For example, kafkabroker1.domain.com:9092,kafkabroker2.domain.com:9092.
properties (optional) - List of Kafka properties shared by all Kafka steps using the Kafka server connection. For a list of all possible properties, see the official Kafka documentation.

If two properties have the same name, a property defined in a Kafka step overrides the one defined in the Kafka server connection.

To use Apache Kafka with SSL authentication, provide the following properties:

Apache Kafka with SSL authentication

<contributedConfigs>
    <config class="com.ataccama.dqc.streaming.config.KafkaContributor">
        <kafkaConnections>
            <kafkaConnection name="KafkaServer" connectString="kafkabroker1.domain.com:9092,kafkabroker2.domain.com:9092">
                <properties>
                    <property name="security.protocol" value="SSL"/>
                    <property name="ssl.truststore.location" value="/some-directory/kafka/certs/kafka.client.truststore.jks"/>
                    <property name="ssl.truststore.password" value="test1234"/>
                    <property name="ssl.keystore.location" value="/some-directory/kafka.client.keystore.jks"/>
                    <property name="ssl.keystore.password" value="test1234"/>
                    <property name="ssl.truststore.password" value="test1234"/>
                </properties>
            </kafkaConnection>
        </kafkaConnections>
    </config>
</contributedConfigs>

To use Apache Kafka in AWS Managed Streaming for Kafka (AWS MSK) with IAM authentication, provide the following properties:

Apache Kafka in AWS Managed Streaming for Kafka (AWS MSK) with IAM authentication

<contributedConfigs>
<config class="com.ataccama.dqc.streaming.config.KafkaContributor">
            <kafkaConnections>
                <kafkaConnection name="AmazonMSK" schemaRegistry="" connectString="<YourConnectString>">
                    <properties>
                        <property name="ssl.truststore.location" value="/some-directory/kafka/certs/kafka.client.truststore.jks"/>
                        <property name="ssl.truststore.password" value="test1234"/>
                        <property name="security.protocol" value="SASL_SSL"/>
                        <property name="sasl.mechanism" value="AWS_MSK_IAM"/>
                        <property name="sasl.jaas.config" value="software.amazon.msk.auth.iam.IAMLoginModule required;"/>
                        <property name="sasl.client.callback.handler.class" value="software.amazon.msk.auth.iam.IAMClientCallbackHandler"/>
                    </properties>
                </kafkaConnection>
            </kafkaConnections>
        </config>
</contributedConfigs>

Ataccama ONE Platform

The configuration takes a number of URL addresses:

Keycloak token URL: value="https://<CLIENT>.<ENV>.ataccama.online/auth".
ONE Data Processing (connection to Data Processing Module, dpm):
- DPM Admin Console URL: url="https://dpm-<CLIENT>.<ENV>.ataccama.online".
- gRPC host: host="https://dpm-grpc-<CLIENT>.<ENV>.ataccama.online".
  - port: The gRPC port where DPM is available.
  - tls: Set to true to use TLS.
ONE Metadata Server (connection to Metadata Management Module, mmm):
- GraphQL URL: apiURL="https://<CLIENT>.<ENV>.ataccama.online/graphql".
- gRPC host (the URL where ONE is running): url="https://<CLIENT>.<ENV>.ataccama.online".

Other properties:

name - The server connection name.
password - Password for the specified user.
user - Username.
type - Selected authentication type:
- OpenID Connect - Authentication using OpenID (recommended).
- Basic - Authentication with a username and a password.
  
  If you are using OpenID Connect, additional authentication properties are required, as shown in the following example.
dpm public key - Public access key for DPM.

Sample Ataccama ONE Platform resource contributor definition

<contributedConfigs>
    <config class="com.ataccama.one2.client.config.One2Contributor">
        <one2Connections>
            <one2Connection name="ONE">
                <auth password="crypted:AES:n2kQSR0s5fqnfVcx7IgGumXYD02Wfx2kSDpR8MFyjAc=" type="OpenID Connect" user="admin">
                    <properties>
                        <authProperty name="openid-connect.token.url" value="https://<CLIENT>.<ENV>.ataccama.online/auth"/>
                        <authProperty name="openid-connect.realm" value="ataccamaone"/>
                        <authProperty name="openid-connect.client.id" value="one-desktop-public-client"/>
                    </properties>
                </auth>
                <dpm publicKey="{"kty":"EC","crv":"P-256","kid":"rsS16kdWaPWHmQysa6kC4lL1xqYWJXfB-Uydd6SQjLc","x":"NQtIiwPXdYyvhXGxtoOBPn9zztHNO8dU8TQUc-S7IlU","y":"-2RFU45NJNSCDiRG6yEEsP8WTPt_6Mgnb6UIujA4H7I","alg":"ES256"}" url="https://dpm-<CLIENT>.<ENV>.ataccama.online">
                    <grpc port="443" host="dpm-grpc-<CLIENT>.<ENV>.ataccama.online" tls="true"/>
                </dpm>
                <mmm apiUrl="https://<CLIENT>.<ENV>.ataccama.online/graphql" url="https://<CLIENT>.<ENV>.ataccama.online"/>
            </one2Connection>
        </one2Connections>
    </config>
</contributedConfigs>

Amazon S3

clientEncryptKey - A key to encrypt the data on the client side. By default, Java limits the maximum key length for encryption to 128 bits. To remove the key length restriction, download JCE Unlimited Strength policy files to the <JAVA_HOME>/lib/security folder.
authType - Use one of the following values to enable S3 server authentication:
- AWS_INSTANCE_IAM - If using IAM roles assigned to EC2 instance.
- AWS_ACCESS_KEY - If using the access key and secret key.
- AWS_WEB_IDENTITY_TOKEN - If using service accounts and assigning IAM roles to Kubernetes pods.
secretKey - Secret access key associated with the S3 account.
accessKey - Access key associated with the S3 account.
name - The server connection name.
sseKey - Select the encryption key from the keys generated by the server. If you leave this field empty, a default service key (generated by the server on a customer by service by region level) is used. The field is available only with SSE-KMS server encryption.
sseType - Specifies how the server encrypts the data. The following options are available:
- None: No server-side data encryption. Default value.
- SSE-S3: Encryption key is generated and selected by the S3 server.
- SSE-KMS: Encryption key is selected by a user from the keys generated on the server.
url - The server URL in the s3a://<bucket>[/<directory>] format.
assumeRole (optional) - Allows to assume a role from a different AWS account (such as AWS Access Key, EC2 Instance Role, or Web Identity Role) instead of using the user’s known identity.
- enabled - If true, specifies that the assume role feature is enabled.
- roleArn - The Amazon Resource Name (ARN) of the IAM role to be assumed.
- sessionName - The session name, used to identify the connection in AWS logs.
- externalId - The optional external ID used in the trust relationship between accounts.
- region - The AWS region where the Security Token Service (STS) should be used.
- class - The class that handles the configuration for the AWS assume role feature.

Sample Amazon S3 resource contributor definition

<contributedConfigs>
    <config class="com.ataccama.dqc.s3.config.S3Contributor">
      <s3Connections>
        <s3Connection
              <!-- authType="AWS_ACCESS_KEY" -->
              <!-- authType="AWS_INSTANCE_IAM" -->
              <!-- authType="AWS_WEB_IDENTITY_TOKEN" -->
              secretKey="crypted:AES:PIJhJbDIXJbr7Gahr67XPNevfmi7X7/QnEMlkW51Ob9pSiNyAkFTplVtwofD52ZLn64h235DICo+hLKNvFkABQ=="
              accessKey="AKIAJAWAMV3F3O37TPTA"
              name="s3" sseKey="SERVER_KEY_ID"
              sseType="NONE"
              url="https://ataccama.s3.amazonaws.com">
            <assumeRole
                enabled="true"
                roleArn="arn:aws:iam::111111111111:role/cross_account_role"
                sessionName="Ataccama_One"
                externalId="ataccama_one_external_id"
                region="us-east-1"
                class="com.ataccama.dqc.aws.auth.runtime.config.AwsAssumeRoleAuthConfig"/>
        </s3Connection>
      </s3Connections>
    </config>
</contributedConfigs>

SMTP

host - The SMTP server host.
port - The connection port used by the server.
user - Username.
password - Password for the specified user. The password can be either in plain or encrypted form; for encryption, you can use the onlinectl utility. See Encrypt Passwords.

Sample SMTP resource contributor definition

<contributedConfigs>
    <config class="com.ataccama.dqc.processor.support.SmtpResourceContributor">
        <smtpConnections>
            <smtpConnection password="crypted:AES:5rNM3amiDCHjOSo3PRdF4scrNEHMhzeKmMr8TlRjLbFvaoDyY18kR8SpS1TXUm/o" port="25" host="smtpserver.company.com" name="SMTPServer" user="test_user"/>
        </smtpConnections>
    </config>
</contributedConfigs>

Keycloak deployment connection

Define the configuration for your Keycloak clients. The settings in the KeycloakDeploymentContributor should correspond to the Keycloak settings for the client. The Keycloak client configuration is mapped to the URL pattern in HTTP Dispatcher.

The option to define Keycloak Deployment Connection is available only for the applications running on ONE Runtime Server (such as ONE Runtime Server Admin). Currently, it does not work with web applications.

keycloakConfigs - Define one or multiple Keycloak configurations. We recommend defining a separate configuration for each Keycloak realm.

keycloakConfig

name - Unique configuration name.

clients - Define one or multiple Keycloak clients.

client - Keycloak client configuration.

id - Unique client ID. Should correspond to the client name in Keycloak.
url - Base URL of the Keycloak server. Should be defined for each client (either directly in the client configuration or inherited from the keycloakConfig parent).
realm - Name of the Keycloak realm. Should be defined for each client (either directly in the client configuration or inherited from the keycloakConfig parent).
secret (optional if using a public client) - Should correspond to the secret in the Keycloak admin console. The password can be either in plain or encrypted form; for encryption, you can use the onlinectl utility. See Encrypt Passwords.

attributes - List of all other Keycloak client configuration attributes.

The url, realm, and attribute properties can be defined either for all clients (as an attribute of the keycloakConfig property) or for individual clients (as an attribute of the client property). In case a property is defined in both places, the client value overrides the value from the parent keycloakConfig.

Sample Keycloak Deployment contributor configuration for the ONE Runtime Server client

<contributedConfigs>
    <config class="com.ataccama.server.keycloak.KeycloakDeploymentContributor">
        <keycloakConfigs>
            <keycloakConfig name="localKeycloak">
                <!-- Define common parameters for all clients.
They can be overridden by client-specific settings.-->
                <url>http://localhost:8083/auth</url>
                <realm>ataccamaone</realm>
                <attributes>
                    <attribute name="ssl-required" value="external"/>
                </attributes>
                <clients>
                    <client id="one-admin-center">
                        <secret>crypted:AES:DZ+36XQlju1sAAAIS6YUxtbN603Ag+Qxz3mLrNeNnSo=</secret>
                        <attributes>
                            <!-- Define client-specific settings.-->
                            <attribute name="use-resource-role-mappings" value="false"/>
                            <attribute name="public-client" value="false"/>
                            <attribute name="bearer-only" value="false"/>
                            <attribute name="autodetect-bearer-only" value="false"/>
                            <attribute name="always-refresh-token" value="false"/>
                            <attribute name="principal-attribute" value="preferred_username"/>
                        </attributes>
                    </client>
                </clients>
            </keycloakConfig>
        </keycloakConfigs>
    </config>
</contributedConfigs>

Data source

The data source represents the information needed for a data source connection (for example, for connection to a database). Data sources can be created in ONE Desktop in the Databases node in the File Explorer. See Databases.

dataSource
- name - Name of the data source.
- driverClass - Driver used to connect to the data source.
- url - URL address of the data source.
- ata.jdbc.aad.authType - Authentication using Azure AD:
  - AAD_MANAGED_IDENTITY - If using AAD Managed Identity.
  - AAD_SERVICE_PRINCIPAL - If using AAD Service Principal with a secret.
- ata.jdbc.aad.keyvault.authType - Authentication using Azure AD with Azure Key Vault:
  - AAD_MANAGED_IDENTITY - If using AAD Managed Identity with Azure Key Vault.
  - AAD_SERVICE_PRINCIPAL - If using AAD Service Principal with a secret kept in Azure Key Vault.
- ata.jdbc.aad.keyvault.vaultUrl - URL of the Azure Key Vault service.
- ata.jdbc.aad.keyvault.clientId - Your client ID that is used to authenticate using Azure Key Vault.
- ata.jdbc.aad.keyvault.tenantId - Your client ID for Azure Key Vault.
- ata.jdbc.aad.keyvault.clientSecret - We recommend storing your secret for the service principal in the <propertiesEncrypted> section although it can be stored unencrypted too.
- user - Username.
- password - User password. The password can be either in plain or encrypted form; for encryption, you can use the onlinectl utility. See Encrypt Passwords.
- validationQuery - An SQL SELECT command used to validate a database connection prior to using it.
- minSize - The minimum number of established connections that are kept in the connection pool at all times. Default value: 1.
  
  Example: If you start the online server with minSize set to 2, then two database connections are established automatically after the server is started.
- maxIdleSize - The maximum number of inactive connections that are kept in the connection pool. All inactive connections exceeding maxIdleSize are disposed of automatically. Default value: 10.
- maxAge - The maximum time (in milliseconds) an inactive connection can be (re)used in the connection pool. Default value: 0 (unlimited).
  
  Example: If maxAge is set to 10000, the particular connection is reused only in the time interval of 10 seconds; if there is another connection request after this interval (for example, you run a ONE plan with JDBC Reader), the mentioned connection is closed and a new connection is established and used instead.
properties - Properties related to the selected database engine (refer to the respective engine documentation, for example, Oracle Driver documentation or MySQL Driver documentation).
- name - Name of the property (for example, user or defaultRowPrefetch in an Oracle database).
- value - Value of the property.

propertiesEncrypted - Stores the value of the property in the AES encrypted format.

ata.jdbc.aad.clientSecret - Your secret for Azure AD Service Principal.

ata.jdbc.aad.keyvault.authType can be used for Databricks, Snowflake, Microsoft SQL Server, and other connections that can be configured via the properties parameter in the runtime configuration.

Sample data source definition

<dataSources>
    <dataSource
                name="name"
                driverclass="com.mysql.jdbc.Driver"
                url="jdbc:mysql://localhost/myDatabase"
                user="root"
                password="root"
                validationQuery="select 1"
                minSize="2"
                maxIdleSize="5"
                maxAge="60000">
        <properties>
            <property name="connectTimeout" value="0" />
        </properties>
    </dataSource>
</dataSources>

Connect to Databricks

If you want to browse Databricks JDBC tables in ONE Desktop, you need to add these properties to the runtime configuration and then import them to ONE Desktop.

The authentication methods for connecting ONE Runtime Server to Databricks as a data source using JDBC (without Ataccama Data Processing modules) are described in the comments in the following example:

Sample Databricks definition

<dataSources>
    <dataSource
                name="Databricks"
                driverName="Databricks"
                url="jdbc:spark://.../;AuthMech=11;Auth_Flow=0;">
         <properties>
            <!-- Generate personal token for authentication at Databricks. -->
            <!-- <property name="ata.jdbc.authType" value="PERSONAL_TOKEN"/> -->

            <!-- Use Azure Active Directory Service Principal with a secret.
We recommend storing your secret in <propertiesEncrypted> section. -->
            ata.jdbc.authType=AAD_SERVICE_PRINCIPAL
            ata.jdbc.aad.tenantUrl="https://login.microsoftonline.com/<TENANT_ID>/oauth2/v2.0/token"
            ata.jdbc.aad.clientId="<CLIENT_ID>"
            ata.jdbc.aad.resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"

            <!-- Use Azure Active Directory Managed Identity for Azure sources.
This method works only for clusters on the same Azure VM as Azure MSI. -->
            ata.jdbc.authType="AAD_MANAGED_IDENTITY"
            ata.jdbc.tokenPropertyKey="<Auth_AccessToken>"
            ata.jdbc.aad.resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"

<!-- Databricks Resource ID at Azure is not cluster-specific and does not change.-->
        </properties>
        <propertiesEncrypted>
            ata.jdbc.aad.clientSecret="<ENCRYPTED_SECRET>"
        </propertiesEncrypted>
    </dataSource>
</dataSources>

Database drivers

The runtime configuration file is generated as follows. The root runtimeconfig element contains a databaseDrivers element with the name and definitions of database drivers. When this approach is used, there is no need to store drivers (.jar files) in the <ATACCAMA_HOME>/lib folder, even if you run a workflow with *Run DQC as Process or Run DQC on Cluster.

The classpath of the driver can contain:

Concrete *.jar files.
Folders with *.jar files.
Folders.

A separate classloader is created for such a driver. All required .jar files have to be included in the classpath entries. Driver .jar files should not be specified in cp/lcp properties.

Runtime configuration file can contain data sources defined using the old approach (attribute driverClass) for backward compatibility. In that case, .jar files have to be placed in the <ATACCAMA_HOME>/lib folder. Driver .jar files have to be specified in cp/lcp properties.

When the runtime configuration file is imported, driver definitions are imported as well. If the same name exists in the ONE Desktop driver definition, it is used, otherwise the new driver definition is created from the runtime configuration file.

If some file is selected (for example, in the File Explorer) and the Import Runtime Configuration wizard is open, the file is filled in in the wizard. On the Database preference screen, you can add and edit database drivers.

Sample data source definition with database drivers

<?xml version='1.0' encoding='UTF-8'?>
<runtimeconfig>
  <dataSources>

    <!-- Oracle defined by old approach - driver class - ONE Runtime Server classloader will be used -->
    <dataSource password="crypted:AES:FX1xWJBTX63gNzB3UFdkCPKvapujpE4TcM2TSdcSftg="
                name="Oracle"
                driverClass="oracle.jdbc.driver.OracleDriver"
                user="test"
                url="jdbc:oracle:thin:@dbase.ataccama.com:1521/ora12c"/>

    <!-- Oracle defined by new approach - driver name - separate classloader will be used -->
    <dataSource password="crypted:AES:FX1xWJBTX63gNzB3UFdkCPKvapujpE4TcM2TSdcSftg="
                name="Oracle"
                driverName="Oracle"
                user="test"
                url="jdbc:oracle:thin:@dbase.ataccama.com:1521/ora12c"/>

    <!-- Hive Knox defined by new approach - driver name - separate classloader will be used -->
    <dataSource password="crypted:AES:RGc9i5omV0SZeSaired+OlVLu5XOl8n9AHxZ9Hj"
                name="Apache Hive Knox"
                driverName="Apache Hive Knox"
                user="sam"
        url="jdbc:hive2://hadr.ataccama.com:8443/;ssl=true;transportMode=http;httpPath=gateway/default/hive"/>

  </dataSources>
  <databaseDrivers>

    <!-- Knox driver defined by JARs -->
    <databaseDriver driverClass="org.apache.hive.jdbc.HiveDriver" name="Apache Hive Knox">
      <classpath>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\slf4j-log4j12-1.7.7.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\commons-codec-1.6.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\httpcore-4.4.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\libthrift-0.9.3.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\log4j-1.2.14.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\guava-14.0.1.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\httpclient-4.4.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\hive-service-1.2.1000.2.6.3.0-235.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\commons-lang-2.6.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\slf4j-api-1.7.7.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\hive-jdbc-1.2.1000.2.6.3.0-235.jar"/>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\commons-logging-1.1.3.jar"/>
      </classpath>
    </databaseDriver>

    <!-- Knox driver defined by JAR folder -->
    <databaseDriver driverClass="org.apache.hive.jdbc.HiveDriver" name="Apache Hive Knox">
      <classpath>
        <classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\*"/>
      </classpath>
    </databaseDriver>

    <!-- Oracle driver defined by JARs -->
    <databaseDriver driverClass="oracle.jdbc.driver.OracleDriver" name="Oracle">
      <classpath>
        <classpathEntry path="C:\Workspaces\oneruntime\lib\jdbc\oracle\ojdbc7-12.1.0.2.0.jar"/>
      </classpath>
    </databaseDriver>

  </databaseDrivers>
</runtimeconfig>

Folder shortcuts

A path to a file can be specified as an absolute path, a relative path, or using folder shortcuts. A folder shortcut is a named path to a file or folder. Folder shortcuts can be created in the Folder Shortcuts node in the File Explorer. For detailed instructions, see Folder Shortcuts.

name - Name of folder shortcut.
value - Folder represented by this shortcut.

Sample folder shortcut definition

<pathVariables>
    <pathVariable name="MyPath" value="D:/ONE/" />
</pathVariables>

For exammple, if the actual path to the file is D:/ONE/MyProject/config.xml, the name could be set to MyPath (arbitrary value) and the value to D:/ONE/. In that case, when using the folder shortcut, the path would look as follows: pathvar://MyPath/MyProject/config.xml.

Runtime components

Runtime components are components that enhance the functionality of ONE Runtime Server. Their parameters are configured in a runtime configuration file. Currently, only one component is supported.

Supported types of runtime components:

File Logger Component - This component is used for monitoring the values of counters in ONE Runtime Server and logging these values to a file.
- class - "com.ataccama.dqc.processor.monitoring.file.FileLoggerComp" (attribute class always has this value when FileLoggerComp is concerned).
- fileName - Name of the file where the values of counters are logged.
- stdout - Boolean flag. If set to true, the values of counters are printed to the console (and to the file). If set to false, the values are logged only to the file.
- loggingIntervalInMins - Counter values interval (in minutes).

Sample runtime component configuration

<runtimeComponents>
    <runtimeComponent class="com.ataccama.dqc.processor.monitoring.file.FileLoggerComp"
        fileName="filename" stdout="true" loggingIntervalInMins="1">
    </runtimeComponent>
</runtimeComponents>

Parallelism level

By default, each step is spawned in a single thread, but the initial number of threads is defined by the parallelismLevel property. Note that only filters can be run in parallel, complex steps ignore this setting and always run in an unmodifiable single step configuration.

Currently the maximum number of threads per step is unlimited, though it is a good practice not to exceed the number of CPUs in the system.

Sample parallelism level configuration

<parallelismLevel>2</parallelismLevel>

Logging

The path to the Logging Configuration configuration file. It can be absolute or relative; the relative path is resolved to the location of this file.

Sample logging configuration reference

<loggingConfig>./etc/logging.xml</loggingConfig>

Workflow resources folder

The location of workflow resources, for example, run result logs. The relative path is resolved to the location of this file.

Sample resources folder definition

<resourcesFolder>./resources</resourcesFolder>

Resources

Configuration of resources allocated for workflows. For details, see Workflow Resource Management.

Sample workflow resources configuration

<resources>
    <resource id="db-oracle" units="4" name="DB Oracle (connections)"/>
    <resource id="memory" units="4096" name="Memory (MB)" />
</resources>

Was this page useful?