Runtime Configuration
The runtime configuration file defines resources available to ONE Runtime Server in the batch mode and when the online server is started.
The batch mode is implemented via the runcif utility ([installation folder]/runtime/bin
) where the path to the runtime configuration file is supplied by the -runtimeConfig <filename>
parameter:
runcif.sh -server -serverPort 4040 -runtimeConfig example.runtimeConfig example.plan
The runtime configuration file is referenced in the Server Configuration file, so the resources defined in the runtime configuration are available to the online server too.
The default runtime configuration file is default.runtimeConfig
, located in [installation folder]/runtime/server/etc/
.
Use this file to create your own runtime configuration.
Runtime resources and parameters
The following runtime resources and parameters can be configured:
-
Contributed configurations (remote server connections)
-
Data sources
-
Folder shortcuts
-
Runtime components
-
Initial parallelism level
-
Logging
-
Resources folder for workflows
-
Resources configuration for workflows
The configuration file can be created in a text editor or by exporting the current settings of folder shortcuts, data sources, and configured servers in ONE Desktop. See Export and Import Runtime Configuration. |
Other runtime variables need to be configured manually according to the specifications in this article. The configuration file is an XML file in the following format:
<?xml version='1.0' encoding='utf-8'?>
<runtimeconfig>
<!-- CONTRIBUTED CONFIGS -->
<contributedConfigs>
<config class="com.ataccama.dqc.processor.support.UrlResourceContributor">
<urls>
<url name="SomeConfiguredServer" user="myusername" password="crypted:DESede:p63913D4fMa175vrXECs1nOHdV1SG5sUto5HhuV6Izg=" url="localhost:22"/>
</urls>
</config>
<config class="com.ataccama.dqc.jms.config.JmsContributor">
<jmsConnections>
<jmsConnection connectionFactory="QueueConnectionFactory" name="someJMSbroker">
<contextParams>
<contextParam name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
<contextParam name="java.naming.provider.url" value="tcp://acme.com:61616"/>
</contextParams>
</jmsConnection>
</jmsConnections>
</config>
</contributedConfigs>
<!-- DATA SOURCES -->
<dataSources>
<dataSource name="name"
driverclass="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/myDatabase"
user="root"
password="root">
<properties>
<property name="name" value="value" />
</properties>
</dataSource>
</dataSources>
<!--FOLDER SHORTCUTS -->
<pathVariables>
<pathVariable name="MyPath" value="D:/ONE/" />
</pathVariables>
<!-- RUNTIME COMPONENTS -->
<runtimeComponents>
<runtimeComponent class="com.ataccama.dqc.processor.monitoring.file.FileLoggerComp"
fileName="filename"
stdout="true"
loggingIntervalInMins="1" />
</runtimeComponents>
<!-- PARALLELISM LEVEL -->
<parallelismLevel>2</parallelismLevel>
<!-- LOGGING CONFIGURATION FILE -->
<loggingConfig>./etc/logging.xml</loggingConfig>
<!-- WORKFLOWS RESOURCES FOLDER -->
<resourcesFolder>./resources</resourcesFolder>
<!-- WORKFLOWS RESOURCE CONFIGURATION -->
<resources>
<resource id="demo" units="4" name="Demo resource" />
</resources>
</runtimeconfig>
Contributed configurations
Contributed configurations contain connection definitions that can be referenced from various steps and configuration files. There are several kinds of such configurations, for example, JMS servers or URLs. Contributed configurations can be created in ONE Desktop in the Servers node in the File Explorer. See Connect to a Server.
URL (generic server)
-
name - Name of the URL resource.
-
url - URL address.
-
authConfig - Select the authentication type according to the server security settings. The following options are available:
-
No authentication:
-
class - Select com.ataccama.dqc.processor.bin.config.auth.NoneAuthConfig.
-
-
Basic authentication:
-
class - Select com.ataccama.dqc.processor.bin.config.auth.BasicAuthConfig.
-
user - Username.
-
password - User password. The password can be either in plain or encrypted form; for encryption, you can use the
onlinectl
utility. See Encrypt Passwords.
-
-
OpenID Connect authentication:
-
class - Select com.ataccama.dqc.processor.bin.config.auth.OpenIdConnectAuthConfig.
-
clientId - Client ID.
-
clientSecret - Client secret.
-
tokenEndPointUrl - URL from which the HTTP client obtains an access token. Contact your admin.
-
-
<contributedConfigs>
<config class="com.ataccama.dqc.processor.support.UrlResourceContributor">
<urls>
<url name="SomeConfiguredServer" url="testserver.ataccama.com:8888">
<authConfig password="crypted:AES:tjfcHC9iTJpjZmV2y/uFKaX+WZuAZMRRSAzvVFmYVwRRWG6drUKfeBEudUYoV339" class="com.ataccama.dqc.processor.bin.config.auth.BasicAuthConfig" user="test_user"/>
</url>
</urls>
</config>
</contributedConfigs>
Azure Data Lake Storage Gen 1
-
name - Name of the URL resource.
-
accountFQDN - Fully qualified domain name of the account. Can be found in user settings in Azure. Account FQDN has the
<account_name>.azuredatalakestore.net
format. -
clientId - Client ID.
-
clientKey - Client key.
-
authenticationTokenEndpoint - URL from which the HTTP client obtains an access token. Contact your admin for details.
-
authenticateUser - Enables username and password authentication. Set to
false
(the feature is not yet supported by ADLS).
<contributedConfigs>
<config class="com.ataccama.dqc.azure.config.AzureContributor">
<azureConnections>
<azureConnection clientId="00000000-0000-0000-0000-000000000000" authenticateUser="false" clientKey="crypted:AES:vKYpslbnAZVGc8dKV5XB8eAJ0iDlESofid/IZtlYIJKFMVsWtXuazDeOfyK4GPVjgb3L1Frd0yniWHyGfcFYa5PpmEy+oMju6ADsDNuzkQE=" name="ADLSServer" accountFQDN="myaccount.azuredatalakestore.net" authTokenEndpoint="https://login.microsoftonline.com/11111111-1111-1111-1111-111111111111/oauth2/token"/>
</azureConnections>
</config>
</contributedConfigs>
Azure Data Lake Storage Gen 2
-
name - Name of the URL resource.
-
accountFQDN - Fully qualified domain name of the account. Can be found in user settings in Azure. Account FQDN has the
<account_name>.http://azuredatalakestore.net[azuredatalakestore.net]
format. -
clientId - Client ID.
-
clientKey - Client key.
-
authenticationTokenEndpoint - URL from which the HTTP client obtains an access token. Contact your admin for details.
-
authenticateUser - Enables username and password authentication. Set to
false
to use Azure Active Directory (AAD) authentication methods. -
authType - Use one of the following values to enable AAD authentication:
-
AAD_CLIENT_CREDENTIAL
- If using Azure AD Service Principal with a secret. -
AAD_MANAGED_IDENTITY
- If using Azure AD Managed Identities.
-
<contributedConfigs>
<config class="com.ataccama.dqc.azure.config.AzureGen2Contributor">
<azureGen2Connections>
<azureGen2Connection
clientId="clientID"
authenticateUser="false"
<!-- authType="AAD_CLIENT_CREDENTIAL" -->
<!-- authType="AAD_MANAGED_IDENTITY" -->
clientKey="crypted:AES:encryptedKey"
containerName="containerName"
name="AzurGen2"
storageAccount="storageAccount"
authTokenEndpoint="https://login.microsoftonline.com/<tokenID>/oauth2/token" <!-- for use with AAD Service Principal-->
clientId="clientID"/>
</azureGen2Connections>
</config>
</contributedConfigs>
Google Cloud Storage
-
bucket - The bucket URL associated with your project within the Google Cloud Platform.
-
keyFile - The key file (
.json
or.p12
) location on your local hard drive. -
name - The server connection name.
-
projectId - The project ID associated with your project within the Google Cloud Platform.
<contributedConfigs>
<config class="com.ataccama.dqc.google.config.GoogleContributor">
<googleConnections>
<googleConnection bucket="ataccama_example" keyFile="C:/Users/test-qa-235308-e21d12de6ee9.json" name="GoogleCloudStorage" projectId="test-qa-235308"/>
</googleConnections>
</config>
</contributedConfigs>
JMS
-
name - Name of the URL resource.
-
connectionFactory - Connection factory class name.
-
user - Username.
-
password - User password. The password can be either in plain or encrypted form; for encryption, you can use the
onlinectl
utility. See Encrypt Passwords. -
contextParams (properties) - Optional array of Java properties passed to the connection.
<contributedConfigs>
<config class="com.ataccama.dqc.jms.config.JmsContributor">
<jmsConnections>
<jmsConnection connectionFactory="QueueConnectionFactory" name="someJMSbroker">
<contextParams>
<contextParam name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
<contextParam name="java.naming.provider.url" value="tcp://acme.com:61616"/>
</contextParams>
</jmsConnection>
</jmsConnections>
</config>
</contributedConfigs>
Apache Kafka
-
name - Name of the URL resource.
-
connectionString - Comma-separated list of Kafka broker servers in the
<host>:<port>
format. For example,kafkabroker1.domain.com:9092,kafkabroker2.domain.com:9092
. -
properties (optional) - List of Kafka properties shared by all Kafka steps using the Kafka server connection. For a list of all possible properties, see the official Kafka documentation.
If two properties have the same name, a property defined in a Kafka step overrides the one defined in the Kafka server connection.
To use Apache Kafka with SSL authentication, provide the following properties:
<contributedConfigs>
<config class="com.ataccama.dqc.streaming.config.KafkaContributor">
<kafkaConnections>
<kafkaConnection name="KafkaServer" connectString="kafkabroker1.domain.com:9092,kafkabroker2.domain.com:9092">
<properties>
<property name="security.protocol" value="SSL"/>
<property name="ssl.truststore.location" value="/some-directory/kafka/certs/kafka.client.truststore.jks"/>
<property name="ssl.truststore.password" value="test1234"/>
<property name="ssl.keystore.location" value="/some-directory/kafka.client.keystore.jks"/>
<property name="ssl.keystore.password" value="test1234"/>
<property name="ssl.truststore.password" value="test1234"/>
</properties>
</kafkaConnection>
</kafkaConnections>
</config>
</contributedConfigs>
To use Apache Kafka in AWS Managed Streaming for Kafka (AWS MSK) with IAM authentication, provide the following properties:
<contributedConfigs>
<config class="com.ataccama.dqc.streaming.config.KafkaContributor">
<kafkaConnections>
<kafkaConnection name="AmazonMSK" schemaRegistry="" connectString="<YourConnectString>">
<properties>
<property name="ssl.truststore.location" value="/some-directory/kafka/certs/kafka.client.truststore.jks"/>
<property name="ssl.truststore.password" value="test1234"/>
<property name="security.protocol" value="SASL_SSL"/>
<property name="sasl.mechanism" value="AWS_MSK_IAM"/>
<property name="sasl.jaas.config" value="software.amazon.msk.auth.iam.IAMLoginModule required;"/>
<property name="sasl.client.callback.handler.class" value="software.amazon.msk.auth.iam.IAMClientCallbackHandler"/>
</properties>
</kafkaConnection>
</kafkaConnections>
</config>
</contributedConfigs>
Ataccama ONE Platform
The configuration takes a number of URL addresses:
-
Keycloak token URL:
value="https://<CLIENT>.<ENV>.ataccama.online/auth"
. -
ONE Data Processing (connection to Data Processing Module,
dpm
):-
DPM Admin Console URL:
url="https://dpm-<CLIENT>.<ENV>.ataccama.online"
. -
gRPC host:
host="https://dpm-grpc-<CLIENT>.<ENV>.ataccama.online"
.-
port: The gRPC port where DPM is available.
-
tls: Set to
true
to use TLS.
-
-
-
ONE Metadata Server (connection to Metadata Management Module,
mmm
):-
GraphQL URL:
apiURL="https://<CLIENT>.<ENV>.ataccama.online/graphql"
. -
gRPC host (the URL where ONE is running):
url="https://<CLIENT>.<ENV>.ataccama.online"
.
-
Other properties:
-
name - The server connection name.
-
password - Password for the specified user.
-
user - Username.
-
type - Selected authentication type:
-
OpenID Connect - Authentication using OpenID (recommended).
-
Basic - Authentication with a username and a password.
If you are using OpenID Connect, additional authentication properties are required, as shown in the following example.
-
-
dpm public key - Public access key for DPM.
<contributedConfigs>
<config class="com.ataccama.one2.client.config.One2Contributor">
<one2Connections>
<one2Connection name="ONE">
<auth password="crypted:AES:n2kQSR0s5fqnfVcx7IgGumXYD02Wfx2kSDpR8MFyjAc=" type="OpenID Connect" user="admin">
<properties>
<authProperty name="openid-connect.token.url" value="https://<CLIENT>.<ENV>.ataccama.online/auth"/>
<authProperty name="openid-connect.realm" value="ataccamaone"/>
<authProperty name="openid-connect.client.id" value="one-desktop-public-client"/>
</properties>
</auth>
<dpm publicKey="{"kty":"EC","crv":"P-256","kid":"rsS16kdWaPWHmQysa6kC4lL1xqYWJXfB-Uydd6SQjLc","x":"NQtIiwPXdYyvhXGxtoOBPn9zztHNO8dU8TQUc-S7IlU","y":"-2RFU45NJNSCDiRG6yEEsP8WTPt_6Mgnb6UIujA4H7I","alg":"ES256"}" url="https://dpm-<CLIENT>.<ENV>.ataccama.online">
<grpc port="443" host="dpm-grpc-<CLIENT>.<ENV>.ataccama.online" tls="true"/>
</dpm>
<mmm apiUrl="https://<CLIENT>.<ENV>.ataccama.online/graphql" url="https://<CLIENT>.<ENV>.ataccama.online"/>
</one2Connection>
</one2Connections>
</config>
</contributedConfigs>
Amazon S3
-
clientEncryptKey - A key to encrypt the data on the client side. By default, Java limits the maximum key length for encryption to 128 bits. To remove the key length restriction, download JCE Unlimited Strength policy files to the
<JAVA_HOME>/lib/security
folder. -
authType - Use one of the following values to enable S3 server authentication:
-
AWS_INSTANCE_IAM
- If using IAM roles assigned to EC2 instance. -
AWS_ACCESS_KEY
- If using the access key and secret key. -
AWS_WEB_IDENTITY_TOKEN
- If using service accounts and assigning IAM roles to Kubernetes pods.
-
-
secretKey - Secret access key associated with the S3 account.
-
accessKey - Access key associated with the S3 account.
-
name - The server connection name.
-
sseKey - Select the encryption key from the keys generated by the server. If you leave this field empty, a default service key (generated by the server on a customer by service by region level) is used. The field is available only with SSE-KMS server encryption.
-
sseType - Specifies how the server encrypts the data. The following options are available:
-
None: No server-side data encryption. Default value.
-
SSE-S3: Encryption key is generated and selected by the S3 server.
-
SSE-KMS: Encryption key is selected by a user from the keys generated on the server.
-
-
url - The server URL in the
s3a://<bucket>[/<directory>]
format. -
assumeRole (optional) - Allows to assume a role from a different AWS account (such as AWS Access Key, EC2 Instance Role, or Web Identity Role) instead of using the user’s known identity.
-
enabled - If
true
, specifies that the assume role feature is enabled. -
roleArn - The Amazon Resource Name (ARN) of the IAM role to be assumed.
-
sessionName - The session name, used to identify the connection in AWS logs.
-
externalId - The optional external ID used in the trust relationship between accounts.
-
region - The AWS region where the Security Token Service (STS) should be used.
-
class - The class that handles the configuration for the AWS assume role feature.
-
<contributedConfigs>
<config class="com.ataccama.dqc.s3.config.S3Contributor">
<s3Connections>
<s3Connection
<!-- authType="AWS_ACCESS_KEY" -->
<!-- authType="AWS_INSTANCE_IAM" -->
<!-- authType="AWS_WEB_IDENTITY_TOKEN" -->
secretKey="crypted:AES:PIJhJbDIXJbr7Gahr67XPNevfmi7X7/QnEMlkW51Ob9pSiNyAkFTplVtwofD52ZLn64h235DICo+hLKNvFkABQ=="
accessKey="AKIAJAWAMV3F3O37TPTA"
name="s3" sseKey="SERVER_KEY_ID"
sseType="NONE"
url="https://ataccama.s3.amazonaws.com">
<assumeRole
enabled="true"
roleArn="arn:aws:iam::111111111111:role/cross_account_role"
sessionName="Ataccama_One"
externalId="ataccama_one_external_id"
region="us-east-1"
class="com.ataccama.dqc.aws.auth.runtime.config.AwsAssumeRoleAuthConfig"/>
</s3Connection>
</s3Connections>
</config>
</contributedConfigs>
SMTP
-
host - The SMTP server host.
-
port - The connection port used by the server.
-
user - Username.
-
password - Password for the specified user. The password can be either in plain or encrypted form; for encryption, you can use the
onlinectl
utility. See Encrypt Passwords.
<contributedConfigs>
<config class="com.ataccama.dqc.processor.support.SmtpResourceContributor">
<smtpConnections>
<smtpConnection password="crypted:AES:5rNM3amiDCHjOSo3PRdF4scrNEHMhzeKmMr8TlRjLbFvaoDyY18kR8SpS1TXUm/o" port="25" host="smtpserver.company.com" name="SMTPServer" user="test_user"/>
</smtpConnections>
</config>
</contributedConfigs>
Keycloak deployment connection
Define the configuration for your Keycloak clients.
The settings in the KeycloakDeploymentContributor
should correspond to the Keycloak settings for the client.
The Keycloak client configuration is mapped to the URL pattern in HTTP Dispatcher.
The option to define Keycloak Deployment Connection is available only for the applications running on ONE Runtime Server (such as ONE Runtime Server Admin). Currently, it does not work with web applications. |
-
keycloakConfigs - Define one or multiple Keycloak configurations. We recommend defining a separate configuration for each Keycloak realm.
-
keycloakConfig
-
name - Unique configuration name.
-
clients - Define one or multiple Keycloak clients.
-
client - Keycloak client configuration.
-
id - Unique client ID. Should correspond to the client name in Keycloak.
-
url - Base URL of the Keycloak server. Should be defined for each client (either directly in the client configuration or inherited from the keycloakConfig parent).
-
realm - Name of the Keycloak realm. Should be defined for each client (either directly in the client configuration or inherited from the keycloakConfig parent).
-
secret (optional if using a public client) - Should correspond to the secret in the Keycloak admin console. The password can be either in plain or encrypted form; for encryption, you can use the
onlinectl
utility. See Encrypt Passwords. -
attributes - List of all other Keycloak client configuration attributes.
The url, realm, and attribute properties can be defined either for all clients (as an attribute of the keycloakConfig property) or for individual clients (as an attribute of the client property). In case a property is defined in both places, the client value overrides the value from the parent keycloakConfig.
-
-
-
-
<contributedConfigs>
<config class="com.ataccama.server.keycloak.KeycloakDeploymentContributor">
<keycloakConfigs>
<keycloakConfig name="localKeycloak">
<!-- Define common parameters for all clients.
They can be overridden by client-specific settings.-->
<url>http://localhost:8083/auth</url>
<realm>ataccamaone</realm>
<attributes>
<attribute name="ssl-required" value="external"/>
</attributes>
<clients>
<client id="one-admin-center">
<secret>crypted:AES:DZ+36XQlju1sAAAIS6YUxtbN603Ag+Qxz3mLrNeNnSo=</secret>
<attributes>
<!-- Define client-specific settings.-->
<attribute name="use-resource-role-mappings" value="false"/>
<attribute name="public-client" value="false"/>
<attribute name="bearer-only" value="false"/>
<attribute name="autodetect-bearer-only" value="false"/>
<attribute name="always-refresh-token" value="false"/>
<attribute name="principal-attribute" value="preferred_username"/>
</attributes>
</client>
</clients>
</keycloakConfig>
</keycloakConfigs>
</config>
</contributedConfigs>
Data source
The data source represents the information needed for a data source connection (for example, for connection to a database). Data sources can be created in ONE Desktop in the Databases node in the File Explorer. See Databases.
-
dataSource
-
name - Name of the data source.
-
driverClass - Driver used to connect to the data source.
-
url - URL address of the data source.
-
ata.jdbc.aad.authType - Authentication using Azure AD:
-
AAD_MANAGED_IDENTITY
- If using AAD Managed Identity. -
AAD_SERVICE_PRINCIPAL
- If using AAD Service Principal with a secret.
-
-
ata.jdbc.aad.keyvault.authType - Authentication using Azure AD with Azure Key Vault:
-
AAD_MANAGED_IDENTITY
- If using AAD Managed Identity with Azure Key Vault. -
AAD_SERVICE_PRINCIPAL
- If using AAD Service Principal with a secret kept in Azure Key Vault.
-
-
ata.jdbc.aad.keyvault.vaultUrl - URL of the Azure Key Vault service.
-
ata.jdbc.aad.keyvault.clientId - Your client ID that is used to authenticate using Azure Key Vault.
-
ata.jdbc.aad.keyvault.tenantId - Your client ID for Azure Key Vault.
-
ata.jdbc.aad.keyvault.clientSecret - We recommend storing your secret for the service principal in the
<propertiesEncrypted>
section although it can be stored unencrypted too. -
user - Username.
-
password - User password. The password can be either in plain or encrypted form; for encryption, you can use the
onlinectl
utility. See Encrypt Passwords. -
validationQuery - An SQL
SELECT
command used to validate a database connection prior to using it. -
minSize - The minimum number of established connections that are kept in the connection pool at all times. Default value:
1
.Example: If you start the online server with minSize set to
2
, then two database connections are established automatically after the server is started. -
maxIdleSize - The maximum number of inactive connections that are kept in the connection pool. All inactive connections exceeding maxIdleSize are disposed of automatically. Default value:
10
. -
maxAge - The maximum time (in milliseconds) an inactive connection can be (re)used in the connection pool. Default value:
0
(unlimited).Example: If maxAge is set to
10000
, the particular connection is reused only in the time interval of 10 seconds; if there is another connection request after this interval (for example, you run a ONE plan with JDBC Reader), the mentioned connection is closed and a new connection is established and used instead.
-
-
properties - Properties related to the selected database engine (refer to the respective engine documentation, for example, Oracle Driver documentation or MySQL Driver documentation).
-
name - Name of the property (for example,
user
ordefaultRowPrefetch
in an Oracle database). -
value - Value of the property.
-
-
propertiesEncrypted - Stores the value of the property in the AES encrypted format.
-
ata.jdbc.aad.clientSecret - Your secret for Azure AD Service Principal.
ata.jdbc.aad.keyvault.authType
can be used for Databricks, Snowflake, Microsoft SQL Server, and other connections that can be configured via theproperties
parameter in the runtime configuration.
-
<dataSources>
<dataSource
name="name"
driverclass="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/myDatabase"
user="root"
password="root"
validationQuery="select 1"
minSize="2"
maxIdleSize="5"
maxAge="60000">
<properties>
<property name="connectTimeout" value="0" />
</properties>
</dataSource>
</dataSources>
Connect to Databricks
If you want to browse Databricks JDBC tables in ONE Desktop, you need to add these properties to the runtime configuration and then import them to ONE Desktop.
The authentication methods for connecting ONE Runtime Server to Databricks as a data source using JDBC (without Ataccama Data Processing modules) are described in the comments in the following example:
<dataSources>
<dataSource
name="Databricks"
driverName="Databricks"
url="jdbc:spark://.../;AuthMech=11;Auth_Flow=0;">
<properties>
<!-- Generate personal token for authentication at Databricks. -->
<!-- <property name="ata.jdbc.authType" value="PERSONAL_TOKEN"/> -->
<!-- Use Azure Active Directory Service Principal with a secret.
We recommend storing your secret in <propertiesEncrypted> section. -->
ata.jdbc.authType=AAD_SERVICE_PRINCIPAL
ata.jdbc.aad.tenantUrl="https://login.microsoftonline.com/<TENANT_ID>/oauth2/v2.0/token"
ata.jdbc.aad.clientId="<CLIENT_ID>"
ata.jdbc.aad.resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"
<!-- Use Azure Active Directory Managed Identity for Azure sources.
This method works only for clusters on the same Azure VM as Azure MSI. -->
ata.jdbc.authType="AAD_MANAGED_IDENTITY"
ata.jdbc.tokenPropertyKey="<Auth_AccessToken>"
ata.jdbc.aad.resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"
<!-- Databricks Resource ID at Azure is not cluster-specific and does not change.-->
</properties>
<propertiesEncrypted>
ata.jdbc.aad.clientSecret="<ENCRYPTED_SECRET>"
</propertiesEncrypted>
</dataSource>
</dataSources>
Database drivers
The runtime configuration file is generated as follows.
The root runtimeconfig element contains a databaseDrivers element with the name and definitions of database drivers.
When this approach is used, there is no need to store drivers (.jar
files) in the <ATACCAMA_HOME>/lib
folder, even if you run a workflow with *Run DQC as Process or Run DQC on Cluster.
The classpath of the driver can contain:
-
Concrete
*.jar
files. -
Folders with
*.jar
files. -
Folders.
A separate classloader is created for such a driver.
All required .jar
files have to be included in the classpath entries.
Driver .jar
files should not be specified in cp/lcp
properties.
Runtime configuration file can contain data sources defined using the old approach (attribute driverClass) for backward compatibility.
In that case, .jar
files have to be placed in the <ATACCAMA_HOME>/lib
folder.
Driver .jar
files have to be specified in cp/lcp
properties.
When the runtime configuration file is imported, driver definitions are imported as well. If the same name exists in the ONE Desktop driver definition, it is used, otherwise the new driver definition is created from the runtime configuration file.
If some file is selected (for example, in the File Explorer) and the Import Runtime Configuration wizard is open, the file is filled in in the wizard. On the Database preference screen, you can add and edit database drivers.
<?xml version='1.0' encoding='UTF-8'?>
<runtimeconfig>
<dataSources>
<!-- Oracle defined by old approach - driver class - ONE Runtime Server classloader will be used -->
<dataSource password="crypted:AES:FX1xWJBTX63gNzB3UFdkCPKvapujpE4TcM2TSdcSftg="
name="Oracle"
driverClass="oracle.jdbc.driver.OracleDriver"
user="test"
url="jdbc:oracle:thin:@dbase.ataccama.com:1521/ora12c"/>
<!-- Oracle defined by new approach - driver name - separate classloader will be used -->
<dataSource password="crypted:AES:FX1xWJBTX63gNzB3UFdkCPKvapujpE4TcM2TSdcSftg="
name="Oracle"
driverName="Oracle"
user="test"
url="jdbc:oracle:thin:@dbase.ataccama.com:1521/ora12c"/>
<!-- Hive Knox defined by new approach - driver name - separate classloader will be used -->
<dataSource password="crypted:AES:RGc9i5omV0SZeSaired+OlVLu5XOl8n9AHxZ9Hj"
name="Apache Hive Knox"
driverName="Apache Hive Knox"
user="sam"
url="jdbc:hive2://hadr.ataccama.com:8443/;ssl=true;transportMode=http;httpPath=gateway/default/hive"/>
</dataSources>
<databaseDrivers>
<!-- Knox driver defined by JARs -->
<databaseDriver driverClass="org.apache.hive.jdbc.HiveDriver" name="Apache Hive Knox">
<classpath>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\slf4j-log4j12-1.7.7.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\commons-codec-1.6.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\httpcore-4.4.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\libthrift-0.9.3.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\log4j-1.2.14.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\guava-14.0.1.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\httpclient-4.4.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\hive-service-1.2.1000.2.6.3.0-235.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\commons-lang-2.6.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\slf4j-api-1.7.7.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\hive-jdbc-1.2.1000.2.6.3.0-235.jar"/>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\commons-logging-1.1.3.jar"/>
</classpath>
</databaseDriver>
<!-- Knox driver defined by JAR folder -->
<databaseDriver driverClass="org.apache.hive.jdbc.HiveDriver" name="Apache Hive Knox">
<classpath>
<classpathEntry path="C:\Hive-jdbc\hdp-hive-1.2-thin-knox\*"/>
</classpath>
</databaseDriver>
<!-- Oracle driver defined by JARs -->
<databaseDriver driverClass="oracle.jdbc.driver.OracleDriver" name="Oracle">
<classpath>
<classpathEntry path="C:\Workspaces\oneruntime\lib\jdbc\oracle\ojdbc7-12.1.0.2.0.jar"/>
</classpath>
</databaseDriver>
</databaseDrivers>
</runtimeconfig>
Folder shortcuts
A path to a file can be specified as an absolute path, a relative path, or using folder shortcuts. A folder shortcut is a named path to a file or folder. Folder shortcuts can be created in the Folder Shortcuts node in the File Explorer. For detailed instructions, see Folder Shortcuts.
-
name - Name of folder shortcut.
-
value - Folder represented by this shortcut.
<pathVariables>
<pathVariable name="MyPath" value="D:/ONE/" />
</pathVariables>
For exammple, if the actual path to the file is D:/ONE/MyProject/config.xml
, the name could be set to MyPath
(arbitrary value) and the value to D:/ONE/
.
In that case, when using the folder shortcut, the path would look as follows: pathvar://MyPath/MyProject/config.xml
.
Runtime components
Runtime components are components that enhance the functionality of ONE Runtime Server. Their parameters are configured in a runtime configuration file. Currently, only one component is supported.
Supported types of runtime components:
-
File Logger Component - This component is used for monitoring the values of counters in ONE Runtime Server and logging these values to a file.
-
class -
"com.ataccama.dqc.processor.monitoring.file.FileLoggerComp"
(attributeclass
always has this value when FileLoggerComp is concerned). -
fileName - Name of the file where the values of counters are logged.
-
stdout - Boolean flag. If set to
true
, the values of counters are printed to the console (and to the file). If set tofalse
, the values are logged only to the file. -
loggingIntervalInMins - Counter values interval (in minutes).
-
<runtimeComponents>
<runtimeComponent class="com.ataccama.dqc.processor.monitoring.file.FileLoggerComp"
fileName="filename" stdout="true" loggingIntervalInMins="1">
</runtimeComponent>
</runtimeComponents>
Parallelism level
By default, each step is spawned in a single thread, but the initial number of threads is defined by the parallelismLevel
property.
Note that only filters can be run in parallel, complex steps ignore this setting and always run in an unmodifiable single step configuration.
Currently the maximum number of threads per step is unlimited, though it is a good practice not to exceed the number of CPUs in the system.
<parallelismLevel>2</parallelismLevel>
Logging
The path to the Logging Configuration configuration file. It can be absolute or relative; the relative path is resolved to the location of this file.
<loggingConfig>./etc/logging.xml</loggingConfig>
Workflow resources folder
The location of workflow resources, for example, run result logs. The relative path is resolved to the location of this file.
<resourcesFolder>./resources</resourcesFolder>
Resources
Configuration of resources allocated for workflows. For details, see Workflow Resource Management.
<resources>
<resource id="db-oracle" units="4" name="DB Oracle (connections)"/>
<resource id="memory" units="4096" name="Memory (MB)" />
</resources>
Was this page useful?