Loading Additional Drivers
This article describes how to add external drivers not included with the MDM packages.
Overview
Drivers for some of the third-party data sources and other components are not included in standard packages due to the size or licensing reasons.
It is, however, possible for the MDM Server to load those additional drivers from a designated external drivers folder named lib-ext.
The individual JARs can be stored in subfolders.
See the following sections for examples of configuration and lists of drivers required by the third-party components.
Data sources
HBase
| Required JAR |
|---|
|
The required configuration is as follows:
<dataSource driverClass="cdata.jdbc.apachehbase.ApacheHBaseDriver" url="jdbc:apachehbase:Server=127.0.0.1;Port=8080;" name="HBase-CData" user="<user_name>" password="<password>">
<properties/>
<propertiesEncrypted/>
<propertiesFile/>
<propertiesFileEncrypted/>
</dataSource>Databricks
| Driver group (image-based) |
|---|
|
The databricks driver group provides the Apache Calcite libraries required to run MDM data operations on Databricks execution clusters.
For cluster connection configuration, see Databricks connector documentation
Amazon Redshift
| Driver group (image-based) |
|---|
|
Required JAR (manual install) |
|
The required configuration is as follows:
<dataSource driverClass="com.amazon.redshift.jdbc42.Driver" url="jdbc:redshift://<host>:<port>/<database>" name="<name>" user="<user_name>" password="<password>">
<properties/>
<propertiesEncrypted/>
<propertiesFile/>
<propertiesFileEncrypted/>
</dataSource>Snowflake
| Driver group (image-based) |
|---|
|
Required JAR (manual install) |
|
The required configuration is as follows:
<dataSource driverClass="net.snowflake.client.jdbc.SnowflakeDriver" url="jdbc:snowflake://<account>.snowflakecomputing.com/?db=<database>&schema=<schema>&warehouse=<warehouse>" name="<name>" user="<user_name>" password="<password>">
<properties/>
<propertiesEncrypted/>
<propertiesFile/>
<propertiesFileEncrypted/>
</dataSource>Teradata
| Driver group (image-based) |
|---|
|
Required JAR (manual install) |
|
The required configuration is as follows:
<dataSource driverClass="com.teradata.jdbc.TeraDriver" url="jdbc:teradata://<host>/database=<database>,charset=UTF8" name="<name>" user="<user_name>" password="<password>">
<properties/>
<propertiesEncrypted/>
<propertiesFile/>
<propertiesFileEncrypted/>
</dataSource>Message queue providers
Amazon SQS
| Driver group (image-based) |
|---|
|
Required JARs (manual install) |
|
The required configuration is as follows:
<component class="com.ataccama.dqc.jms.JmsProviderComponent">
<connectionPoolSize>5</connectionPoolSize>
<jmsResources>
<resource>awssqsmdm</resource>
</jmsResources>
</component><config class="com.ataccama.dqc.jms.config.JmsContributor">
<jmsConnections>
<jmsConnection connectionFactory="QueueConnectionFactory" name="awssqsmdm">
<contextParams>
<contextParam name="java.naming.factory.initial" value="com.ataccama.dqc.jms.sqs.SQSInitialContextFactory"/>
<contextParam name="java.naming.provider.url" value="https://sqs.eu-central-1.amazonaws.com/773634404733/awssqsmdmqueue"/>
<contextParam name="region" value="eu-central-1"/>
<contextParam name="queue.awssqsmdmqueue" value="awssqsmdmqueue"/>
<contextParam name="authType" value="AWS_ACCESS_KEY"/>
<contextParam name="accessKey" value="crypted:AES:{value}"/>
<contextParam name="secretKey" value="crypted:AES:{value}"/>
</contextParams>
</jmsConnection>
</jmsConnections>
</config>Apache Kafka
| Driver group (image-based) |
|---|
|
The required configuration is as follows:
<config class="com.ataccama.dqc.streaming.config.KafkaContributor">
<kafkaConnections>
<kafkaConnection name="<connection-name>" connectString="<broker-host>:<port>"/>
</kafkaConnections>
</config>
Kafka stream ingestion is configured in the model’s nme-stream.gen.xml.
For Avro deserialization with Schema Registry, set value.deserializer to io.confluent.kafka.serializers.KafkaAvroDeserializer and provide schema.registry.url in the stream source Kafka properties.
|
Rabbit MQ, IBM Websphere MQ, Tibco MQ
| Active MQ is included in the standard package. |
These drivers are not available via the driver image and must be installed manually into the lib-ext folder.
|
| Rabbit MQ |
|---|
|
IBM WebSphere |
|
Tibco MQ |
|
The required configuration is as follows:
<component class="com.ataccama.dqc.jms.JmsProviderComponent">
<connectionPoolSize>5</connectionPoolSize>
<jmsResources>
<resource>{QUEUE NAME}</resource>
</jmsResources>
</component><config class="com.ataccama.dqc.jms.config.JmsContributor">
<jmsConnections>
<jmsConnection connectionFactory="QueueConnectionFactory" name="{QUEUE NAME}">
<contextParams>
<contextParam name="java.naming.factory.initial"
value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
<contextParam name="java.naming.provider.url" value="tcp://localhost:61616"/>
</contextParams>
</jmsConnection>
...
</jmsConnections>
</config>Other
Azure Data Lake Storage Gen 2
| Driver groups (image-based) |
|---|
|
The required configuration is as follows:
<contributedConfigs>
<config class="com.ataccama.dqc.azure.config.AzureGen2Contributor">
<azureGen2Connections>
<azureGen2Connection
clientId="clientID"
authenticateUser="false"
<!-- authType="AAD_CLIENT_CREDENTIAL" -->
<!-- authType="AAD_MANAGED_IDENTITY" -->
clientKey="crypted:AES:encryptedKey"
containerName="containerName"
name="AzurGen2"
storageAccount="storageAccount"
authTokenEndpoint="https://login.microsoftonline.com/<tokenID>/oauth2/token" <!-- for use with AAD Service Principal-->
clientId="clientID"/>
</azureGen2Connections>
</config>
</contributedConfigs>ADLS Gen 1 vs. Gen 2
The adls driver group on its own supports only legacy Azure Data Lake Storage Gen 1 accounts (azureGen1Connection configuration, adl:// protocol, accounts on *.azuredatalakestore.net).
Microsoft retired ADLS Gen 1 in February 2024, so this applies to historical installations only.
For Azure Data Lake Storage Gen 2 (azureGen2Connection configuration, abfss:// protocol, that is, any current Azure storage account with hierarchical namespace enabled), you must also enable the hadoop driver group.
This is because Gen 2 access is implemented on top of the Hadoop ABFS file system.
The required library (cif.azure.storage.jar) is delivered together with hadoop-azure in the hadoop driver package, not in the adls one, so enabling only the adls driver group for a Gen 2 connection does not work.
Apache Avro
| Driver group (image-based) |
|---|
|
Required JAR (manual install) |
|
The avro driver group provides standalone Apache Avro file format support for DQC steps that read or write Avro files, without requiring the full Kafka stack.
No additional XML configuration is required.
Parquet File Reader
The hadoop driver group must be enabled — it includes parquet-hadoop-bundle together with all required Hadoop ecosystem dependencies.
For manual installations, the following JARs are required:
| Required JARs |
|---|
|
Additional JARs for use with AWS S3 |
|
Additional JARs for use with Azure Blob Storage |
|
Salesforce
Drivers are included in the standard package.
The required configuration is as follows:
<config class="com.ataccama.extension.salesforce.dqc.config.SalesforceContributor">
<salesforceConnections>
<connection name="SFServer" credentialsType="login" password="<password>" secretToken="<secret_token>" username="<user_name>" />
</salesforceConnections>
</config>Google Cloud Storage
| Driver group (image-based) |
|---|
|
The required configuration is as follows:
<contributedConfigs>
<config class="com.ataccama.dqc.google.config.GoogleContributor">
<googleConnections>
<googleConnection bucket="<gcs_bucket>" keyFile="<gcs_credentials_json>" name="GoogleCloudStorage" projectId="<gcs_project_id>"/>
</googleConnections>
</config>
</contributedConfigs>Google Big Query
| Available from 17.1.0-patch7. |
| Driver group (image-based) |
|---|
|
The required configuration is as follows:
<dataSource name="GoogleBigQuery"
driverclass="com.ataccama.shaded.bigquery.com.google.cloud.bigquery.jdbc.BigQueryDriver"
url="jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=<bq_project_name>;">
<properties>
<property name="OAuthType" value="0"/>
<property name="OAuthServiceAcctEmail" value="<bq_account_email>"/>
<property name="OAuthPvtKeyPath" value="<bq_credentials_json>"/>
<property name="EnableSession" value="1"/>
<property name="Timeout" value="3600"/>
</properties>
<propertiesEncrypted/>
<propertiesFile/>
<propertiesFileEncrypted/>
</dataSource>Migrating from the Simba driver
Starting from version 17.1.0, the bigquery driver group ships Google’s own BigQuery JDBC driver (Apache 2.0 license), with its classes repackaged ("shaded") under the com.ataccama.shaded.bigquery package so they cannot collide with MDM Server classes.
The Simba driver cannot be used on 17.1.0 and later: it bundles unshaded gRPC, protobuf, and OpenTelemetry classes that do collide, and its license does not permit repackaging them.
If your plans read BigQuery BYTES columns, adjust them before switching the datasource to the Google driver.
Otherwise, loads either fail or silently store differently formatted identifiers.
What changed
The plan reads every column declared as STRING with ResultSet.getString(), whatever the SQL type of the column is.
For BYTES columns, the text form is chosen by the driver: Simba returns hexadecimal, the Google driver returns Base64.
The bytes are identical, only the string representation differs.
For example, the same 16-byte SAP GUID arrives as 005056935CD81FE19884977C082D7CE0 from Simba and as AFBWk1zYH+GYhJd8CC184A== from the Google driver.
No driver property changes this behavior.
Columns of type STRING, INT64, NUMERIC, DATE, and TIMESTAMP are not affected.
How plans are affected
When a BYTES value read in the plan is sent back to BigQuery through FROM_HEX(…), for example, in a lookup CTE built from a placeholder, the plan fails.
BigQuery rejects the Base64 text:
Failed to decode invalid hexadecimal string due to character at offset 3: AFBWk1zYH+GYhJd8CC184A==When a BYTES value is used as (or inside of) a source ID, an external system ID, or any matching key, data is silently updated.
Delta loads keep running but produce Base64 IDs that never match the hexadecimal IDs already stored in MDM, which creates duplicates.
Update affected plans
In every SQL Select or JDBC Reader step that reads a BigQuery BYTES column into a STRING plan column, convert the value to hexadecimal in the query and keep the original column name as the alias, so that column mappings and downstream steps stay unchanged:
-- Before: Simba delivered hex, the Google driver delivers Base64
SELECT A.partner_guid, ...
-- After: the same hex text with both drivers
SELECT TO_HEX(A.partner_guid) AS partner_guid, ...Existing FROM_HEX(…) expressions that consume these values stay as they are.
Using FROM_BASE64(…) instead is technically possible but not recommended, because it changes the ID format everywhere else in MDM.
To find the affected columns, list the BYTES columns of the datasets the plans read:
SELECT table_name, column_name
FROM `<project>.<dataset>`.INFORMATION_SCHEMA.COLUMNS
WHERE data_type = 'BYTES'
ORDER BY table_name, column_name;Typical BYTES columns are GUIDs and RAW fields from SAP sources (PARTNER_GUID, PARTNERGUID, CONTPGUID, RECORD_GUID) and hashes.
Was this page useful?