Downloads

Loading Additional Drivers

This article describes how to add external drivers not included with the MDM packages.

Overview

Drivers for some of the third-party data sources and other components are not included in standard packages due to the size or licensing reasons. It is, however, possible for the MDM Server to load those additional drivers from a designated external drivers folder named lib-ext. The individual JARs can be stored in subfolders.

See the following sections for examples of configuration and lists of drivers required by the third-party components.

Data sources

HBase

Drivers
Required JAR

cdata.jdbc.apachehbase.jar

The required configuration is as follows:

mdm.runtimeConfig
<dataSource driverClass="cdata.jdbc.apachehbase.ApacheHBaseDriver" url="jdbc:apachehbase:Server=127.0.0.1;Port=8080;" name="HBase-CData" user="<user_name>" password="<password>">
    <properties/>
    <propertiesEncrypted/>
    <propertiesFile/>
    <propertiesFileEncrypted/>
</dataSource>

Databricks

Drivers
Driver group (image-based)

databricks

The databricks driver group provides the Apache Calcite libraries required to run MDM data operations on Databricks execution clusters. For cluster connection configuration, see Databricks connector documentation

Amazon Redshift

Drivers
Driver group (image-based)

redshift

Required JAR (manual install)

redshift-jdbc42-<VERSION>.jar

The required configuration is as follows:

mdm.runtimeConfig
<dataSource driverClass="com.amazon.redshift.jdbc42.Driver" url="jdbc:redshift://<host>:<port>/<database>" name="<name>" user="<user_name>" password="<password>">
    <properties/>
    <propertiesEncrypted/>
    <propertiesFile/>
    <propertiesFileEncrypted/>
</dataSource>

Snowflake

Drivers
Driver group (image-based)

snowflake

Required JAR (manual install)

snowflake-jdbc-<VERSION>.jar

The required configuration is as follows:

mdm.runtimeConfig
<dataSource driverClass="net.snowflake.client.jdbc.SnowflakeDriver" url="jdbc:snowflake://<account>.snowflakecomputing.com/?db=<database>&amp;schema=<schema>&amp;warehouse=<warehouse>" name="<name>" user="<user_name>" password="<password>">
    <properties/>
    <propertiesEncrypted/>
    <propertiesFile/>
    <propertiesFileEncrypted/>
</dataSource>

Teradata

Drivers
Driver group (image-based)

teradata

Required JAR (manual install)

terajdbc4-<VERSION>.jar

The required configuration is as follows:

mdm.runtimeConfig
<dataSource driverClass="com.teradata.jdbc.TeraDriver" url="jdbc:teradata://<host>/database=<database>,charset=UTF8" name="<name>" user="<user_name>" password="<password>">
    <properties/>
    <propertiesEncrypted/>
    <propertiesFile/>
    <propertiesFileEncrypted/>
</dataSource>

Message queue providers

Amazon SQS

The MDM Server can work with Amazon SQS queues in two ways:

  • Native SQS stream source (recommended): The stream consumer reads from the queue through the SQS connection of the MDM Server. The SQS connection and the stream source are part of the standard package, so no driver group is needed.

  • JMS over SQS: The Amazon SQS Java Messaging Library exposes an SQS queue through the JMS interface, so the JMS stream source and the JMS Writer step can work with SQS. This requires the sqs-jms driver group. The JMS-based SQS connection is deprecated for stream consumption and supported only for a transitional period (see MDM 15.4.0 Upgrade Notes).

Use the native stream source to consume messages. The native integration has no writer step, so to send messages to SQS from a plan, for example from an event handler publisher, use the JMS Writer step over the JMS connection described in the following section. Both connection methods can be used in one MDM Server at the same time.

Drivers
Driver group (image-based)

None, included in the standard package

The required configuration is as follows. For the full list of connection properties and the AWS_POD_IDENTITY_TOKEN and Assume Role authentication, see Amazon SQS Connection.

mdm.runtimeConfig
<config class="com.ataccama.nme.sqs.config.SqsContributor">
    <sqsConnections>
        <sqsConnection name="awssqsmdm">
            <properties>
                <property name="url" value="https://sqs.eu-central-1.amazonaws.com/123456789012/awssqsmdmqueue"/>
                <property name="region" value="eu-central-1"/>
                <property name="authType" value="AWS_ACCESS_KEY"/>
                <property name="accessKey" value="crypted:AES:{value}"/>
                <property name="secretKey" value="crypted:AES:{value}"/>
            </properties>
        </sqsConnection>
    </sqsConnections>
</config>

The stream consumer in nme-stream.gen.xml references the connection by its name (see Streaming Interface):

nme-stream.gen.xml
<source class="com.ataccama.nme.engine.stream.SqsStreamSource">
    <connectionName>awssqsmdm</connectionName>
</source>

JMS over SQS

Drivers
Driver group (image-based)

sqs-jms

Required JARs (manual install)

  • cif.jms.sqs-<VERSION>.jar

  • amazon-sqs-java-messaging-lib-<VERSION>.jar

  • sqs-<VERSION>.jar

  • kafka-clients-<VERSION>.jar

  • kafka-avro-serializer-<VERSION>.jar

  • kafka-schema-registry-client-<VERSION>.jar

  • kafka-schema-serializer-<VERSION>.jar

  • aws-msk-iam-auth-<VERSION>.jar

  • avro-<VERSION>.jar

  • avatica-core-<VERSION>.jar

The JMS provider itself (cif.jms) is part of the standard package.

The required configuration is as follows:

mdm.serverConfig
<component class="com.ataccama.dqc.jms.JmsProviderComponent">
    <connectionPoolSize>5</connectionPoolSize>
    <jmsResources>
        <resource>awssqsmdm</resource>
    </jmsResources>
</component>
mdm.runtimeConfig
<config class="com.ataccama.dqc.jms.config.JmsContributor">
    <jmsConnections>
        <jmsConnection connectionFactory="QueueConnectionFactory" name="awssqsmdm">
            <contextParams>
                <contextParam name="java.naming.factory.initial" value="com.ataccama.dqc.jms.sqs.SQSInitialContextFactory"/>
                <contextParam name="java.naming.provider.url" value="https://sqs.eu-central-1.amazonaws.com/123456789012/awssqsmdmqueue"/>
                <contextParam name="region" value="eu-central-1"/>
                <contextParam name="queue.awssqsmdmqueue" value="awssqsmdmqueue"/>
                <contextParam name="authType" value="AWS_ACCESS_KEY"/>
                <contextParam name="accessKey" value="crypted:AES:{value}"/>
                <contextParam name="secretKey" value="crypted:AES:{value}"/>
            </contextParams>
        </jmsConnection>
    </jmsConnections>
</config>

Each queue.<destination> context parameter binds a JMS destination name to an SQS queue name. Use the destination name as the input destination of the stream consumer or as the request destination of the JMS Writer step. The queues are looked up in SQS when the JMS provider component initializes, so they must exist before the MDM Server starts.

nme-stream.gen.xml
<source class="com.ataccama.nme.engine.stream.JmsStreamSource">
    <connectionName>awssqsmdm</connectionName>
    <inputDestination>awssqsmdmqueue</inputDestination>
</source>

Apache Kafka

Drivers
Driver group (image-based)

kafka

The required configuration is as follows:

mdm.runtimeConfig
<config class="com.ataccama.dqc.streaming.config.KafkaContributor">
    <kafkaConnections>
        <kafkaConnection name="<connection-name>" connectString="<broker-host>:<port>"/>
    </kafkaConnections>
</config>
Kafka stream ingestion is configured in the model’s nme-stream.gen.xml. For Avro messages with a Confluent Schema Registry, add the registry URL to the Kafka connection (schemaRegistry="http://<registry-host>:<port>") and set <inputFormat>Avro</inputFormat> on the stream source. The MDM Server then decodes the messages through the registry itself, so do not set value.deserializer in the stream source Kafka properties.

Rabbit MQ, IBM Websphere MQ, Tibco MQ

Active MQ is included in the standard package.
These drivers are not available via the driver image and must be installed manually into the lib-ext folder.
Drivers
Rabbit MQ

com.rabbitmq:amqp-client.jar

IBM WebSphere

  • com.ibm.mq.headers.jar

  • com.ibm.mq.jar

  • com.ibm.mq.jmqi.jar

  • com.ibm.mqjms.jar

Tibco MQ

  • jms-2.0.jar

  • slf4j-api-1.4.2.jar

  • slf4j-simple-1.4.2.jar

  • tibcrypt.jar

  • tibjms.jar

  • tibjmsadmin.jar

  • tibjmsapps.jar

  • tibjmsufo.jar

  • tibrvjms.jar

The required configuration is as follows:

mdm.serverConfig
<component class="com.ataccama.dqc.jms.JmsProviderComponent">
    <connectionPoolSize>5</connectionPoolSize>
    <jmsResources>
        <resource>{QUEUE NAME}</resource>
    </jmsResources>
</component>
mdm.runtimeConfig
<config class="com.ataccama.dqc.jms.config.JmsContributor">
    <jmsConnections>
        <jmsConnection connectionFactory="QueueConnectionFactory" name="{QUEUE NAME}">
            <contextParams>
                <contextParam name="java.naming.factory.initial"
                              value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
                <contextParam name="java.naming.provider.url" value="tcp://localhost:61616"/>
            </contextParams>
        </jmsConnection>
        ...
    </jmsConnections>
</config>

Other

Azure Data Lake Storage Gen 2

Drivers
Driver group (image-based)

hadoop

The required configuration is as follows:

mdm.runtimeConfig
<contributedConfigs>
    <config class="com.ataccama.dqc.azure.config.AzureGen2Contributor">
        <azureGen2Connections>
            <azureGen2Connection
                clientId="clientID"
                authenticateUser="false"
                <!-- authType="AAD_CLIENT_CREDENTIAL" -->
                <!-- authType="AAD_MANAGED_IDENTITY" -->
                clientKey="crypted:AES:encryptedKey"
                containerName="containerName"
                name="AzurGen2"
                storageAccount="storageAccount"
                authTokenEndpoint="https://login.microsoftonline.com/<tokenID>/oauth2/token" <!-- for use with AAD Service Principal-->
                clientId="clientID"/>
        </azureGen2Connections>
    </config>
</contributedConfigs>

ADLS Gen 1 vs. Gen 2

The adls driver group contains only the SDK for legacy Azure Data Lake Storage Gen 1 accounts (azureGen1Connection configuration, adl:// protocol, accounts on *.azuredatalakestore.net). Microsoft retired ADLS Gen 1 in February 2024, so this applies to historical installations only.

For Azure Data Lake Storage Gen 2 (azureGen2Connection configuration, abfss:// protocol, that is, any current Azure storage account with hierarchical namespace enabled), enable only the hadoop driver group; the adls group is not needed. This is because Gen 2 access is implemented on top of the Hadoop ABFS file system.

The required library (cif.azure.storage.jar) is delivered together with hadoop-azure in the hadoop driver package, not in the adls one, so enabling only the adls driver group for a Gen 2 connection does not work.

Apache Avro

Drivers
Driver groups (image-based)
  • avro

  • hadoop

Required JARs (manual install)

  • avro-<VERSION>.jar

  • the Hadoop JARs listed in Parquet File Reader, except parquet-hadoop-bundle-<VERSION>.jar

The Avro Reader and Avro Writer steps need both driver groups: avro provides the Apache Avro library, and hadoop provides the Hadoop MapReduce API the steps are built on. The full Kafka stack is not required. No additional XML configuration is required.

Parquet File Reader

The hadoop driver group must be enabled — it includes parquet-hadoop-bundle together with all required Hadoop ecosystem dependencies.

For manual installations, the following JARs are required:

Drivers
Required JARs
  • parquet-hadoop-bundle-<VERSION>.jar

  • hadoop-common-<VERSION>.jar

  • hadoop-auth-<VERSION>.jar

  • hadoop-hdfs-client-<VERSION>.jar

  • hadoop-mapreduce-client-core-<VERSION>.jar

  • hadoop-shaded-guava-<VERSION>.jar

  • hadoop-shaded-protobuf_3_25-<VERSION>.jar

  • commons-configuration2-<VERSION>.jar

  • reload4j-<VERSION>.jar

  • wildfly-openssl-<VERSION>.jar

Additional JARs for use with AWS S3

  • hadoop-aws-<VERSION>.jar

  • s3-transfer-manager-<VERSION>.jar

  • analyticsaccelerator-s3-<VERSION>.jar

Additional JARs for use with Azure Blob Storage

  • hadoop-azure-<VERSION>.jar

  • hadoop-azure-datalake-<VERSION>.jar

Salesforce

Drivers are included in the standard package.

The required configuration is as follows:

mdm.runtimeConfig
<config class="com.ataccama.extension.salesforce.dqc.config.SalesforceContributor">
      <salesforceConnections>
           <connection name="SFServer" credentialsType="login" password="<password>" secretToken="<secret_token>" username="<user_name>" />
      </salesforceConnections>
</config>

Google Cloud Storage

Drivers
Driver group (image-based)

gcs

The required configuration is as follows:

mdm.runtimeConfig
<contributedConfigs>
    <config class="com.ataccama.dqc.google.config.GoogleContributor">
        <googleConnections>
            <googleConnection bucket="<gcs_bucket>" keyFile="<gcs_credentials_json>" name="GoogleCloudStorage" projectId="<gcs_project_id>"/>
        </googleConnections>
    </config>
</contributedConfigs>

Google Big Query

Drivers
Driver group (image-based)

bigquery

The required configuration is as follows:

mdm.runtimeConfig
<dataSource name="GoogleBigQuery"
	driverclass="com.ataccama.shaded.bigquery.com.google.cloud.bigquery.jdbc.BigQueryDriver"
	url="jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=<bq_project_name>;">
	<properties>
		<property name="OAuthType" value="0"/>
		<property name="OAuthServiceAcctEmail" value="<bq_account_email>"/>
		<property name="OAuthPvtKeyPath" value="<bq_credentials_json>"/>
		<property name="EnableSession" value="1"/>
		<property name="Timeout" value="3600"/>
	</properties>
    <propertiesEncrypted/>
    <propertiesFile/>
    <propertiesFileEncrypted/>
</dataSource>

Migrating from the Simba driver

Starting from version 17.1.0, the bigquery driver group ships Google’s own BigQuery JDBC driver (Apache 2.0 license), with its classes repackaged ("shaded") under the com.ataccama.shaded.bigquery package so they cannot collide with MDM Server classes. The Simba driver cannot be used on 17.1.0 and later: it bundles unshaded gRPC, protobuf, and OpenTelemetry classes that do collide, and its license does not permit repackaging them.

If your plans read BigQuery BYTES columns, adjust them before switching the datasource to the Google driver. Otherwise, loads either fail or silently store differently formatted identifiers.

What changed

The plan reads every column declared as STRING with ResultSet.getString(), whatever the SQL type of the column is. For BYTES columns, the text form is chosen by the driver: Simba returns hexadecimal, the Google driver returns Base64. The bytes are identical, only the string representation differs. For example, the same 16-byte SAP GUID arrives as 005056935CD81FE19884977C082D7CE0 from Simba and as AFBWk1zYH+GYhJd8CC184A== from the Google driver.

No driver property changes this behavior. Columns of type STRING, INT64, NUMERIC, DATE, and TIMESTAMP are not affected.

How plans are affected

When a BYTES value read in the plan is sent back to BigQuery through FROM_HEX(…​), for example, in a lookup CTE built from a placeholder, the plan fails. BigQuery rejects the Base64 text:

Failed to decode invalid hexadecimal string due to character at offset 3: AFBWk1zYH+GYhJd8CC184A==

When a BYTES value is used as (or inside of) a source ID, an external system ID, or any matching key, data is silently updated. Delta loads keep running but produce Base64 IDs that never match the hexadecimal IDs already stored in MDM, which creates duplicates.

Update affected plans

In every SQL Select or JDBC Reader step that reads a BigQuery BYTES column into a STRING plan column, convert the value to hexadecimal in the query and keep the original column name as the alias, so that column mappings and downstream steps stay unchanged:

-- Before: Simba delivered hex, the Google driver delivers Base64
SELECT A.partner_guid, ...

-- After: the same hex text with both drivers
SELECT TO_HEX(A.partner_guid) AS partner_guid, ...

Existing FROM_HEX(…​) expressions that consume these values stay as they are. Using FROM_BASE64(…​) instead is technically possible but not recommended, because it changes the ID format everywhere else in MDM.

To find the affected columns, list the BYTES columns of the datasets the plans read:

SELECT table_name, column_name
FROM `<project>.<dataset>`.INFORMATION_SCHEMA.COLUMNS
WHERE data_type = 'BYTES'
ORDER BY table_name, column_name;

Typical BYTES columns are GUIDs and RAW fields from SAP sources (PARTNER_GUID, PARTNERGUID, CONTPGUID, RECORD_GUID) and hashes.

Verify the migration
  1. Run the full load on a non-production environment.

  2. Check that the converted columns are 32-character uppercase hexadecimal strings.

  3. Compare a few source IDs and external IDs with records loaded before you switch the drivers to ensure they match.

Was this page useful?