Loading Additional Drivers

This article describes how to add external drivers not included with the MDM packages.

Overview

Drivers for some of the third-party data sources and other components are not included in standard packages due to the size or licensing reasons. It is, however, possible for the MDM Server to load those additional drivers from a designated external drivers folder named lib-ext. The individual JARs can be stored in subfolders.

See the following sections for examples of configuration and lists of drivers required by the third-party components.

Data sources

HBase

Drivers
Required JAR
`cdata.jdbc.apachehbase.jar`

The required configuration is as follows:

mdm.runtimeConfig

<dataSource driverClass="cdata.jdbc.apachehbase.ApacheHBaseDriver" url="jdbc:apachehbase:Server=127.0.0.1;Port=8080;" name="HBase-CData" user="<user_name>" password="<password>">
    <properties/>
    <propertiesEncrypted/>
    <propertiesFile/>
    <propertiesFileEncrypted/>
</dataSource>

Databricks

Drivers
Driver group (image-based)
`databricks`

The databricks driver group provides the Apache Calcite libraries required to run MDM data operations on Databricks execution clusters. For cluster connection configuration, see Databricks connector documentation

Amazon Redshift

Drivers
Driver group (image-based)
`redshift`
Required JAR (manual install)
`redshift-jdbc42-<VERSION>.jar`

The required configuration is as follows:

mdm.runtimeConfig

<dataSource driverClass="com.amazon.redshift.jdbc42.Driver" url="jdbc:redshift://<host>:<port>/<database>" name="<name>" user="<user_name>" password="<password>">
    <properties/>
    <propertiesEncrypted/>
    <propertiesFile/>
    <propertiesFileEncrypted/>
</dataSource>

Snowflake

Drivers
Driver group (image-based)
`snowflake`
Required JAR (manual install)
`snowflake-jdbc-<VERSION>.jar`

The required configuration is as follows:

mdm.runtimeConfig

<dataSource driverClass="net.snowflake.client.jdbc.SnowflakeDriver" url="jdbc:snowflake://<account>.snowflakecomputing.com/?db=<database>&amp;schema=<schema>&amp;warehouse=<warehouse>" name="<name>" user="<user_name>" password="<password>">
    <properties/>
    <propertiesEncrypted/>
    <propertiesFile/>
    <propertiesFileEncrypted/>
</dataSource>

Teradata

Drivers
Driver group (image-based)
`teradata`
Required JAR (manual install)
`terajdbc4-<VERSION>.jar`

The required configuration is as follows:

mdm.runtimeConfig

<dataSource driverClass="com.teradata.jdbc.TeraDriver" url="jdbc:teradata://<host>/database=<database>,charset=UTF8" name="<name>" user="<user_name>" password="<password>">
    <properties/>
    <propertiesEncrypted/>
    <propertiesFile/>
    <propertiesFileEncrypted/>
</dataSource>

Message queue providers

Amazon SQS

The MDM Server can work with Amazon SQS queues in two ways:

Native SQS stream source (recommended): The stream consumer reads from the queue through the SQS connection of the MDM Server. The SQS connection and the stream source are part of the standard package, so no driver group is needed.
JMS over SQS: The Amazon SQS Java Messaging Library exposes an SQS queue through the JMS interface, so the JMS stream source and the JMS Writer step can work with SQS. This requires the sqs-jms driver group. The JMS-based SQS connection is deprecated for stream consumption and supported only for a transitional period (see MDM 15.4.0 Upgrade Notes).

Use the native stream source to consume messages. The native integration has no writer step, so to send messages to SQS from a plan, for example from an event handler publisher, use the JMS Writer step over the JMS connection described in the following section. Both connection methods can be used in one MDM Server at the same time.

Native SQS stream source (recommended)

Drivers
Driver group (image-based)
None, included in the standard package

The required configuration is as follows. For the full list of connection properties and the AWS_POD_IDENTITY_TOKEN and Assume Role authentication, see Amazon SQS Connection.

mdm.runtimeConfig

<config class="com.ataccama.nme.sqs.config.SqsContributor">
    <sqsConnections>
        <sqsConnection name="awssqsmdm">
            <properties>
                <property name="url" value="https://sqs.eu-central-1.amazonaws.com/123456789012/awssqsmdmqueue"/>
                <property name="region" value="eu-central-1"/>
                <property name="authType" value="AWS_ACCESS_KEY"/>
                <property name="accessKey" value="crypted:AES:{value}"/>
                <property name="secretKey" value="crypted:AES:{value}"/>
            </properties>
        </sqsConnection>
    </sqsConnections>
</config>

The stream consumer in nme-stream.gen.xml references the connection by its name (see Streaming Interface):

nme-stream.gen.xml

<source class="com.ataccama.nme.engine.stream.SqsStreamSource">
    <connectionName>awssqsmdm</connectionName>
</source>

JMS over SQS

Drivers
Driver group (image-based)
`sqs-jms`
Required JARs (manual install)
`cif.jms.sqs-<VERSION>.jar` `amazon-sqs-java-messaging-lib-<VERSION>.jar` `sqs-<VERSION>.jar` `kafka-clients-<VERSION>.jar` `kafka-avro-serializer-<VERSION>.jar` `kafka-schema-registry-client-<VERSION>.jar` `kafka-schema-serializer-<VERSION>.jar` `aws-msk-iam-auth-<VERSION>.jar` `avro-<VERSION>.jar` `avatica-core-<VERSION>.jar`

The JMS provider itself (cif.jms) is part of the standard package.

The required configuration is as follows:

mdm.serverConfig

<component class="com.ataccama.dqc.jms.JmsProviderComponent">
    <connectionPoolSize>5</connectionPoolSize>
    <jmsResources>
        <resource>awssqsmdm</resource>
    </jmsResources>
</component>

mdm.runtimeConfig

<config class="com.ataccama.dqc.jms.config.JmsContributor">
    <jmsConnections>
        <jmsConnection connectionFactory="QueueConnectionFactory" name="awssqsmdm">
            <contextParams>
                <contextParam name="java.naming.factory.initial" value="com.ataccama.dqc.jms.sqs.SQSInitialContextFactory"/>
                <contextParam name="java.naming.provider.url" value="https://sqs.eu-central-1.amazonaws.com/123456789012/awssqsmdmqueue"/>
                <contextParam name="region" value="eu-central-1"/>
                <contextParam name="queue.awssqsmdmqueue" value="awssqsmdmqueue"/>
                <contextParam name="authType" value="AWS_ACCESS_KEY"/>
                <contextParam name="accessKey" value="crypted:AES:{value}"/>
                <contextParam name="secretKey" value="crypted:AES:{value}"/>
            </contextParams>
        </jmsConnection>
    </jmsConnections>
</config>

Each queue.<destination> context parameter binds a JMS destination name to an SQS queue name. Use the destination name as the input destination of the stream consumer or as the request destination of the JMS Writer step. The queues are looked up in SQS when the JMS provider component initializes, so they must exist before the MDM Server starts.

nme-stream.gen.xml

<source class="com.ataccama.nme.engine.stream.JmsStreamSource">
    <connectionName>awssqsmdm</connectionName>
    <inputDestination>awssqsmdmqueue</inputDestination>
</source>

Apache Kafka

Drivers
Driver group (image-based)
`kafka`

The required configuration is as follows:

mdm.runtimeConfig

<config class="com.ataccama.dqc.streaming.config.KafkaContributor">
    <kafkaConnections>
        <kafkaConnection name="<connection-name>" connectString="<broker-host>:<port>"/>
    </kafkaConnections>
</config>

Kafka stream ingestion is configured in the model’s nme-stream.gen.xml. For Avro messages with a Confluent Schema Registry, add the registry URL to the Kafka connection (schemaRegistry="http://<registry-host>:<port>") and set <inputFormat>Avro</inputFormat> on the stream source. The MDM Server then decodes the messages through the registry itself, so do not set value.deserializer in the stream source Kafka properties.

Rabbit MQ, IBM Websphere MQ, Tibco MQ

Active MQ is included in the standard package.

These drivers are not available via the driver image and must be installed manually into the lib-ext folder.

Drivers
Rabbit MQ
`com.rabbitmq:amqp-client.jar`
IBM WebSphere
`com.ibm.mq.headers.jar` `com.ibm.mq.jar` `com.ibm.mq.jmqi.jar` `com.ibm.mqjms.jar`
Tibco MQ
`jms-2.0.jar` `slf4j-api-1.4.2.jar` `slf4j-simple-1.4.2.jar` `tibcrypt.jar` `tibjms.jar` `tibjmsadmin.jar` `tibjmsapps.jar` `tibjmsufo.jar` `tibrvjms.jar`

The required configuration is as follows:

mdm.serverConfig

<component class="com.ataccama.dqc.jms.JmsProviderComponent">
    <connectionPoolSize>5</connectionPoolSize>
    <jmsResources>
        <resource>{QUEUE NAME}</resource>
    </jmsResources>
</component>

mdm.runtimeConfig

<config class="com.ataccama.dqc.jms.config.JmsContributor">
    <jmsConnections>
        <jmsConnection connectionFactory="QueueConnectionFactory" name="{QUEUE NAME}">
            <contextParams>
                <contextParam name="java.naming.factory.initial"
                              value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
                <contextParam name="java.naming.provider.url" value="tcp://localhost:61616"/>
            </contextParams>
        </jmsConnection>
        ...
    </jmsConnections>
</config>

Other

Azure Data Lake Storage Gen 2

Drivers
Driver group (image-based)
`hadoop`

The required configuration is as follows:

mdm.runtimeConfig

<contributedConfigs>
    <config class="com.ataccama.dqc.azure.config.AzureGen2Contributor">
        <azureGen2Connections>
            <azureGen2Connection
                clientId="clientID"
                authenticateUser="false"
                <!-- authType="AAD_CLIENT_CREDENTIAL" -->
                <!-- authType="AAD_MANAGED_IDENTITY" -->
                clientKey="crypted:AES:encryptedKey"
                containerName="containerName"
                name="AzurGen2"
                storageAccount="storageAccount"
                authTokenEndpoint="https://login.microsoftonline.com/<tokenID>/oauth2/token" <!-- for use with AAD Service Principal-->
                clientId="clientID"/>
        </azureGen2Connections>
    </config>
</contributedConfigs>

ADLS Gen 1 vs. Gen 2

The adls driver group contains only the SDK for legacy Azure Data Lake Storage Gen 1 accounts (azureGen1Connection configuration, adl:// protocol, accounts on *.azuredatalakestore.net). Microsoft retired ADLS Gen 1 in February 2024, so this applies to historical installations only.

For Azure Data Lake Storage Gen 2 (azureGen2Connection configuration, abfss:// protocol, that is, any current Azure storage account with hierarchical namespace enabled), enable only the hadoop driver group; the adls group is not needed. This is because Gen 2 access is implemented on top of the Hadoop ABFS file system.

The required library (cif.azure.storage.jar) is delivered together with hadoop-azure in the hadoop driver package, not in the adls one, so enabling only the adls driver group for a Gen 2 connection does not work.

Apache Avro

Drivers
Driver groups (image-based)
`avro` `hadoop`
Required JARs (manual install)
`avro-<VERSION>.jar` the Hadoop JARs listed in Parquet File Reader, except `parquet-hadoop-bundle-<VERSION>.jar`

The Avro Reader and Avro Writer steps need both driver groups: avro provides the Apache Avro library, and hadoop provides the Hadoop MapReduce API the steps are built on. The full Kafka stack is not required. No additional XML configuration is required.

Parquet File Reader

The hadoop driver group must be enabled — it includes parquet-hadoop-bundle together with all required Hadoop ecosystem dependencies.

For manual installations, the following JARs are required:

Drivers
Required JARs
`parquet-hadoop-bundle-<VERSION>.jar` `hadoop-common-<VERSION>.jar` `hadoop-auth-<VERSION>.jar` `hadoop-hdfs-client-<VERSION>.jar` `hadoop-mapreduce-client-core-<VERSION>.jar` `hadoop-shaded-guava-<VERSION>.jar` `hadoop-shaded-protobuf_3_25-<VERSION>.jar` `commons-configuration2-<VERSION>.jar` `reload4j-<VERSION>.jar` `wildfly-openssl-<VERSION>.jar`
Additional JARs for use with AWS S3
`hadoop-aws-<VERSION>.jar` `s3-transfer-manager-<VERSION>.jar` `analyticsaccelerator-s3-<VERSION>.jar`
Additional JARs for use with Azure Blob Storage
`hadoop-azure-<VERSION>.jar` `hadoop-azure-datalake-<VERSION>.jar`

Salesforce

Drivers are included in the standard package.

The required configuration is as follows:

mdm.runtimeConfig

<config class="com.ataccama.extension.salesforce.dqc.config.SalesforceContributor">
      <salesforceConnections>
           <connection name="SFServer" credentialsType="login" password="<password>" secretToken="<secret_token>" username="<user_name>" />
      </salesforceConnections>
</config>

Was this page useful?