Cloud Portal ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
latest (16.3.0) 16.2.0 16.1.0 16.0.0 15.4.0 15.3.0 15.2.0 15.1.0 14.5.x 13.9.x 13.8.x 13.7.x 13.6.x 13.5.0 13.4.x 13.3.x 13.2.x
User Community Service Desk Downloads

Add a JDBC Driver to DPE

To configure a new JDBC driver for Data Processing Engine (DPE), follow this guide. To learn how to configure other data source supported in DPE, refer to Data Sources Configuration.

Choosing a unique driverId

The driver identifier (driverId) needs to be unique to avoid conflicts with officially supported connectors in future releases. We recommend using a prefix such as x-<datasourceName> or custom-<datasourceName> instead of a simple <datasourceName>.

For example, use custom-bigquery or x-bigquery rather than just bigquery.

Driver file dependencies

A custom driver definition should not depend on JDBC drivers or other files included in the DPE distribution, as these may be replaced or updated in later versions. If such files must be used, the custom JDBC configuration should be explicitly reviewed and validated after every upgrade.

Configuration steps

  1. Download and copy the files related to JDBC drivers and its dependencies to appropriate subfolders. Make sure you have a separate folder for each driver.

    In the example of a BigQuery JDBC driver configuration given here, the JDBC driver and related files are put into the /dpe/lib/jdbc/bq folder.

  2. In on-premise deployments, edit the /dpe/etc/application.properties file.

    1. Add a custom JDBC driver configuration to the file using the following properties as a template. Make sure to modify the values accordingly.

      In the example, you can see a BigQuery JDBC driver configuration.

      • Replace the identifier <driverId> in all properties with your chosen identifier.

      • The values for driver-class and \*-pattern properties can be found in the driver’s official documentation. For information about placeholders you can use in SQL query patterns, see SQL query pattern placeholders.

      • If your driver is not stored in the default location (/dpe/lib/jdbc/), include the absolute path to the driver in the property plugin.jdbcdatasource.ataccama.one.driver.<driverId>.driver-class-path. For example, C:\\drivers\\postgresql-*.jar.

      • To provide multiple locations, use a semicolon (;) as a separator, for example:

        plugin.jdbcdatasource.ataccama.one.driver.custom-bq.driver-class-path=GoogleBigQueryJDBC42.jar;bigQueryLibs/*.jar
      		 
      plugin.jdbcdatasource.ataccama.one.driver.<driverId>.name=<datasource_display_name>
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.connection-pattern=<JDBC_connection_string>
# The following two properties include examples for driver-class-path and driver-class properties for BigQuery
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.driver-class-path=bq/GoogleBigQueryJDBC42.jar
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.driver-class=com.simba.googlebigquery.jdbc42.Driver
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.pooling-enabled=true
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.connection-timeout=20000
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.idle-timeout=300000
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.max-lifetime=900000
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.minimum-idle=1
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.maximum-pool-size=5
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.full-select-query-pattern=SELECT {columns} FROM {table}
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.preview-query-pattern=SELECT {columns} FROM {table} LIMIT {previewLimit}
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.dsl-query-preview-query-pattern=SELECT * FROM ({dslQuery}) dslQuery LIMIT {previewLimit}
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.dsl-query-import-metadata-query-pattern=SELECT * FROM ({dslQuery}) dslQuery LIMIT 0
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.row-count-query-pattern=SELECT COUNT(*) FROM {table}
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.sampling-query-pattern=SELECT {columns} FROM {table} WHERE RANDOM() < {percentageLimit} limit {limit}
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.query-quotation-mark=\"
plugin.jdbcdatasource.ataccama.one.driver.<driverId>.disallowed-indexes-table-types=SYNONYM

    2. Save your changes.

  3. Restart your DPE. The new data source type is included as one of the options listed when creating a new data source connection in the web application. For more detailed instructions, see Connect to a Source.

If you want to use post-processing plans to write into the JDBC source you are connecting, it is also necessary to add the source to the global runtime configuration (the JDBC data source should be specified in DPM even if it is already declared in DPE).

SQL query pattern placeholders

In SQL query patterns, you can use placeholders for database objects. The following patterns support specific placeholders:

full-select-query-pattern

Used for loading full data.

Allowed placeholders: {columns}, {table}, {simpleTable}, {schema}, {database}.

preview-query-pattern

Used for loading data preview.

Allowed placeholders: {columns}, {table}, {previewLimit}, {simpleTable}, {schema}, {database}.

row-count-query-pattern

Used for counting the number of rows in a catalog item.

If the database-specific pattern is missing, the pattern SELECT {columns} FROM {table} is used instead.

Allowed placeholders: {table}, {simpleTable}, {schema}, {database}.

sampling-query-pattern

Used for retrieving sample data from a catalog item.

Allowed placeholders: {table}, {columns}, {limit}, {percentageLimit}, {simpleTable}, {schema}, {database}.

Resolving driver ID conflicts during upgrade

If a DPE upgrade introduces an officially supported driver that conflicts with your custom driverId, you have two options.

We recommend updating the driverId of your existing custom driver:

  1. Rename the custom driver configuration in the DPE properties file (that is, in the property names).

    For example, change:

    plugin.jdbcdatasource.ataccama.one.driver.databricks-jdbc.*

    to:

    plugin.jdbcdatasource.ataccama.one.driver.custom-databricks-jdbc.*

  2. Update the database records for existing connectors. Before making any changes, create a database backup and ensure Metadata Management Module (MMM) is not running.

    In the mmm.public.connection_q table, update the executorType_s column for all connections that use the renamed custom driverId.

    The value stored in the database is the capitalized form of the driverId.

  3. Restart MMM.

While it is possible to remove the new driver definition in DPE (including hybrid) and continue using the legacy custom driver configuration, this approach is generally not recommended because:

  • This prevents you from using the new official driver configuration, even for newly created data sources.

  • After every future upgrade, you will need make the same change manually.

Was this page useful?