IOMETE Connection

This article describes how to connect to IOMETE.

IOMETE is a self-hosted data lakehouse platform built on Apache Iceberg, Apache Spark, and Kubernetes. It can be deployed on premise, in private or public cloud environments (AWS, Azure, GCP), or in a hybrid configuration.

Create a source

To connect to IOMETE:

Navigate to Data Catalog > Sources.
Select Create.
Provide the following:
- Name: The source name.
- Description: A description of the source.
- Deployment (Optional): Choose the deployment type.
  
  You can add new values if needed. See Lists of Values.
- Stewardship: The source owner and roles. For more information, see Stewardship.

Alternatively, add a connection to an existing data source. See Connect to a Source.

Add a connection

Select Add Connection.
In Select connection type, choose Relational Database > IOMETE.

Provide the following:

Name: A meaningful name for your connection. This is used to indicate the location of catalog items.
Description (Optional): A short description of the connection.
Dpe label (Optional): Assign the processing of a data source to a particular data processing engine (DPE) by entering the DPE label assigned to the engine. For more information, see DPM and DPE Configuration in DPM Admin Console.
Override catalog item name: Select this option if you want metadata imports to overwrite the names of existing catalog items with the latest names from the data source.

If the option is not selected, catalog item names you set in ONE are preserved when metadata is reimported. See Edit catalog item metadata.

JDBC: A JDBC connection string pointing to the IP address or the URL where the data source can be reached.

IOMETE uses Arrow Flight SQL compatible JDBC drivers. The format of your connection string is:

jdbc:arrow-flight-sql://<host>:<port>?cluster=<cluster>&data-plane=<data-plane>

The variables are defined as follows:

<host> is the DNS or IP address of the IOMETE lakehouse endpoint.
<port> is the number of the TCP port to connect to.
<cluster> is the name of the lakehouse cluster.
<data-plane> is the name of the data plane.

Example:

jdbc:arrow-flight-sql://my-lakehouse.iomete.com:443?cluster=cluster-test&data-plane=iomete-system

In the Pushdown processing section, optionally enable pushdown processing to run profiling and DQ evaluation directly in IOMETE. See Pushdown processing.
In Additional settings:
- Select Enable exporting and loading of data if you want to export data from this connection and use it in ONE Data or outside of ONE.
  
  If you want to export data to this source, you also need to configure write credentials. See Connection credentials.
  
  Consider the security and privacy risks of allowing the export of data to other locations.

Pushdown processing

Pushdown processing lets you run profiling and DQ evaluation directly in IOMETE instead of on Ataccama Data Processing Engines (DPEs). To learn how pushdown works and when to use it, see SQL Pushdown Processing and SQL Pushdown for DQ Evaluation.

Prerequisites

Pushdown processing uses an operational database, that is, a dedicated schema in IOMETE, where Ataccama stores processing functions, caches lookup keys, and stores intermediate results. Once you create the schema, point the connection to it.

The operational database is required for DQ evaluation in pushdown.

For pushdown profiling, it’s optional: configure it to speed up profiling of views, SQL catalog items, and sliced sources. Ataccama then profiles a single cached copy instead of reading the source again for each group of columns.

See How pushdown profiling works.

To set up the schema:

In the IOMETE catalog you are connecting to, create a new schema. Use this schema only for Ataccama processing data, not for data storage. Don’t reuse an existing schema.
Make sure the credentials configured on the connection can create, read, write, and delete tables in this schema.

Enable pushdown profiling

Pushdown profiling is enabled on the connection level, either when adding a connection (see Add a connection) or when editing an existing one:

In the Pushdown processing section of the connection settings, enable Profile data using pushdown processing.
Choose whether you want to run term detection when pushdown processing is enabled, or disable term detection.

Term detection jobs can’t run in pushdown. See Term detection in pushdown.
Optionally, if you profile views, SQL catalog items, or sliced sources, configure the operational database. To determine whether you need one and for setup instructions, see Prerequisites. Otherwise, you can skip this step.

Enable pushdown for DQ evaluation

In the Pushdown processing section of the connection settings, enable Profile data using pushdown processing and Evaluate Data Quality using pushdown processing.

You can’t enable pushdown for DQ evaluation without first enabling pushdown profiling.
Provide the details of the schema you created in IOMETE (see Prerequisites):
- Iomete operational catalog: The name of the catalog in IOMETE where the schema is located.
- Iomete operational schema: The name of the schema.
Ataccama accesses the operational schema with the credentials labeled as Operational on the connection. If there are none, the default credentials are used. For details, see Credentials.

Limitations of IOMETE pushdown

When exporting invalid records, the monitoring project Export tab does not provide a direct link to the export table in IOMETE. Instead, the full table name is displayed in the format <catalog_name>.<schema_name>.<table_name>.

This is because the URL depends on the specific IOMETE deployment, so the link cannot be constructed automatically. To view the exported records, locate the table in IOMETE using this identifier.
For general pushdown limitations, see Limitations of pushdown processing.

Add credentials

Different sets of credentials can be used for different tasks. One set of credentials must be set as default for each connection.

To determine whether you need to configure more than a single set of credentials, see Connection credentials.

Select Add Credentials.
Choose an authentication method and proceed with the corresponding step:
- Username and password credentials.
- Secret management service credentials.

Username and password credentials

Select Username and password credentials and provide the following:
- Name (Optional): A name for this set of credentials.
- Description (Optional): A description for this set of credentials.
- Username: The username for authentication with the IOMETE lakehouse.
- Password: The password for the specified user.
If you want to use this set of credentials by default when connecting to the data source, select Set as default.

See also Connection credentials.
Proceed with Test the connection.

Secret management service credentials

Select Username and password credentials and provide the following:
- Name (Optional): A name for this set of credentials.
- Description (Optional): A description for this set of credentials.
- Select a secret management service: Specify which secret management service should be used to retrieve the credential values. For more information, see Secret Management Services.
- Username: Enable Use secret management service and provide the name the username is stored under in your selected secret management service.
- Password: Enable Use secret management service and provide the name the password is stored under in your selected secret management service.
If you want to use this set of credentials by default when connecting to the data source, select Set as default.

See also Connection credentials.
Proceed with Test the connection.

Test the connection

To test and verify whether the data source connection has been correctly configured, select Test Connection.

If the connection is successful, continue with the following step. Otherwise, verify that your configuration is correct and that the data source is running.

Save and publish

Once you have configured your connection, save and publish your changes. If you provided all the required information, the connection is now available for other users in the application.

In case your configuration is missing required fields, you can view a list of detected errors instead. Review your configuration and resolve the issues before continuing.

Next steps

You can now browse and profile assets from your connection.

In Data Catalog > Sources, find and open the source you just configured. Switch to the Connections tab and select Document. Alternatively, opt for Import or Discover documentation flow.

Or, to import or profile only some assets, select Browse on the Connections tab. Choose the assets you want to analyze and then the appropriate profiling option.

Was this page useful?