Ataccama 14.5.0 Release Notes

DQ Firewalls allow you to apply data quality rules to your data using API calls, specifically, the evaluation APIs provided by the new DQF Service. Both GraphQL and REST options are available. This means that you can maintain one central rule library and use Ataccama data quality evaluation rules on your data in real-time in the data stream.

For example, you have an ETL pipeline in Python that processes data, and you want to make sure that it filters out invalid records. After defining the DQ rule in ONE, the pipeline for each record (or batch of records) can call the DQF endpoint, and records will be split up by their validity. For more information, see DQ Firewalls.

A new DQF service was created. This service now handles rule debug in rules and in monitoring projects, as well as allows live data quality validation in ONE Data and the new DQ firewall feature.

Two types of API are provided by the service:

Freshness is an essential factor for good data quality. It indicates how often data is updated at the source. With the new data freshness feature in Data Observability, you can set thresholds for freshness either manually or using AI, and the system checks your data source for updates, alerting you when a table is updated late or not at all.

Where this information is drawn from differs depending on the data source: by default, data freshness checks can work with Snowflake, BigQuery, PostgreSQL, and Oracle. For more information, see Data Observability.

In the dashboard (see Data Observability Dashboards), you can see information such as the number of missing updates, the time between the last update and the last check, the expected time between updates, and the detection type (AI or manual).

We have reworked how you import and explore lineage metadata from MANTA in ONE. Previously, browsing lineage in ONE required a direct connection to MANTA. From this version, you first generate a compatible snapshot of lineage metadata in MANTA and then import and map it to relevant data sources in ONE.

With this update, we have also expanded the support for lineage to additional connection types. You can use lineage with the following data sources:

Furthermore, we have redesigned the catalog item Lineage tab to make the navigation more intuitive and introduced new exploration options. For example, search (or filter) source or target items, open the selected item in MANTA viewer, or view transformation details for attributes that are subject to complex transformation. To find out more about how you can leverage all-new capabilities, see Lineage.

To learn how to prepare lineage metadata in MANTA and import it to ONE, see the following topics: Configure MANTA and [lineage:import-lineage].

You can now create isolated groups or branches, which consist of an isolated group and children. When a group is isolated, members of the group or branch are restricted from sharing assets and assigning stewardship to assets outside of their group or branch.

This results in the following restrictions:

For more information, see Groups and Share Access to Assets.

We have added support for the following Apache Parquet assets: files, tables, and partitioned tables. When assets other than Parquet files are imported, ONE analyzes the asset and then creates a catalog item with the attributes based on the asset.

For more information, see Apache Parquet.

You can now work with Microsoft Excel files in ONE. When you import a Microsoft Excel file, each sheet is loaded as a separate catalog item, while the file itself corresponds to a location within your data source. This way, you can see all catalog items imported from the same Microsoft Excel file.

To find this information, check the catalog item Overview tab, where you can see a list of other catalog items from the single file import. Alternatively, you can see the same information on the catalog item Relationships tab.

You can profile and evaluate catalog items imported from Microsoft Excel files as you would any other catalog item. Previously, Microsoft Excel files could be imported to ONE, but no further processing was possible.

You can import Excel files of up to 50 MB. For more information about file size limitations, see Microsoft Excel Files.

By default, audit logs are now stored in an internal audit database for 90 days in newly installed environments and upgraded environments without existing audit logs. In environments with existing audit logs, the default retention is extended to one year.

You can now change the default retention and cleanup settings for the audit database directly from Audit. You can also set up a schedule for exporting, retention, and cleanup of audit logs in a designated ONE Object Storage (MinIO) bucket.

You can also manage access to the Audit module with newly added identity provider (Keycloak) roles. For more information, see Audit.

You can now connect to multiple Databricks clusters from a single DPE with new plugin.metastoredatasource.ataccama.one.cluster.<clusterId> property patterns you can specify in the dpe/etc/application.properties file. To use multiple Databricks clusters, properties related to each cluster configuration should use the cluster ID associated with that cluster.

For more information, see Metastore Data Source Configuration.

Along with other minor improvements, ONE landing pages now support two additional widgets:

Additionally, access to landing pages is now managed through a new View page access level. Viewers with roles that include this access level can view landing pages and their content but cannot make any changes.

For more information, see The Home Page and Access Levels.

Slack, MS Teams, and email notifications now use new default templates. Custom templates from older versions will still be useable after upgrade, but if you want to make further edits to custom templates, you must convert them to the new template syntax. For more information, see Notification Templates.

The rule conditions is from reference data and is not from reference data are renamed to is from lookup and is not from lookup.

In addition, when creating a new lookup item directly from a catalog item attribute, this option is now called Create new lookup instead of Create new lookup item.

You can now change the language of the application to Italian. For more information, see Application-wide Features.

A Cancel all jobs option has been added to the DPM Admin Console. When selected, all jobs that are not in the state KILLED, FAILURE, or SUCCESS are cancelled and can’t be resumed.

When creating a Run DPM Job workflow task, you have the option to set a DPE label that is then matched against labels assigned to available DPEs.

For more information, see Run DPM Job.

The JDBC drivers for SQL have been updated to version 12.x. With the update, the connection string parameter encrypt is now set to true by default.

If connecting to the MS SQL Server without a valid certificate, set encrypted=false to use an unencrypted connection. Or, you can set trustServerCertificate=true to accept the server’s self-signed certificate.

You can now deduplicate your datasets to easily create managed reference data. Once your reference data is ready, use it in DQ rules to continuously improve the quality of your data.

Start by opening the catalog item from which you want to extract reference data, then expand the three dots menu for a particular attribute and select Create reference table.

Choose your deduplication key (one or a combination of several attributes) and select any additional attributes that help you better describe your data. In addition to these, the table will also contain a frequency attribute, which stores the number of occurrences for each record.

Review and update your data as needed and apply it in DQ rules. For more detailed information about deduplication, see Import Data from a Catalog Item

DQ metadata used in DQ filters is automatically updated each time you run DQ evaluation and each time you edit a record (except for bulk edits). This ensures you’re working with the freshest DQ results available, which in turn facilitates the process of data remediation.

It also allows you to use DQ filters on any ONE Data table, not only those created by importing invalid records from an existing catalog item.

If you edit one of the filtered records, it is validated on the fly and removed from your filtering results if it no longer fits the criteria. Additionally, a warning is displayed, informing you that the overall quality of your table might be outdated.

After upgrade, your existing reports are migrated as follows:

We have introduced a REST API for task management in MDM. Using REST interfaces, you can perform operations on individual or multiple tasks.

The REST API currently supports the following operations:

To learn more, see REST API, section Task Management.

We have introduced new functionality to MDM Web App Admin Center. You can now stop and start MDM Server from the Admin Center, reset the environment, as well as enable automatic synchronization of the lookups from MinIO to locally available folders after every restart.

To use the functionality, you need to have special permissions. For production environments, it is not possible to stop the server and reset the environment.

The new options are available on the Server Dashboard tab under Administration in the Admin Center. To learn more, see MDM Web App Admin Center, section Server Dashboard.

If you cannot access the MDM Web App because of the server failure, the limited version of the Admin Center opens, which allows you to see the cause of the failure and to start the server or reset the environment after you fix the issue.

To determine your upgrade path, see MDM Upgrade Procedure and MDM 14.5.0 Upgrade Notes.

Schedule full validation on all records that will run when you restart the application. Previously, full validation was performed when RDM was first started as well as on every restart. To do this, go to RDM Admin Console and select Schedule revalidation, then restart the application at your convenience.

In addition to this, records are now validated at the application restart only if a previous validation failed on a particular table or the table structure was modified, which helps speed up the startup time.

If your RDM permissions are provided using the configuration model, you can now edit them directly in the web application. Once you make any changes to permissions, this custom configuration is applied instead of the roles defined in the configuration model.

To determine your upgrade path, see RDM Upgrade Procedure and RDM 14.5.0 Upgrade Notes.

We have introduced new DQ Firewall (DQF) and Metadata Extraction (MDE) services. For a summary of the features that these services deliver, see sections DQF Service and MANTA Lineage Integration Reworked in the release notes. For a more in-depth look at the features, refer to DQ Firewalls and Lineage respectively.

Now that rule debug uses the backend of the new DQF service instead of DPE, there are some rules which are not supported in debug anywhere in the application: see details below. DQF limitations do not, however, affect rule evaluation results as this processing is still done by DPE.

Data Stories is now automatically deployed as part of Ataccama ONE PaaS solution, both in Ataccama ONE Data Governance Suite and in Ataccama ONE Data Quality and Governance Suite. Previously, the module was optional. To learn more, see Data Stories.

To access Data Stories, look for the following icon in the left navigation menu. Select it to get redirected to Data Stories.