Lead your team forward
OCT 24 / 9AM ET Register nowAtaccama 13.5.0 Release Notes
This page contains the release notes for version 13.5.0 of the entire Ataccama ONE Gen2 platform. Release information affecting all modules in the platform can be found directly below; module specific information can be found in the ONE, MDM, and RDM sections.
Release Information
Products |
ONE Data Governance, ONE Data Quality and Governance, ONE RDM, ONE MDM |
---|---|
Release date |
2022-01-06 |
Downloads |
Known Issues
These issues could impact the functionality of your build; make sure to check them before proceeding with the upgrade. |
Module | Issue | To be resolved in |
---|---|---|
RDM |
ONE-27130: After upgrade, RDM workflows are by default disabled and need to be enabled manually. |
13.6.0 |
MDM |
ONE-25239: MDM History Event Handler fails. Workaround: Before migration or upgrade, all events have to be committed and event storage has to be empty. |
13.6.0 |
MDM |
ONE-31283: OnlineCtl commands for workflow (EWF) do not work in MDM Server. Affects versions 13.4 and higher. |
13.9.0 |
MDM |
RunDQCProcess workflow (EWF) task does not work in MDM Server. See the JIRA ticket for a workaround. Affects versions 13.4 and higher. |
n/a |
MDM |
Native services on HttpEndpoint are not secured by default. Affects versions 13.4 and higher. |
13.8.0 |
MDM |
ONE-38906: It is not possible to hide the RDM Webapp link in the MDM Webapp. |
13.9.3, 14.2.0 |
ONE |
ONE-28274: It is not possible to exit the right-hand sidebar when the heading runs off the edge of the screen. This is heavily affecting the bulk term suggestion resolution process. |
13.6.0 |
ONE |
ONE-28280: The Save and publish button works as normal even though there is an error in the source JDBC string. |
13.6.0 |
ONE |
ONE-31494: This version includes changes in DQ storage which improve performance when saving and querying DQ results. However this change affects all APIs querying monitoring project results and means that ONE metadata reader step can no longer be used to access DQ results. Until further changes results will be accessible from plans only via API. |
13.9.0 |
ONE |
ONE-33115: A metadata import or MANTA processing job remains in the RUNNING status and prevents other queued jobs from starting. For more information, see ONE Troubleshooting Guide. |
13.9.0 |
Known Issues Resolved
We report known issues in each release. Below is the list of the issues that have been resolved across the platform for this version.
Module | Issue | Reported in |
---|---|---|
ONE |
ONE-27542: Sometimes, after publishing changes to Catalog Items' attributes and then opening the Catalog Items list, the error screen appears. Refresh the page to load the Catalog Items list. |
13.4.0 |
ONE |
ONE-26970: Jobs submitted via DPM CLI without mapping parameters may end with a failure. |
13.4.0 |
ONE
Secure Communication Between DPE and DPM
All DPE instances need to be registered individually with DPM: during this registration they can announce whether or not they are TLS enabled. TLS is required to ensure secure communication between DPE and DPM in hybrid deployments when firewall-friendly communication cannot be used. Previously, all connections had to use either secured connections or none could.
ONE Desktop
New Driver Parameters Field for Amazon Redshift
We have introduced a new field called Driver Parameters in the Database Connection Details window to provide additional configuration for Amazon Redshift connections.
By default this field will contain the parameter fetchRingBufferSize=1M
.
This is optional but recommended as it modifies the default buffer size of the Redshift driver (2GB) to fit the maximum memory capacity of ONE Desktop (1 GB), so prevents memory-related issues when fetching a large number of records.
ONE Web Application
Change in DQ Storage
This version includes changes in DQ storage which greatly improve performance when saving and querying DQ results. This change affects all APIs querying monitoring project results however, which in turn means that ONE metadata reader step can no longer be used to access DQ results, and that from version 13.5.0 results will be accessible from plans only via API.
User Upgrade Commands
In addition to system upgrades, the Upgrade Center is now also used for applying model changes. This means that, starting from this version, changes made to the metametadata (MMD) model are published using the Upgrade Center instead of an approval workflow. When users create a draft version of an entity in the model, the Upgrade Center generates an appropriate upgrade command that is listed in the Upgrade Center under Pending Changes. Users can also upload custom upgrade commands.
As a preparation step, the upgrade procedure includes creating a database backup before applying changes. In addition, changes are validated before they are published and users are notified of any conflicts. For more information, see System Changes
Audit UI
Audit logs can now be visualized in the web application. Provided that the Audit Module is configured for your ONE instance, go to Organization > Audit to access the audit information. Here you can see a list of all collected audit logs as well as display related operations and affected assets when viewing an entry’s details. For more information, see Audit.
Exporting Audit Logs
Depending on your distribution, the Audit Module may include the Export feature. When the Export feature is configured, audit logs are exported in CSV format to a designated ONE Object Storage bucket following a predefined schedule. For more information, see Audit, section Exporting Audit Logs.
Spark Processing of Relational Databases
In order to process large volumes of data you can use your Spark cluster. This will improve how ONE works with relational databases when profiling them and during data quality tasks. Your Spark cluster can connect to an RDBMS database (e.g. Oracle, SQL Server) with multiple JDBC connections at once in order to retrieve the data or use open sourced Spark Connectors for cloud DWHs (e.g. Snowflake, Azure Synapse). In the latter case, the data for processing is quickly moved from the DWH to Databricks cluster using ONE Object Storage.
Filters in DQ Reports
It is now possible to use attribute filters in DQ Reports. If a catalog item has attributes for which the 'Filter by' option has been enabled in the project configuration, the same filters will be available in the DQ Report tab, on the catalog item level.
Configurable retention period for monitoring project results
Both at a global level (power-users only) and at a project level, you can configure how long you would like to store monitoring project results. If both global and project settings are defined, global settings take precedence.
Aggregation Rules
Previously, in order to use aggregate functions in DQ rules it was necessary to use ONE Desktop. Now, aggregate functions are available in the rule logic for DQ rules. You can also use the new Group by section to separate data into multiple groups of rows based on the expressions provided. The primary use case for aggregation rules—uniqueness—has been provided as a business friendly rule, meaning it can be configured even using just basic grouping by attribute (or attributes) and the Condition Builder.
This means there are three possible methods of rule creation:
-
Use of aggregate functions in Rule Logic Advanced Expressions.
-
Grouping of the data (using Group by) followed by use of aggregate functions in Rule Logic Advanced Expressions.
-
Grouping of the data (using Group by) followed by use of hidden aggregate function in the Rule Logic Condition Builder.
MDM
In this version, native services available on HttpEndpoint in both SOAP and XML formats are not secured by default and additional configuration must be used. For more information, see Native Services. |
Actuator and Prometheus Support
MDM Server now exposes standard production-ready endpoints for application monitoring and management:
-
Prometheus endpoint
-
Actuator endpoints, e.g.
/info
and/health
All endpoints can be secured by selected authentication mechanism:
-
basic authentication — the recommended default
-
JWT (Keycloak)
-
internal security
Default settings (defined in MDM Server Application Properties):
-
authentication role
ONE_PLATFORM_MONITORING
(required if security for these endpoints is turned on, name can be configured) -
ports:
-
MDM:8051 → MDM:8057
-
MDM webapp:8050 → MDM webapp:8056
-
-
/info
and/health
endpoints are not public -
on-premise = app as is (secured, dedicated port)
-
cloud = docker (unsecured, dedicated port)
# endpoints for monitoring
management.endpoints.enabled-by-default=false
management.endpoint.health.enabled=true
management.endpoint.health.show-details=always
management.endpoint.health.show-components=always
management.endpoint.info.enabled=true
management.endpoint.prometheus.enabled=true
management.endpoints.web.exposure.include=health,info,prometheus
management.endpoint.health.status.order=down,out-of-service,unknown,up
management.info.git.mode=full
# Timing metrics to all spring endpoints (true by default)
management.metrics.web.server.request.autotime.enabled=false
VLDB Commands Improvement
VLDB persistence is able to run custom SQL during inlist after keys are inserted into INL_S
or INL_L
table and before running JOIN of inlist and data table.
Symbolic names {inlist}
, {data}
and {keyName}
can be used.
See database-hints.adoc.
<vldb-commands>
<inlistBeforeCommand>
<variants>
<commandVariant>
<expectedDataSize greaterThan="15" />
<template>
INSERT INTO mdc_table_assert(id, val) VALUES(1, '{data}-{keyName}-{inlist}')
</template>
</commandVariant>
</variants>
</inlistBeforeCommand>
</vldb-commands>
Upgrade Notes
There are no version specific upgrade notes for 13.5.0, to determine your upgrade path see MDM Upgrade Procedure.
RDM
Upgrade Notes
There are no version specific upgrade notes for 13.5.0, to determine your upgrade path see RDM Upgrade Procedure.
Fixes
ONE
Click here to expand
-
ONE-24555: Now you can process data from your relational databases using your Spark cluster. ONE Spark DPE processing can be used for data profiling and data quality evaluation.
-
ONE-28092: Two new SQL placeholders
{simpleTable}
and{schema}
can now be used in all query patterns when editing the JDBC driver configuration templates for DPE. -
ONE-27476: AI Matching is not stuck in invalid phase during start anymore.
-
ONE-27424: Now you can set TLS security for communication between DPM and each of your DPE instances individually.
-
ONE-28154: SQL migration script produces invalid data for anomaly detection checks.
-
ONE-27692: Insufficient FE properties for upgrade.
-
ONE-27691: Missing locales bucket in Minio.
-
ONE-27690: Updating rule breaks monitoring project layout.
-
ONE-27621: Outdated rules show incorrect DQ value.
-
ONE-27503: Monitoring project filter value cannot contain commas.
-
ONE-27407: Use check ID instead of check footprint in DQ Storage.
-
ONE-27300: Fix rule label 'does not match mask'.
-
ONE-27151: Deleted condition still detected in Rule debug
ONE Desktop
Click here to expand
-
ONE-25147: Now you can set the ring buffer size for Amazon Redshift in Database Connection Details.
ONE Web Application
Click here to expand
-
ONE-24555: Now you can process data from your relational databases using your Spark cluster. ONE Spark DPE processing can be used for data profiling and data quality evaluation.
-
ONE-28263: In the rich text field component in the whole application two problems were fixed:
-
When text is too long, it is not overflowing the field anymore.
-
Pressing Ctrl together with another key does not result in frontend render failure anymore.
-
MDM
Click here to expand
-
ONE-8671: Matching step parallelism issue resolved.
-
ONE-24344: MDM Server now supports platform-standard management endpoints.
-
ONE-25296: VLDB persistence is able to run custom SQL during inlist.
-
ONE-26524: processDelta service now recognizes XSD DateTime format for source_timestamp attribute.
RDM
Click here to expand
-
ONE-26908: Relationships with boolean foreign key attribute now work properly.
-
ONE-26909: UI font color issue resolved.
-
ONE-26910: Missing 'Show More' button added to Admin Console.
-
ONE-27255: 'Environment' email template variable added.
-
ONE-27289: Email template link generation fixed.
-
ONE-27556: Filter persistence issue in MN field lookup resolved.
Was this page useful?