DQ&C 17.2.0 Upgrade Notes

While the index is being updated:

The duration depends on the version you upgrade from and the number of term suggestions.

To reduce the duration of the update:

To check whether the update is still running, use the ataccama_one_mmm_catalogsearch_reindex_pending metric on the MMM Prometheus endpoint (/actuator/prometheus). It shows how many search index entries are still waiting to be updated. It rises shortly after MMM starts and falls to 0 when the update is done. Right after MMM starts, the metric can still show 0 until the update is queued.

If the pending metric (see Check the update progress) stays at 0 for a few minutes, and the switch is still unavailable after you reload the page, the update did not complete.

To find the cause, check the MMM log from the last time MMM started. If this log is no longer available, restart MMM and check the new log.

Terms are no longer assigned automatically in the following cases:

Terms assigned before the upgrade are kept, even after the catalog item is profiled again. You can remove them manually.

To see how often assignments are skipped, use the ataccama_one_mmm_term_assignment_skipped_no_shared_user_total metric on the MMM Prometheus endpoint (/actuator/prometheus). To find the affected terms and catalog items, search the MMM log for the message ID term-assignment-skipped-no-shared-user. Each entry contains the IDs of the term and the catalog item. If you don’t see the metric, no term assignments have been skipped since MMM last started.

Set ataccama.one.metadata.processing.termAssignmentPermissionCheck.enabled=false in the MMM configuration file mmm-backend/etc/application.properties and restart MMM. See Term detection.

If the dataType property of attributes was added to the search configuration in your environment, reindex the catalogItem configuration after the upgrade. The default search configuration doesn’t index this property, so most environments require no action.

Until you reindex, searching catalog items by data type returns no results and any search filter on the property isn’t displayed.

To reindex:

Reindexing can take hours on large catalogs, and search is limited while it runs. See Reindex search.

The assets and operations queries now require the size argument, which must be between 1 and 10000. A query that omits size fails with a validation error, and a query that requests a larger page is rejected. Previously, a query without size returned all matching records at once, which could exhaust the memory of the Audit module on instances with a large audit history.

Before upgrading, add size to every assets and operations query you run against the Audit API. To retrieve more records than fit in a single page, use pagination. Also restrict queries to a time range and omit totalCount unless you need it. Without these, the query scans the whole audit history.

The root queries assetNames and assetActions no longer exist. They returned every distinct asset name and asset action ever recorded, which required scanning the whole audit database.

If a script calls either query, update it:

The operationNames, operationActions, users, violations, and moduleNames queries aren’t affected.

If any asset that you publish through the API requires review under the Review workflow configuration, the publish mutation, such as termPublish or termBulkPublish, fails with error code PUBLISH_FAILED and reason REVIEW_REQUIRED. No assets are published, including those that don’t require review. Unpublished changes in child assets, such as the attributes of a catalog item, are published together with the parent, so they are checked too.

To find the assets that require review, check reviewRequiredNodes in the error. Request a review for those assets in the web application, and publish the remaining assets in a separate call.

If a bulk publish mutation selects more than 1000 assets, it fails with error code PUBLISH_FAILED and reason SELECTION_TOO_LARGE. This applies however the assets are selected: by a filter without the size argument, by a size over 1000, or by listing more than 1000 asset IDs. Unpublished changes in child assets count toward the limit.

To publish the assets, split them into smaller selections.

If the Workflow service can’t be reached, publishing in the web application and through the API fails with error code PUBLISH_FAILED and reason WORKFLOW_CHECK_UNAVAILABLE, even for assets that don’t require review. Only asset types that are published directly, without drafts, such as sources and connections, are published as before.

By default, publishing is blocked while the Workflow service can’t be reached. To change this, set one of the following properties in the MMM configuration file mmm-backend/etc/application.properties and restart MMM:

Each approval step of a review or access request has its own subtask. To approve or reject a step through the API, update its subtask with the updateSubtask mutation, and set state to DONE and resolution to APPROVED or DENIED in the same call. A call that sets only state or only resolution fails, and a subtask that is already approved or rejected can’t be changed.

If the user who makes the call isn’t an approver for the subtask, the call fails, so make the call with an approver’s account. Approvers are:

The task of a review or access request and its subtasks can’t be deleted, and the task’s state and type can’t be changed.

Before upgrading, update scripts that call <entity>TransferStewardship or <entity>BulkTransferStewardship, such as termTransferStewardship, so that they also publish the change with a publish mutation, such as termPublish. For asset types that are published directly, the transfer call publishes the change. If the publish mutation fails with error code PUBLISH_FAILED and reason REVIEW_REQUIRED, request a review for the asset in the web application.

In earlier versions, the onConflict argument decided what happened when owner group access couldn’t be granted during the transfer. Owner group access is now checked when the change is published, so the <entity>BulkTransferStewardship mutations, such as termBulkTransferStewardship, no longer accept the argument.

A call that passes onConflict fails with a GraphQL validation error, so before upgrading, remove the argument from existing calls. The conflicts field in the mutation result is deprecated and always empty, so remove the field from the selection in existing calls.

If the new owner group can’t be assigned to the asset, or the access level selected for the previous owner group can’t be shared, publishing the change fails and the draft stays. If the asset’s type is published directly, the transfer call fails instead.

To fix the failure, transfer stewardship again with an owner group and access level that the asset allows. You don’t need to discard the draft first.

When replacing str.jsonParse, keep in mind that json.parse accepts only valid JSON syntax and returns null otherwise. str.jsonParse also accepts invalid JSON syntax, such as unquoted field names or single quotes instead of double quotes. If your data contains such strings, correct it before switching to json.parse.

In BigQuery pushdown, str.jsonParse is not supported and rules using it are treated as not supported in pushdown. Use json.parse instead.

For details, see JSON functions.

In self-managed deployments, DPM runs on your side and the limit is granted by your DPM license through the parallelJobLimit field. DPM refuses to start if it doesn’t find a valid license that grants a limit, and licenses issued before 17.2.0 don’t contain this field.

Before you upgrade:

After the upgrade, DPM logs the resolved limit at startup, and the License tab in DPM Admin Console shows the same value. See License.

Keep in mind the following:

For details about the properties, see Concurrent job limit and license.

No action is needed on your side. The limit is set for each environment according to your contract, and jobs beyond it wait in the queue.

If you run hybrid DPEs on your own infrastructure, you no longer need a license file on them once both the platform and the DPE are on 17.2.0 or later. DPEs on earlier versions keep using their license file.