ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
latest (15.4.0) 15.3.0 15.2.0 15.1.0 14.5.x 13.9.x 13.8.x 13.7.x 13.6.x 13.5.0 13.4.x 13.3.x 13.2.x
User Community Service Desk Downloads
If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Monitoring Project Results, Reports, and Notifications

Results

Once you have run your monitoring project, you can see issues or anomalies detected in your data set.

500

Issues

Issues relate to failed DQ checks. From the Configuration & Results screen, you can access invalid samples, that is, a sample of the records which failed data quality checks.

This can be done both at the project level and at the catalog item level: select Show invalid samples.

From the Report tab, it is also possible to see invalid samples from individual dimensions.
400

Detailed results of each rule are available on the Report tab. For more information, see Reports.

What should I do with failed records?

Consider creating a post-processing plan from the catalog item. Using the plan you can apply data transformations and write to a ONE Data table for subsequent data remediation.

For more information, see Post-Processing Plans and Data Remediation with ONE Data accordingly.

Anomalies

Anomalies shown in Anomaly Detection are those detected on the metrics of catalog items and attributes, such as value distribution, standard deviation, numeric statistics,and string statistics. The value of the metric is considered to be an anomaly if it is beyond the usual range.

Anomalies can be confirmed or dismissed from the Profile inspector.

See Anomaly Detection in Monitoring Projects, starting from the section Detected anomalies.

Reports

DQ reports show the DQ results and detected anomalies over time for all monitoring runs. They provide an overview of issues and statistics, meaning it is possible to see the changes in DQ for the whole monitoring project, for catalog items, for specific DQ dimensions, and on specific checks assigned to the attributes. You can also explore any of the identified issues in more detail.

DQ stewards are notified when a new report is available and can see DQ issues (and expand details) as well as anomalies and the corresponding profiling results.

Once monitoring has finished, the DQ report is available on the Report tab. From here, you can see where anomalies have been detected, invalid samples, data quality results (overall and for individual dimensions), and individual rule results.

500

If you selected attribute filters in your monitoring project, these are also available in your report when you select the catalog item for which you enabled the filters.

100

If you are particularly interested in an individual rule, you can create a section in the report for that rule.

To do this:

  1. For the required catalog item, select the three dots menu and then Create section.

    200

  2. Specify a name for the section and select from the list of the rules available for the item.

    200

  3. Select Create.

If you do not have access to the full data set, you see results for the relevant subset.

To see the data quality for all rules used in the monitoring project, expand the rules in the Detailed results list.

400
400
Use the Expand rules option to see details of all rules without having to manually expand each one, or select a dimension to filter results.
200

In the report, you can view aggregated DQ results using the Overall quality tab, or only results for a specific dimension. There is a tab for all dimension types of checks applied in your monitoring project.

The overall quality percentage is always the same or lower than the pass and fail percentage of any of the contributing dimensions. This is a result of how the results are aggregated, that is, a record is considered passed if all applied rules result in passed.
400

Which dimensions are contributing to the overall quality aggregation is indicated by the colors visible on the Overall quality tab. These can be verified by hovering over the Overall quality tab, or by checking the DQ dimension configuration in the monitoring project overview. For more information about DQ dimensions, see Data Quality Dimensions.

400

Results retention

You can select how long you would like to store the results of monitoring projects. This can be for maintenance reasons or for security, for example, in case there is company policy that forbids data to be kept longer.

Retention period can be based on days or runs, that is, you can choose to remove results older than the defined number of days or runs.

  • Per project configuration:

    • Keep last X runs.

    • Remove runs after X days.

There is an additional global configuration via API (available to power users only) that, if configured, determines the retention settings in place of the project configuration.

  • Global configuration:

    • Keep last X runs.

    • Remove runs after X days.

If global settings are configured, you are notified in ONE when configuring project settings.

200

For more information about configuring global settings, see ONE API, section Monitoring projects results global retention.

Data is deleted when:

  • A run is older that project configuration allows.

  • There are more runs than project configuration allows.

  • A power user manually deletes processing in Metadata Management Module (MMM).

  • A run is older than global configuration allows.

  • There are more runs than global configuration allows.

When results are cleared, the following is removed:

  • Processing in MMM.

  • Invalid samples from ONE Object Storage.

  • DQ results from DQ storage.

  • Filter values from DQ storage.

  • Post-processing plan (PPP) exports from ONE Object Storage.

Configure results storage

To configure removal of the results:

  1. In the required monitoring project, select the three dots menu and then Advanced settings.

    500

  2. Select the option Remove results after to enable removal.

    100

  3. Select between days and runs and enter the number as required.

    • If you select runs, the number corresponds to the number of runs that are kept. For example, Remove results after 50 runs means only the results for latest 50 runs are stored.

      This is most suitable if your aim is to reduce the amount of unnecessary data that is retained.

    • If you select days, the number corresponds to the number of days the results are stored for. For example, Remove results after 50 days means all results older than 50 days are removed.

      This is most suitable if data can only be stored for a limited amount of time for security reasons.

  4. Close the sidebar and publish the changes.

You can also configure global autoremoval settings. If the number of days defined in the global settings is reached before the number of days or runs defined in your custom configuration, results are removed according to the global settings.

Export and post-processing plans

Using post-processing plans, you can transform and export monitoring project results to be used in DQIT, ONE Data, or other business intelligence tools.

Post-processing plans and their outputs are found on the Export tab.

To implement a post-processing plan, you need to use ONE Desktop. For more information, see Post-Processing Plans.

Once the post-processing plan has finished, download the output from the Export tab.

100

Notifications

There are two available types of notifications: general and specific.

Both general notifications (monitoring failed, critical error in structure, new results, DQ issue or anomalies) and specific notifications are listed on the monitoring projects Notifications tab. From here, they can be edited, turned on or off, or removed as required.

500

  1. Add general notifications.

  2. Edit or delete general notifications.

  3. Enable or disable specific notifications.

  4. Edit or delete specific notifications.

However, the steps for adding new notifications differ for these two sections. New general notifications can be added directly from the Notifications tab but new specific notifications are added from the Report tab.

General notifications

To set notifications for one of the preselected scenarios when running monitoring:

  1. Navigate to the Notifications tab of the project and under General, select Add Notification.

    Or, click the row to edit an existing notification.

  2. Select the events for which you would like notifications to be triggered.

    200

  3. Select the channels on which you want to receive the notification.

    • In app: Select the users you want to receive the in app notification.

    • Email: Select the users you want to receive the email notification.

    • Slack: Enter the channel name. If ONE is not connected to Slack, you can connect to your workspace in General settings. See Slack Integration.

    • Microsoft Teams: Insert the webhook URL. See MS Teams Integration.

Roles correspond to those defined in your identity and access management tool.

Email notifications are turned off by default.

If you want to receive email notifications, you must first enable the plugin and configure the SMTP server. For more information see MMM Configuration, section SMTP server.

Delete an existing notification by selecting the three dots menu and then Delete.

100

Specific notifications

Specific notifications have much greater granularity than general notifications and can also be assigned to individual users (Persons) as well as Roles.

You can define specific data quality thresholds which determine when these notifications should be triggered, as well as filter conditions. For example, configure a notification that is sent only if the quality is lower than 70% and is from a specific date range.

Notifications configured in this way are delivered to the user as an email. In the email template design, the Overall results section contains the information resulting from General notifications, whereas the DQ results section contains notifications from Specific notifications. The email includes direct links to items in the report.

While specific notifications can be edited and deleted directly from the Notifications tab, to add new specific notifications, you need to go to the Report tab.

To add a new notification:

  1. Go to the Report tab.

  2. Select the instance or rule for which you’d like to add the notification. Use the three dots menu and select Add Notification.

    200
    400

    • If notifications are already configured for that instance, you see Open Notifications instead.

      200

  3. After selecting Add or Open notifications, a sidebar opens:

    1. If no notifications exist, you can now create one.

      200

    2. If notifications already exist on that level, you are shown an overview instead. From here, you can add a new notification (select Add Notification) or edit an existing notification (select the three dots menu and then Edit).

      200

  4. When creating or editing notifications, provide the following information:

    1. A custom name for the notification. If no name is specified, the default format is applied.

    2. The data quality in percentage (%) and whether the result should be lower or higher than this number in order for the notification to be triggered.

      200

  5. Filter conditions. If filters are available on the item (that is, if Filter by has been enabled for one or multiple attributes on the catalog item during monitoring project configuration) you can define filter conditions.

    Use the dropdown for the required attribute filters and select the values which apply.

    400
    Filters displayed in notification settings reflect the settings that existed the last time the monitoring project was run. If changes have been made since, you need to re-run the project in order to see the currently configured filters.

  6. Recipients. Select Add recipient and select the Persons, or Roles who should receive the notification.

    400

  7. Select Save and publish.

Was this page useful?