User Community Service Desk Downloads
If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Monitoring Projects

Use monitoring projects for manual monitoring of your most critical data assets. Select catalog items on which to run anomaly detection and schema checks, and evaluate data quality based on the data quality rules you apply.

Monitoring projects provide reactive monitoring of selected data. If you want to monitor entire data sources or schemas without needing to apply rules manually, consider using the Data Observability feature.

Create monitoring project

If you would like to add additional items to a current project or edit or copy the configuration of a current project, head to Add catalog items or Additional Configuration Options, respectively.

If you want to understand the results of your project better, see Monitoring Project Results, Reports, and Notifications.

To create a new monitoring project:

  1. In the navigation menu, select Data Quality and then Monitoring Projects.

  2. Select Create.

  3. Provide a name for the project.

  4. Describe the aim of the monitoring project (optional).

  5. In Stewardship, select the user group which will be owners of the asset. After you select a group, the list of users assigned to the governance roles within the selected group is displayed.

    For more information, see Stewardship

  6. Select Save.

    400

You can edit the information provided here anytime by selecting the project, using the three dots menu, and selecting Edit.

Add catalog items

After you have created your project, add the required catalog items.

To do this:

  1. Navigate to the Configuration & Results tab.

  2. Select catalog items:

    • If you are working with a new project, select Add catalog items and select the required items.

      300
      600
    • If you want to add additional items to a preconfigured project, in the Items to Monitor section, select Add.

      600
  3. To remove an existing item, use the three dots menu and select Delete.

    600
  4. Once you have selected the necessary items for the project, you can find these listed under Items to Monitor, along with the following information:

    • Structure: Number of rows and columns in the catalog item.

    • Anomalies: Shows if anomalies were detected on any of the catalog item attributes.

    • DQ Rules: Shows the number of DQ rules applied to the catalog item attributes.

    • Overall: Shows an aggregated value for quality based on DQ rules applied to the item attributes. Only results of DQ rules which are contributing to overall quality are included in the aggregation.

    • Results for individual data quality dimensions, listed under their assigned abbreviation.

    • Origin: Catalog item source.

Open any catalog item in the list by selecting its name. This lets you see the attribute view, and add or edit rules on the attributes.

Apply structure checks, DQ rules, and anomaly detection

Once you have selected the required catalog items, you need to select each item individually to start adding checks. Select an item by clicking the item name.

Three types of checks can be applied:

To remove wrongly applied checks:

  • Hover over the applied check in the table and delete using the X icon.

    150

After both adding or removing checks, make sure to publish your changes.

After a monitoring project has been run, an overview of the results for structure checks, anomaly detection, and data quality rules are displayed on the Configuration & Results tab.

600

More detailed results can be found on the monitoring project Report tab. For more information, see Monitoring Project Results, Reports, and Notifications.

Structure checks

Structure checks alert you in the event of any missing columns in the data, or of a change in data type.

They are enabled on an attribute-by-attribute basis by selecting Make Mandatory for the required attributes.

150

Anomaly detection

Anomaly detection is powered by AI and alerts you when potential anomalies that have been detected. For more information about how anomaly detection works in ONE, see Anomaly Detection: Behind the Scenes.

If you do not want to use the default settings and would like more information about how anomaly detection can be tweaked within your project, see Anomaly Detection in Monitoring Projects.

Anomaly detection is enabled on an attribute-by-attribute basis by selecting Enable Detection.

150

DQ rules

Data quality evaluation rules are defined in Data Quality > Rules (for more information, see Detection and DQ Evaluation Rules). You can manually apply them to attributes or use the rule suggestions.

Of all the DQ rules applied, the Overall Quality section takes into account only the results of rules of DQ dimensions which contribute to overall quality. Columns for the results of individual data quality dimensions are named according to the abbreviation provided during configuration (for example, VAL, COM, ACC). For more information about data quality dimensions in monitoring projects, see Additional Configuration Options.

To apply DQ rules, either:

  • Use the plus icon under Applied DQ Rules and select rules from the list provided.

    300
  • Accept rule suggestions.

    1. Click the rule suggestions icon.

      100
    2. A list of suggested rules for the catalog item is shown. Accept or reject suggestions one-by-one, using the relevant icon (tick for accept and cross for reject), or accept or reject in bulk using the dropdown.

      400
Rule suggestions

The application suggests rules for DQ evaluation. Rule suggestions can be seen for the project as a whole, or for individual catalog items.

Rules are suggested in the following situations:

  • When the rule is applied to a term applied to an attribute.

  • When a term applied to an attribute is also applied to the rule input or rule itself.

  • When the rule is applied to the same attribute in another monitoring project.

Rule suggestions can be turned off. See Additional Configuration Options.

Update rule references

If your monitoring project contains mappings to rules that have since been edited, you are notified on the Configuration & Results screen.

600
This notification isn’t visible if you have view-only permissions for monitoring projects.

To update all rules to the latest version, select Update for this project for the monitoring project or Update for this catalog item for a specific catalog item.

To update only selected rules, find the outdated rules in the list of attributes, marked with a lightbulb icon.

400

Select the rule to open the rule information, and then select See the details. You are redirected to the Preview changes screen: review the changes and then select Update Rule.

Application admins can turn on and off these notifications for all users through the core:reference trait. If you change the trait after applying rules in the project, the trait doesn’t apply to those rules.

Make sure you define the required reference strategy before you configure your project. For more information, see Traits.

Run monitoring

If you have made any changes or created a new project, you need to publish them before you can run monitoring.

Select Run monitoring.

400

You are notified once monitoring finishes. Alternatively, schedule the project to be run at certain times by using the three dots menu and selecting Schedule.

If monitoring fails, check out ONE Troubleshooting Guide for possible solutions.

Run monitoring on partitioned tables

When partitioned tables are present in monitoring projects, you can identify them by the following icon:

30

Where partitioned tables are present, there are two options: Run monitoring and Run custom partition monitoring.

300

The Run monitoring option runs the project with default settings, that is, the default selected for the partitioned item, which can be either full analysis or last partition. For more information, see Set partition defaults.

Selecting Run custom partition monitoring opens a new dialog. Select the required partitions from the list provided, and then select Run.

400

Set partition defaults

You can define how you would like to run monitoring on the partitioned table by using the three dots menu and selecting either Last partition analysis as default or Full analysis as default (1).

The chosen default is shown next to the catalog item name (2), and these settings are used on partitioned tables automatically if you select Run monitoring.

600

Was this page useful?