User Community Service Desk Downloads
If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Monitoring Projects

Monitoring projects provide reactive monitoring of selected data. Use monitoring projects for manual monitoring of your most critical data assets. Select catalog items on which to run anomaly detection and schema checks, and evaluate data quality based on the data quality checks you assign.

Create monitoring project

If you would like to add additional items to a current project or edit or copy the configuration of a current project, head to Add catalog items or Additional Configuration Options, respectively.

If you want to understand the results of your project better, see Monitoring Project Results, Reports, and Notifications.

To create a new monitoring project:

  1. In the navigation menu, select Data Quality and then Monitoring Projects.

  2. Select Create.

  3. Provide a name for the project.

  4. Describe the aim of the monitoring project (optional).

  5. Add a Purpose (optional). This assists you when working with multiple monitoring project instances (for example, dev and prod). For more information, see Monitoring project lifecycles in monitoring-projects:import-monitoring-project-configuration.adoc. It is possible to assign a purpose later in the project Overview tab.

  6. Select Save.

    New project

You can edit the information provided here anytime by selecting the project, using the three dots menu, and selecting Edit.

Add catalog items

After you have created your project, add the catalog items you want to monitor.

To do this:

  1. Navigate to the Configuration & Results tab.

  2. Select catalog items:

    • If you are working with a new project, select Add catalog items and select the required items.

      New items
    • If you want to add additional items to a preconfigured project, in the Items to Monitor section, select Add.

      Add item
  3. To remove an existing item, use the three dots menu and select Delete.

    Delete item
  4. Once you have selected the necessary items for the project, you can find these listed under Items to Monitor, along with the following information:

    • Structure: Number of rows and columns in the catalog item.

    • Anomalies: Shows if anomalies were detected on any of the catalog item attributes.

    • DQ Checks: Shows the number of DQ checks assigned to the catalog item attributes.

    • Validity: Shows an aggregated value for validity based on DQ checks of the validity dimension assigned to attributes within that item.

Open any catalog item in the list by selecting its name. This lets you see the attribute view, and add or edit checks on the attributes.

Apply structure checks, DQ checks, and anomaly detection

Once you have selected the required catalog items, you need to select each item individually to start adding checks. Select an item by clicking the item name.

Three types of checks can be applied:

To remove wrongly applied checks:

  • Hover over the applied check in the table and delete using the X icon.

    Remove check

After both adding or removing checks, make sure to publish your changes.

After a monitoring project has been run, an overview of the results for structure checks, anomaly detection, and data quality checks are displayed on the Configuration & Results tab.

Configuration and results

More detailed results can be found on the monitoring project Report tab. For more information, see Monitoring Project Results, Reports, and Notifications.

Structure checks

Structure checks alert you in the event of any missing columns in the data, or of a change in data type.

They are enabled on an attribute-by-attribute basis by selecting Make Mandatory for the required attributes.

Structure checkS

Anomaly detection

Anomaly detection is powered by AI and alerts you when potential anomalies that have been detected. For more information about how anomaly detection works in ONE, see Anomaly Detection: Behind the Scenes.

If you do not want to use the default settings and would like more information about how anomaly detection can be tweaked within your project, see Anomaly Detection in Monitoring Projects.

Anomaly detection is enabled on an attribute-by-attribute basis by selecting Enable Detection.

Enable detection

DQ checks

DQ checks are data quality evaluation rules defined in Data Quality > Rules (for more information, see Detection and DQ Evaluation Rules). You can manually assign DQ checks to attributes or use the rule suggestions.

Of all the DQ checks applied, the Overall Quality section takes into account only the results of rules of DQ dimensions which contribute to overall quality. Columns for the results of individual data quality dimensions are named according to the abbreviation provided during configuration (for example, VAL, COM, ACC). For more information about data quality dimensions in monitoring projects, see Additional Configuration Options.

To assign DQ checks, either:

  • Use the plus icon under Applied DQ Checks and select rules from the list provided.

    Add DQ check
  • Accept rule suggestions.

    1. Click the rule suggestions icon.

      Rule suggestions
    2. A list of rules suggested for the catalog item is shown. Accept or reject suggestions one-by-one, using the relevant icon (tick for accept and cross for reject), or accept or reject in bulk using the dropdown.

      Rule suggestions accept or reject
Rule suggestions

The application suggests rules for DQ checks. Rule suggestions can be seen for the project as a whole, or for individual catalog items.

Rules are suggested in the following situations:

  • When the rule is assigned to a term assigned to an attribute.

  • When a term assigned to an attribute is also assigned to the rule input or rule itself.

  • When the rule is assigned to the same attribute in another monitoring project.

Rule suggestions can be turned off. See Additional Configuration Options.

Currently, we recommend using rules of the DQ dimension Validity only, as results of other DQ dimensions are not reflected in the Overall validity calculations used in monitoring projects.

Run monitoring

If you have made any changes or created a new project, you need to publish them before you can run monitoring.

Select Run monitoring.

Run monitoring

You are notified once monitoring finishes. Alternatively, schedule the project to be run at certain times by using the three dots menu and selecting Schedule.

If monitoring fails, check out ONE Troubleshooting Guide for possible solutions.

Run monitoring on partitioned tables

When partitioned tables are present in monitoring projects, you can identify them by the following icon:

Partition icon

Where partitioned tables are present, there are two options: Run monitoring and Run custom partition monitoring.

Run options

The Run monitoring option runs the project with default settings, that is, the default selected for the partitioned item, which can be either full analysis or last partition. For more information, see Set partition defaults.

Selecting Run custom partition monitoring opens a new modal window. Select the partition you want from the list provided, and then select Run.

Run custom monitoring

Set partition defaults

You can define how you would like to run monitoring on the partitioned table by using the three dots menu and selecting either Last partition analysis as default or Full analysis as default (1).

The chosen default is shown next to the catalog item name (2), and these settings are used on partitioned tables automatically if you select Run monitoring.

Set default partition

Was this page useful?