Additional Configuration Options

DQ dimensions

By default, which DQ dimensions contribute to overall quality matches the global settings, defined in Data Quality > Settings > DQ Dimensions. However, you can override this configuration on the monitoring project level.

To define a custom configuration for DQ dimensions on your monitoring project:

Navigate to the project Overview tab.
In Overall quality contribution, select which dimensions you want to contribute to overall quality results for that project.

Revert to default

If you no longer want a custom configuration and instead want the contributing DQ dimensions configuration to match the global settings, select Revert to default.

Parallel processing

By default, one DQ evaluation job is created for each data source.

Application admins can increase global parallelization, splitting each of these jobs into multiple smaller jobs to improve performance. This can be done both globally or per monitoring project.

If both are configured, project settings take priority over global settings.

Global configuration

Parallel processing is configured globally via the property plugin.monitoring-project.ataccama.one.monitoring-project.dq-processing-parallelism. See MMM Configuration for more information.

Project configuration

To configure parallel processing on a project-by-project basis:

Open your project and using the three dots menu, select Advanced settings.
In Parallel processing, select Split jobs from each source into.
Define the required number of jobs.

How does parallelism work?

Let’s imagine there are 20 catalog items in a project: five from source A and 15 from source B.

When DQ evaluation starts as a part of the project run, we first group catalog items by source, with one job being created for each group. So, in our example, by default, there are 2 groups, meaning two jobs.

Group A - 5 catalog items.
Group B - 15 catalog items.

If you changed the parallelism to three, each of these groups is further split into three. That means the final jobs look like this:

A1 - 2 catalog items.
A2 - 2 catalog items.
A3 - 1 catalog items.
B1 - 5 catalog items.
B2 - 5 catalog items.
B3 - 5 catalog items.

The higher the dq-processing-parallelism value is, the smaller the jobs are and the quicker the processing time.

Filter by attribute

It is possible to use attributes as filters on the Report tab when you are viewing results at the catalog item level. To do this:

In the required project, select the Configuration & Results tab.
Select a catalog item.
For the required attribute either:
- Select the Filter by option to enable the filter.
- Use the dropdown in the Filter by column if you want to add a custom name for the filter:
  1. Select Use attribute as a filter for DQ results to enable filtering with the selected attribute.
  2. Specify a name for the filter. If you do not specify the filter name, the filter name is the same as the attribute.
Publish the changes to the project.

Attribute filters are not visible on the screen from which you enable them. If filters are enabled, you can use them on the Report tab when looking at results for the relevant catalog items.

Filter complexity limits

When using attributes for filtering, be aware of the following limitations to ensure your monitoring project runs successfully.

Number of attribute values

Attributes with a high number of values increase filter complexity and increase the risk of monitoring project failing due to processing limits.

Depending on the number of distinct values in a given attribute:

<100 values: The attribute can be safely used as a filter.
100-20 000 values: Using such attribute increases risk of monitoring project failing.
>20 000 values: Using such attribute results in the monitoring project failing.

Number of filter combinations in pushdown processing

For pushdown processing, the maximum total number of filter combinations is 10 000. If this limit is exceeded, the job fails.

You can increase this limit by configuring the plugin.snowflake.ataccama.one.dim-comb-limit property in the dpe/etc/application.properties file.

Increasing this limit might result in higher resource consumption on the database side.

Turn off rule suggestions

Power users can turn off rule suggestions in monitoring projects via the property plugin.monitoring-project.ataccama.one.rule-suggestions-enabled. For more information, see MMM Configuration.

Anomaly detection configuration

See Anomaly Detection in Monitoring Projects.

Update rule references

If your monitoring project contains mappings to rules that have since been edited, you are notified on the Configuration & Results screen.

If you have view-only permissions for monitoring projects, this notification is not visible.

To update the rule instances to the latest version, select Update for this project.
- If the update is successful, publish the changes to continue.
- If there are validation errors you are notified and it is necessary to resolve the issues before you can continue working with the monitoring project.
  
  Validation errors can occur, for example, if an input attribute data type is changed, and if an input attribute was added or removed.

Invalid results samples

The invalid results sample provides an opportunity to see a sample of records that are non-compliant with the DQ rules applied.

To change the configuration of invalid results samples:

Navigate to the Configuration & Results tab.
Using the three dots menu in the Data Quality section, select Configuration of invalid results samples.
1. In Configuration of Invalid Results Samples, do the following:
  1. Select Prepare invalid samples.
  2. Set the size of the sample per catalog item by entering the required number.
2. Select Save to apply the changes.

If invalid results samples are disabled in a monitoring project, they are not generated at all (that is, not saved).

Where are invalid samples saved?

If invalid samples are enabled, they are stored in the drillthrough bucket in ONE Object Storage. For more information, see ONE Object Storage Configuration.

You can define how long the samples should be stored for on a project basis or globally. Samples are deleted from the storage accordingly. For more information, see Retention Settings.

Was this page useful?