Set up DQ Configuration
Each catalog item has one or multiple DQ configurations that control how DQ evaluation and profiling operate. These configurations determine what rules are applied, how processing runs, and when it executes.
Where do I find DQ configuration?
DQ configuration is located in the catalog item Data Quality > Configuration tab:
-
From the left navigation menu, select Catalog > Data Catalog.
-
Select a catalog item from the list to open its detail.
-
Switch to the Data Quality > Configuration tab.
Configuration options
You can set up configurations for DQ evaluation and profiling:
-
To configure DQ evaluation:
-
Apply DQ rules to catalog item attributes.
-
Configure DQ evaluation settings.
-
-
To configure profiling: Adjust Profiling settings to control profiling type and parameters.
Most configurations include both DQ evaluation and profiling. However, you can create specialized configurations that run only one of these processes when needed for specific use cases.
To automate these processes, set up a schedule that runs them at defined intervals.
Processing settings
The following settings in the Processing settings widget control how DQ evaluation and profiling are performed.
DQ evaluation settings
You can use the following settings:
-
Collect invalid samples (Recommended): When selected, the system saves examples of records that fail data quality checks. These provide valuable insights into specific data issues.
Profiling settings
Select the appropriate Profiling type for your needs.
Full profiling
Analyzes all data in the catalog item. Typically, you schedule full profiling to regularly monitor catalog items important for your use case.
To set full profiling:
-
Profiling type: Select FULL
-
Detect anomalies: Enable anomaly detection to identify potential inconsistencies in your data.
Using anomaly detection is recommended when you have profiling runs scheduled and want to detect changes in data between the runs, or for easier interpretation of data with seasonality (predictable fluctuations in data, for example, during weekends or holidays).
Sample profiling
Analyzes a subset of data. Useful for large tables.
To set sample profiling:
-
Profiling type: Select SAMPLE
-
Limit number of profiled rows: Define how many records to included in sample profiling. Rows are selected randomly if the database supports this functionality. Otherwise, the set number of rows is selected from the top.
Applied rules
To apply DQ rules to your attributes:
-
Decide which DQ rules you want to apply:
You can do this based on:
-
Profiling results in the catalog item Data Quality > Profiling tab.
For example, if an attribute email frequently appears in an incorrect format, select a rule to validate that the email is in the correct format.
-
Rule suggestions. Select Check for rule suggestions to see, and accept or reject the suggestions.
-
-
Apply the DQ rules to attributes:
-
Go to the Applied Rules widget.
-
For each attribute you want to evaluate, select Add in the Applied Rules column.
-
A dialog opens, showing all available rules.
Which rules are available depends on the data type of the attribute you want to apply them to: you are only shown rules with the matching input data type.
To find your rule quickly, use the filter or check the Suggestion tab that contains rules recommended by the application. 
-
Select Apply Rule next to each rule you want to use.
-
If the rule you need is not there, you can create it. Select Create Rule and follow the instructions in DQ Rules.
-
Scheduling
To automate DQ processes, set up scheduling. Regular, scheduled runs help identify issues early and track results over time.
Determine frequency
Schedule profiling and DQ evaluation according to your needs. The things to consider are, for example:
-
Catalog item update frequency. For items updated daily, run profiling daily; for weekly updates, schedule it weekly. This ensures up-to-date results while saving resources.
-
How often you need to create reports based on the catalog item.
-
How often you need the data for the issue resolution.
Configure the schedule
To configure a schedule:
-
In the Scheduling widget, select Schedule
-
In Repeat, select how often DQ evaluation and profiling (if configured) should run.
-
In At, specify the time when the DQ evaluation and profiling (if configured) should run, and select a time zone.
-
If required, select Queue action for later if the platform is not accessible at the scheduled time.
-
In Valid from, define the date from which the schedule should be followed.
-
Confirm with Schedule.
Pushdown processing
If pushdown processing is enabled on your data source, you see the option on the Processing settings widget.
When to disable pushdown processing?
If DQ configuration contains a complex DQ rule that is not supported in pushdown mode:
-
(Recommended) Attempt rewriting the rule to make it compatible with pushdown processing.
-
Consider disabling pushdown processing for this configuration.
If a DQ configuration contains an unsupported rule and you don’t disable pushdown, the rule is skipped in the DQ evaluation.
Set up multiple DQ configurations
Each catalog item has one primary (default) DQ configuration. In addition, you can create multiple secondary DQ configurations to:
-
Test different rule combinations.
-
Schedule different DQ evaluation frequencies for different use cases (for example, run sample profiling for daily monitoring and full profiling for monthly reports.)
-
Run different types of profiling.
-
Share the configuration with specific privacy settings.
To create a DQ configuration:
-
From the Configuration tab, select the three dots menu and then Add new configuration.
-
Name the configuration and confirm with Create.
Was this page useful?