Create DQ Rule in the Rule Library

Create rule

To create a rule:

Go to Data Quality > DQ Rules.
Select Create.
On the Create Rule screen, provide the following information:
- Name: The name of the rule.
- Rule type: Use the dropdown to choose a data quality dimension for your rule. See Rule type for details.
- Description (optional): A description of the rule and how to use it.
- Reference of rule definition (optional): A descriptive name for the data source associated with the rule.
- Find similar rules: Use the AI-powered feature to identify existing rules with similar names or descriptions. Review the results to avoid creating duplicates.
Select Save to create a draft of the rule.
Before publishing or submitting an approval request, you need to define the rule logic consisting of at least one input attribute and condition. Continue to Define rule implementation.

Define rule implementation

Once you have created the rule, go to the Implementation tab.

Generate rule with AI

Select this option to generate rule using AI:

Define the rule and select Generate.
Review the rule inputs and logic.
If needed, adjust the rule definition and Re-generate.
Continue to Test rule.

Rule type

In the Rule type dropdown, under DQ Evaluation Rules, choose a data quality dimension for your rule: image::create-dq-rule-rule-library-select-rule-type.png[Select rule type,500]

Validity: Validity rules verify the usability of the data (for example, regarding data format, data content or attribute relations). The default results are Valid and Invalid.
Uniqueness: Use this dimension to verify that there are no duplicate values and only one instance appears in the dataset. The default results are Unique, Not populated, and Not Unique.
Completeness: Use this dimension to verify that the value field is filled. The default results are Complete and Not complete.
Accuracy: Use this dimension to check whether values are accurate and reflect the true values, for example, based on reference data. The default results are Accurate, No reference available, and Not accurate.
Timeliness: Use this dimension to verify whether data is available at the time it is needed. The default results are Timeliness ok, Minor delay, and Major delay.
Custom: If you have created any custom dimensions, they can also be selected here.

DQ dimensions are fully customizable. Validity, Uniqueness, Accuracy, Completeness, and Timeliness are available by default, but their configuration and results can be changed if required.

Inputs

Select rule inputs: Attributes (at least one is required), Parameters (optional), and Variables (optional). Once added, the inputs are available for building rule conditions in the Rule logic section.

How to name inputs

The input name does not affect the implementation.
We recommend using meaningful input names, as that helps keep the rule logic easy to understand for other users too.
If special characters or reserved words are used, they must be enclosed in square brackets, otherwise publishing fails.

How many inputs should you use?

For domain-based validation, you should have only one input.

Multiple inputs are required if you want to check the data from one attribute based on the values from another attribute or when you need to check the combination of values from several attributes. In this case, create as many inputs as there are attributes you will be validating. Rules with multiple inputs are used for data quality validation and monitoring in monitoring projects.

Attributes

Select the attribute data type and specify its name. Select Add attribute if you require additional fields.

(Optional) Select Add term to add terms to the attribute.

Adding terms to input attributes during implementation results in this rule being suggested for other attributes that also have this term.

Scored and hidden attributes

When a rule references multiple input attributes, each attribute has a Scored or Hidden setting that controls which attribute the rule’s result is reported on:

Scored: The rule’s result is reported on this attribute, contributing to its DQ score and invalid samples.
Hidden: The attribute participates in the rule’s logic and can affect the result, but the result is not reported on it. The attribute’s DQ score and invalid samples are unaffected by this rule.

Rule input attribute with the Scored or Hidden dropdown

Set an attribute to Hidden when it’s auxiliary in the rule — a reference value or condition — and the result shouldn’t be reported on it. Without this setting, every attribute the rule touches receives the rule’s result, which can distort the DQ score of an attribute that isn’t really the rule’s target.

Common cases:

Conditional validation: A rule evaluates one attribute only when another meets a condition. For example, "if country is US, then zip_code must be five digits" — set country to Hidden so only zip_code carries the result.
Reference comparison: A rule compares one attribute against a value derived from another. For example, "the total must equal the sum of line_amount`" — set `line_amount to Hidden so only total carries the result.

Leave all attributes as Scored when each is equally part of what’s being evaluated, for example a rule that requires both first_name and last_name to be non-empty.

The Scored or Hidden setting only appears for rules with multiple input attributes.

For cross-table rules, the rule inputs are grouped into a Primary table and a Related table. Attributes in the Related table group are automatically set to Hidden and are not scored: results are reported only on the primary table.

Scored is the default. You can update the setting later when applying or editing the rule on a catalog item, so the same rule can be scored differently depending on context.

Parameters

Parameters are placeholders for values provided by the user. Use a parameter in a condition instead of a constant value. You supply the value when you apply the rule, not when you create it, which is what lets one rule be reused with different values on different attributes.

For example, you can use parameters to set up a DQ rule that checks whether the value of an age attribute falls within a certain interval, which is not known upfront. This rule will use any age interval specified by the user while applying the rule to the age attribute.

Parameter values are case-sensitive.

To add parameters:

Select Add parameter.
Provide a meaningful parameter name.
Provide a data type.

Select Add parameter again for each additional parameter you need.

A parameter definition consists of a name and a data type only, with no field for a value. You cannot apply the rule until you fill in every required parameter. See Set parameter values.

Which data types can be used with parameters

There are two additional data types that can be used with parameters:

List of values: Useful for validating whether the value is or is not from a predefined list. You provide the list of values when you apply the rule.

Used in combination with the is from parameter list or is not from parameter list conditions.
List of masks: Useful for validating whether masks and patterns come from a predefined list or not. This is useful especially for autogenerated rules where the masks are prefilled from profiling (an option preferred to specifying masks manually).

This data type is not supported in the Condition Builder yet, use ONE expression matches([Valid masks], Attribute) instead.

Example: Validate values against a parameter list

In this example, you build a rule that checks whether a value belongs to an allowed list, where the list itself is chosen separately each time the rule is applied.

Because the list lives in a parameter rather than in the rule, one rule covers every attribute that needs this kind of check. The same rule can validate country codes on one attribute and order statuses on another.

To build the rule:

Create a rule and open the Implementation tab.
Set the attribute data type to String and name the attribute, for example VALUE.
Select Add parameter in the Parameters section.
Name the parameter, for example ALLOWED_VALUES, and set the data type to List of values.

The is from parameter list and is not from parameter list conditions do not appear until the rule has a List of values or List of masks parameter, which is why you add the parameter first.
Select VALUE in the Rule logic section.
Set the condition to is from parameter list.

An operand row appears underneath the condition.
Select your parameter in the operand row.

The condition now checks the value against whatever list the parameter holds.
Set the Result to Valid.
In the IF none of the conditions above apply THEN section, set the Result to Invalid.
Test the rule, supplying the list you want to test against in the test panel.

The parameter holds no values on the rule itself, so the test uses the list you enter here.
Publish the rule.
Apply the rule to an attribute.
Enter each allowed value in its own field, selecting the plus sign (+) to add another.

The values are joined by or, so a value passes when it matches any of them.
Set the Rule instance name to describe this list, for example Allowed country codes.

The instance name appears in the results, so a descriptive name tells you which check ran.

Applying the same rule to a second attribute lets you supply a different list, with no second rule to maintain.

Variables

In the Variables section, you can apply various transformations to your input attributes. Variables store this transformed data. They can be used directly in the rule implementation logic, or to define groups for aggregation rules.

For example, you can create a variable that contains a trimmed value of a string attribute and then apply the rule conditions only on the trimmed version.

To create variables from your attributes:

Select a name for your variable. Select Add variable if you require additional fields.
Select the attributes that you want to transform. Use the dropdown to choose from the list of input attributes. The transformations will be applied in the next step.

Alternatively, if you are an advanced user, use the dropdown to select Advanced Expression and define the attribute and the transformations using ONE expression language.
Add transformations using the Add transformation option. The transformations available depend on the data type of the selected attribute: for a full list of types and available transformations, see List of available transformations.

How to apply multiple transformations to one attribute

You can apply multiple transformations to a single attribute, but you can only add them in a logical sequence. For example, if you start with an attribute of data type String and apply the To integer transformation, you can’t then apply the Uppercase transformation, as it is only available for strings.

For the same reason, you can only remove the last transformation in the sequence at any given time. To remove a transformation, hover over it and select x to delete it.

List of available transformations

Expand to see all available transformations

Datatype Available transformations Description

Datatype	Available transformations	Description
`String`	`To float`	Changes data type from string to float.
`To integer`	Changes data type from string to integer.
`To long`	Changes data type from string to long.
`Uppercase`	Converts string characters to uppercase.
`Lowercase`	Converts string characters to lowercase.
`Trim`	Removes spaces from both ends of the string.
`Squeeze spaces`	Trims string and replaces repeated spaces with a single space.
`Remove non-digits`	Removes non-digit characters from string.
`Remove non-letters`	Removes non-letter characters from string.
`Integer`	`To float`	Changes data type from integer to float.
`To date`	Changes data type from integer to date.
`To long`	Changes data type from integer to long.
`Boolean`	`To String`	Changes data type from Boolean to string.
`Date`	`To datetime`	Changes data type from date to datetime.
`To string`	Changes data type from date to string.
`Datetime`	`To date`	Changes data type from datetime to date.
`To string`	Changes data type from datetime to string.
`Long`	`To float`	Changes data type from long to float.
`To string`	Changes data type from long to string.
`To date`	Changes data type from long to date.
`To datetime`	Changes data type from long to datetime.
`Float`	`To string`	Changes data type from float to string.
`Floor`	Rounds to the nearest integer that is less than or equal to float value.
`Ceiling`	Rounds to the nearest integer that is greater than or equal to float value.
`Round`	Rounds to nearest integer.

String

To float

Changes data type from string to float.

To integer

Changes data type from string to integer.

To long

Changes data type from string to long.

Uppercase

Converts string characters to uppercase.

Lowercase

Converts string characters to lowercase.

Trim

Removes spaces from both ends of the string.

Squeeze spaces

Trims string and replaces repeated spaces with a single space.

Remove non-digits

Removes non-digit characters from string.

Remove non-letters

Removes non-letter characters from string.

Integer

To float

Changes data type from integer to float.

To date

Changes data type from integer to date.

To long

Changes data type from integer to long.

Boolean

To String

Changes data type from Boolean to string.

Date

To datetime

Changes data type from date to datetime.

To string

Changes data type from date to string.

Datetime

To date

Changes data type from datetime to date.

To string

Changes data type from datetime to string.

Long

To float

Changes data type from long to float.

To string

Changes data type from long to string.

To date

Changes data type from long to date.

To datetime

Changes data type from long to datetime.

Float

To string

Changes data type from float to string.

Floor

Rounds to the nearest integer that is less than or equal to float value.

Ceiling

Rounds to the nearest integer that is greater than or equal to float value.

Round

Rounds to nearest integer.

Rule logic

Define the rule logic by defining the rule conditions and applying them to the inputs.

Select whether you are creating a standard Rule or an Aggregation rule. Aggregation rules require additional configuration, as described in Aggregation Rules.

Provide the rule conditions and an explanation of the results.

There are two ways to do this: through the Condition Builder or via Advanced Expression:

Select Advanced Expression to leverage ONE expressions in your rule condition.

Select Condition Builder to define the rule logic using the predefined options.

If you selected Advanced Expression, enter the expression in the space provided.

If you selected Condition Builder:

Select the input attributes to use in the rule logic.

Select any required modifiers, for example, Trim or Round. Different modifiers are available for string and integer inputs.

Expand to see all available modifiers

Modifier Description

Modifier	Description
`Value`	The given value of an attribute.
`Uppercase`	Uppercase version of string.
`Lowercase`	Lowercase version of string.
`Trim`	Removes any whitespaces from both sides of strings.
`Trim left`	Removes any leading whitespaces from strings.
`Trim right`	Removes any trailing whitespaces from strings.
`Squared`	Squares the value of an integer, float, or long data type attributes.
`Round`	Rounds the value of an integer, float, or long data type attributes to the nearest whole number.
`Average`	Averages the value of integer, float, or long data type attributes (available only when using aggregation rules and grouping).
`Min`	The minimum of string, integer, or long data types attributes (available only when using aggregation rules and grouping).
`Max`	The maximum of string, integer, or long data type attributes (available only when using aggregation rules and grouping).
`Sum`	The sum of of string, integer, or long data type attributes (available only when using aggregation rules and grouping).

Value

The given value of an attribute.

Uppercase

Uppercase version of string.

Lowercase

Lowercase version of string.

Trim

Removes any whitespaces from both sides of strings.

Trim left

Removes any leading whitespaces from strings.

Trim right

Removes any trailing whitespaces from strings.

Squared

Squares the value of an integer, float, or long data type attributes.

Round

Rounds the value of an integer, float, or long data type attributes to the nearest whole number.

Average

Averages the value of integer, float, or long data type attributes (available only when using aggregation rules and grouping).

Min

The minimum of string, integer, or long data types attributes (available only when using aggregation rules and grouping).

Max

The maximum of string, integer, or long data type attributes (available only when using aggregation rules and grouping).

Sum

The sum of of string, integer, or long data type attributes (available only when using aggregation rules and grouping).

Select the required conditions, for example, matches mask or has length of, and provide the necessary requirements.

The conditions available depend on the data type of the input. See Conditions available by data type.

For comparison conditions, you also choose what to compare the value against: a value you type, another input attribute, or a parameter.
Select whether the condition defined should produce the result that is, for example, Valid or Invalid. Options are different depending on the dimension chosen.

Optionally, enter an explanation to help distinguish between multiple rules of the same dimension.

Explanations are used as names of the conditions, and are also displayed as a results description in all kind of dashboards or reports. Use the explanation that would make sense to you when further analyzing data quality results.

How rule conditions are evaluated

A rule can contain more than one condition. Select Add condition to add another.

Conditions are evaluated in the order they are listed, and the first condition a record matches decides the outcome. The remaining conditions are not evaluated, so a record receives one result and one explanation even when it satisfies several conditions. This applies to every condition in the rule, whether you define it with the Condition Builder or as an Advanced Expression.

Records that match none of your conditions fall through to the IF none of the conditions above apply THEN section. Set the Result for those records there. The explanation for the fallback is predefined and cannot be changed.

Because the first match decides, list your conditions from the most specific to the most general. For example, if one condition catches an empty value and another catches a badly formatted one, put the empty-value condition first, so an empty value is reported as empty rather than as badly formatted.

To change the order later, select the three-dot menu on a condition, then select Move Up or Move Down. To turn off a condition without deleting it, select Disable in the same menu.

How combining or splitting conditions across rules affects your results

A rule returns one explanation per record, no matter how many conditions it contains.

Use several conditions in one rule when they are alternative reasons for the same result and the first applicable reason is enough. For example, a range check works as one rule with two conditions: too low and too high both mean the value is out of range.

Use separate rules when you need each check counted independently. A record that matches an earlier condition never reaches the later ones, so the count for a later explanation includes only the records that had no earlier problem. Separate rules avoid this, because each rule evaluates every record and carries its own result and its own threshold.

Test rule

Test the rule by selecting Test rule and trying different inputs in the Test section. The results for each input are shown automatically in Message next to the test input row.

Publish rule

Publish your changes.

Conditions available by data type

The conditions available in the Condition Builder depend on the data type of the input.

An input counts as empty only when it has no content at all. Text such as NULL, null, N/A, n/a, ., ,, -, or _ is content, so is empty does not match it.

If the rule has a List of values parameter, two further conditions become available for String and numeric inputs: is from parameter list and is not from parameter list. Both check the value against the list supplied through that parameter.

For conditions that read from a catalog item, select ON DATA CHANGE in Data updates so that the rule always uses the latest data. If you select a catalog item that is not managed as reference data, no automatic updates are available.

Aggregation rules add is unique and is not unique, which evaluate a group rather than a single value. See Aggregation Rules.

Conditions for String attributes

Condition Description

Condition	Description
`is empty`	The input has no value.
`is not empty`	The input has a value.
`=`	The value is the same as the one you compare it to.
`≠`	The value differs from the one you compare it to.
`contains`	The value contains the substring you define.
`does not contain`	The value does not contain the substring you define.
`starts with`	The value begins with the text you define.
`ends with`	The value ends with the text you define.
`has length of`	The value has the length you define.
`does not have length of`	The value does not have the length you define.
`like`	The value matches the pattern you define.
`is from Catalog item`	The value belongs to the catalog item attribute you select.
`is not from Catalog item`	The value does not belong to the catalog item attribute you select.
`is from Reference Data Catalog item`	The value belongs to the reference data catalog item you select.
`is not from Reference Data Catalog item`	The value does not belong to the reference data catalog item you select.
`is from list`	The value belongs to a list you type in when defining the condition.
`is not from list`	The value does not belong to a list you type in when defining the condition.
`is a number`	The value is numeric.
`is not a number`	The value is not numeric.
`matches mask`	The value corresponds to the mask you define.
`does not match mask`	The value does not correspond to the mask you define.
`matches regexp`	The value corresponds to the regular expression you define.
`does not match regexp`	The value does not correspond to the regular expression you define.

is empty

The input has no value.

is not empty

The input has a value.

=

The value is the same as the one you compare it to.

≠

The value differs from the one you compare it to.

contains

The value contains the substring you define.

does not contain

The value does not contain the substring you define.

starts with

The value begins with the text you define.

ends with

The value ends with the text you define.

has length of

The value has the length you define.

does not have length of

The value does not have the length you define.

like

The value matches the pattern you define.

is from Catalog item

The value belongs to the catalog item attribute you select.

is not from Catalog item

The value does not belong to the catalog item attribute you select.

is from Reference Data Catalog item

The value belongs to the reference data catalog item you select.

is not from Reference Data Catalog item

The value does not belong to the reference data catalog item you select.

is from list

The value belongs to a list you type in when defining the condition.

is not from list

The value does not belong to a list you type in when defining the condition.

is a number

The value is numeric.

is not a number

The value is not numeric.

matches mask

The value corresponds to the mask you define.

does not match mask

The value does not correspond to the mask you define.

matches regexp

The value corresponds to the regular expression you define.

does not match regexp

The value does not correspond to the regular expression you define.

Conditions for numeric attributes

Integer, Float, and Long attributes support the same conditions.

Condition Description

Condition	Description
`is empty`	The input has no value.
`is not empty`	The input has a value.
`=`	The value is the same as the one you compare it to.
`≠`	The value differs from the one you compare it to.
`<`	The value is lower than the one you compare it to.
`≤`	The value is lower than or equal to the one you compare it to.
`>`	The value is higher than the one you compare it to.
`≥`	The value is higher than or equal to the one you compare it to.
`is from Catalog item`	The value belongs to the catalog item attribute you select.
`is not from Catalog item`	The value does not belong to the catalog item attribute you select.
`is from Reference Data Catalog item`	The value belongs to the reference data catalog item you select.
`is not from Reference Data Catalog item`	The value does not belong to the reference data catalog item you select.
`is from list`	The value belongs to a list you type in when defining the condition.
`is not from list`	The value does not belong to a list you type in when defining the condition.

is empty

The input has no value.

is not empty

The input has a value.

=

The value is the same as the one you compare it to.

≠

The value differs from the one you compare it to.

<

The value is lower than the one you compare it to.

≤

The value is lower than or equal to the one you compare it to.

>

The value is higher than the one you compare it to.

≥

The value is higher than or equal to the one you compare it to.

is from Catalog item

The value belongs to the catalog item attribute you select.

is not from Catalog item

The value does not belong to the catalog item attribute you select.

is from Reference Data Catalog item

The value belongs to the reference data catalog item you select.

is not from Reference Data Catalog item

The value does not belong to the reference data catalog item you select.

is from list

The value belongs to a list you type in when defining the condition.

is not from list

The value does not belong to a list you type in when defining the condition.

Conditions for Date and Datetime attributes

Date and Datetime attributes support the same conditions.

The today conditions compare against the current date, so you do not have to enter a date yourself. Use them for timeliness checks, where what counts as late shifts every day.

Condition Description

Condition	Description
`is empty`	The input has no value.
`is not empty`	The input has a value.
`=`	The date is the same as the one you compare it to.
`≠`	The date differs from the one you compare it to.
`<`	The date is earlier than the one you compare it to.
`≤`	The date is the same as or earlier than the one you compare it to.
`>`	The date is later than the one you compare it to.
`≥`	The date is the same as or later than the one you compare it to.
`today`	The date is the current date.
`< today`	The date is earlier than the current date.
`≤ today`	The date is the current date or earlier.
`> today`	The date is later than the current date.
`≥ today`	The date is the current date or later.
`is from Catalog item`	The value belongs to the catalog item attribute you select.
`is not from Catalog item`	The value does not belong to the catalog item attribute you select.
`is from Reference Data Catalog item`	The value belongs to the reference data catalog item you select.
`is not from Reference Data Catalog item`	The value does not belong to the reference data catalog item you select.

is empty

The input has no value.

is not empty

The input has a value.

=

The date is the same as the one you compare it to.

≠

The date differs from the one you compare it to.

<

The date is earlier than the one you compare it to.

≤

The date is the same as or earlier than the one you compare it to.

>

The date is later than the one you compare it to.

≥

The date is the same as or later than the one you compare it to.

today

The date is the current date.

< today

The date is earlier than the current date.

≤ today

The date is the current date or earlier.

> today

The date is later than the current date.

≥ today

The date is the current date or later.

is from Catalog item

The value belongs to the catalog item attribute you select.

is not from Catalog item

The value does not belong to the catalog item attribute you select.

is from Reference Data Catalog item

The value belongs to the reference data catalog item you select.

is not from Reference Data Catalog item

The value does not belong to the reference data catalog item you select.

Conditions for Boolean attributes

Condition Description

Condition	Description
`is empty`	The input has no value.
`is not empty`	The input has a value.
`=`	The value is the same as the one you compare it to.
`≠`	The value differs from the one you compare it to.
`is true`	The value is true.
`is false`	The value is false.

is empty

The input has no value.

is not empty

The input has a value.

=

The value is the same as the one you compare it to.

≠

The value differs from the one you compare it to.

is true

The value is true.

is false

The value is false.

Next steps

Assign Stewardship — Assign an owner group to the rule.
Run DQ Evaluation — Run DQ evaluation to see how your data performs against the rule.
Data Quality Thresholds — Configure a threshold to flag results that fall below an acceptable level and trigger alerts automatically.

Was this page useful?