DQ Evaluation in ONE Data
Using DQ evaluation in ONE Data helps you keep track of how your data evolves and makes sure you stay on top of any data quality issues. This article explains how DQ functions in ONE Data and how to evaluate data quality in your ONE Data tables.
In addition to this, you can use ONE Data tables in DQ rules and apply them to validate data in other data sources. For more information, see Validate Data using ONE Data Tables.
To learn how to resolve data quality issues in ONE Data, see Data Remediation with ONE Data.
How is data quality calculated?
In ONE Data, DQ results are available on attribute, row, and cell level. They are calculated during DQ evaluation by applying DQ rules to glossary terms assigned to specific attributes. If you import a catalog item from the Knowledge Catalog to ONE Data, the assigned glossary terms are also copied but the DQ results might not be, as this depends on the import settings you select.
In general, glossary terms are automatically assigned to attributes during profiling based on predefined detection rules mapped to these terms. Additionally, if AI-based term detection is enabled, you are also shown term suggestions, that is, terms that might be a good fit for your data based on the similarity of your data with other data to which the same terms are already assigned.
In the latter case, accept or reject proposed terms as this helps improve the quality of the suggestions. For more details, see getting-started-with-term-statistics-and-settings.adoc.
You can also apply glossary terms manually as needed, using the attribute details sidebar.
In addition, you can add any DQ rule directly to catalog item attributes from the table or the attribute details sidebar. This allows you to apply specific DQ rules on particular attributes without modifying the default rule configuration for any of the assigned terms.
|
To apply a rule directly to an attribute, expand the list of DQ rules on the attribute and select Add Rule, then choose all rules that you want applied. Alternatively, right-click the attribute name and select Open details. The Add Rule option is available under DQ Rules. If you change your mind or no longer want to use a manually assigned rule, select the X icon next to it to remove it and publish the changes. For more detailed instructions, see dq-quality:add-dq-rules-to-attributes.adoc. 
You can also pause applied rules either directly from the table or from the attribute details sidebar by clicking the pause icon. After you publish the change and rerun DQ evaluation, the rule is no longer used in evaluation until you reactivate it. 
|
In a nutshell, the results that you see on the Data tab in ONE Data correspond to the overall quality determined using globally configured DQ dimensions. In other words, the percentage shown is the aggregated result of all contributing DQ rules.
You can view more detailed results, including the specific DQ rules used, by switching to the Data Quality tab of the ONE Data table. To see how DQ is calculated for a particular attribute, right-click the attribute header and select Open details.
| To get a deeper understanding of how DQ is evaluated in this case, see Data Quality Dimensions. |
Evaluate DQ in ONE Data tables
Once you create a table in ONE Data, you can check the validity of your data by running Full Profiling & DQ Evaluation on it (available from the Knowledge Catalog or, if the table hasn’t yet been profiled, from the Profile & DQ insights tab of the table in ONE Data). This performs the full set of analyses on the whole table and is therefore best suited for the initial processing. For more information about other profiling options, see [running-data-discovery-and-profiling].
| For detailed instructions about how to create a table in ONE Data, see Create a Table, Import Data from a Catalog Item, or Import Data from a File, depending on where your data is stored. |
As a result, appropriate business terms appear in the attribute header and DQ results are visible on the attribute and the row headers. To interpret the results of DQ evaluation, check the following:
-
Overall quality: How many records in the table passed all applied DQ rules.
-
Attribute quality: How many records within a particular attribute passed the DQ rules.
-
Invalid records: Records or cells that failed any rules contributing to the Overall Quality metric are highlighted in red. A record is flagged as invalid if there is at least one invalid cell.
Hover over the warning to see which DQ rules the record didn’t pass.
Once initial DQ results are available for your table, records are validated on the fly when you edit them. This is reflected in DQ results on the cell and row level as well as in DQ filtering results. However, DQ results on the attribute level and the overall DQ quality remain the same until you run DQ evaluation again.
| If you are editing records in bulk, on-the-fly validation is not supported. |
Was this page useful?