Anomaly Detection in Catalog Items
Potential anomalies are detected during the profiling of catalog items on two levels:
-
Catalog item level, on metrics such as number of records, summary statistics, and so on.
-
Attribute level, on metrics such as value distribution, standard deviation, numeric sum.
The value of the metric is considered to be an anomaly if it is beyond the usual range. Anomaly detection is a process handled with AI, meaning that with every confirmed or dismissed alert, ONE learns more about the data and its possible values, which helps in detecting anomalies more precisely with every profiling.
| Anomaly Detection is only run if it is enabled in the profiling configuration used. For more information, see xref:[Configure Profiling Settings]. |
| Anomaly detection starts working from the 6th profile on, meaning that it is not activated for fewer profiles. |
Run anomaly detection
To run anomaly detection on catalog items:
-
In Data Catalog > Catalog Items, select the required item.
-
Use the dropdown to select Full profiling
If custom profilings have been added which include anomaly detection, you can also select one of these. For more information, see xref:[Configure Profiling Settings].
To run anomaly detection on sources:
-
In Data Catalog > Sources, select the required source.
-
Use the dropdown to select the Document documentation flow.
Detected anomalies
The presence of anomalies is marked using the warning icon. You can see information about any potential anomalies detected during profiling at a number of points in the Data Catalog:
-
Catalog item list view.
-
Attribute list view within a catalog item.
-
In the Relationships and Lineage graphs (if Show Anomalies is selected in graph settings, see xref:[Configure Graph Style]).
-
On the Profile and DQ Insights tab of a catalog item or attribute.
You can also view an aggregated list of all the catalog items with detected anomalies on the Anomaly Overview screen:
| You can filter the detected anomalies according to a date range. If no date is provided in the To field, the current date applies. |
View anomalous metrics
To view the metrics in the catalog item or attribute which are considered anomalous, open the Anomaly Inspector by clicking on the warning icon and then selecting Show more information.
Anomaly inspector
Once you have opened the anomaly inspector, you can select the following filters:
-
Only anomalous: Shows the metrics with detected anomalies.
-
Anomalous items, all charts: Shows the charts calculated for the attribute with metrics with detected anomalies.
-
All items and charts: Shows all the attributes in the catalog item with their metric charts.
Before confirming or dismissing the anomaly, you can view the metrics in detail.
Catalog item metrics
-
Number of records: the number of records in the catalog item is checked with every profiling.
The gray background in the chart indicates the expected range of the values. Hover over the data points on the chart to see more details.
Attribute metrics
The anomalous results for a particular metric are shown over time with the highlighted outliers. Hover over the points on the chart to see more details such as values, profiling versions, and time.
-
Number of records
-
Mean
-
Minimum
-
Standard deviation
-
Numeric sum
-
Variance
-
Distinct count
-
Duplicate count
-
Non-unique count
-
Null count
-
Maximum
-
Unique count
-
Frequency, masks, and patterns
Confirm or dismiss anomalies
Once anomalies have been detected, you can either confirm them or dismiss them. The anomaly detection model is constantly improved based on this user feedback.
To do this:
-
In Is this an anomaly, select Yes to confirm the anomaly, or No, it’s expected to dismiss the anomaly.
Unconfirmed anomalies
| If detected anomalies are not confirmed, the system does not know to exclude them from the expected range. After some time (depending on the length of profiling history), the unsolved anomalies are considered the 'new normal', and a return to the expected values can subsequently be identified as anomalous. |
Was this page useful?