Data Transformation Plans - DQ Monitoring Results

This document explains how to create and use transformation plans for data quality monitoring results. For general information about data transformation plans, including the full list of available steps and their configuration, see Data Transformation Plans.

Overview

When creating a transformation that processes not only the source data but also data quality results (such as passed or failed rules), you should start creating your transformation plan from the desired monitoring project. This ensures the plan is run whenever the monitoring project runs, and that the output is available on the monitoring project Export tab.

When creating plans to transform monitoring project results, initiate the plan from the monitoring project itself to ensure correct processing.

Create plan to transform data quality monitoring results

In this example, we will create a plan which inputs results from a catalog item in a monitoring project, filters the results, and writes them to a .csv file.

The basic flow is as follows:

Initiate plan from your monitoring project
Add and configure transformation steps
Run monitoring

Step 1: Initiate the plan from a monitoring project

To create a new transformation plan from a monitoring project item, select the required project and navigate to the Configuration & Results tab.
On the relevant catalog item, select the three dots menu and then Add post-processing transformation.
Provide a name for your transformation plan.
This opens the plan editor with preconfigured inputs and outputs.

Step 2: Configure output and add transformation steps

By default, the output is a csv file named after the catalog item from which the project has been created. In this example, we will filter out all results which are not from the US, and all those without data quality issues, so we can rename the output file accordingly.
To filter out all data without data quality issues and which is not from the US, we can use the Filter step. To add a new step, use the plus icon on the connection, or select Add new step.
Select Filter. For our use case, we need the expression invalid_rules is not null and country = 'USA'.

For detailed information about other available steps and their configuration, see the Available Steps section in the main Data Transformation Plans documentation.

Step 3: Run the monitoring project

Running the monitoring project automatically triggers the transformation plan:

After navigating to the monitoring project from which you initiated the transformation plan, select Run monitoring.
After the jobs have finished, the results of the transformation plans are found on the monitoring project Export tab.

Monitoring project post-processing input step

The Monitoring project post-processing input step is specifically designed for data quality monitoring result transformation.

This step inputs the data and DQ evaluation results from a monitoring project.

When using the monitoring project post-processing input step, always initiate the plan from the monitoring project itself to ensure correct processing: don’t add the step in the plan editor using Add data input or Add step.

To use this step correctly:

Locate the required catalog item on the Configuration & Results tab of a monitoring project.
Select the three dots menu and then Add post-processing transformation.
Provide a name for the plan and select Confirm.
Your input step is ready to use with the monitoring project results.

File output for monitoring results

When transforming monitoring project results, the default output step is a File output step, which writes data into a text file in ONE Object Storage.

Text files generated as part of a DQ monitoring project are available on the Export tab of the relevant monitoring project. These files are only accessible when generated using the Monitoring project post-processing input step which is initialized from the monitoring project.

To configure the file output step:

In File name, provide a meaningful name for the output file that reflects its content.
Keep in mind that when writing to a text file, columns will be separated by delimiters.

Accessing transformation results

After running the monitoring project with a configured transformation plan:

Navigate to the monitoring project
Select the Export tab

Was this page useful?