Data Transformations

Data transformations is a set of capabilities that goes beyond simple data quality validation and provides powerful tools to transform your data.

ONE lets you apply these tools in the form of transformation plans. With transformation plans, you can create data processing job definitions in a visual way. You can then apply these plans to transform your data.

Why transform your data?

Transforming your data can be essential to start getting value from it. The following data transformation use cases might be a reason why you would want to transform your data and can be achieved with transformation plans.

Prepare data

Prepare your datasets for use in ONE. This can include modifying and combining source datasets, parsing selected attributes, and applying complex filters.

Standardize data

Unify data you want to use in and out of ONE. This can include standardizing structure, format, business domains, and other kinds attributes used within your datasets.

Enrich data

Add additional value to your data with data enrichment processes. This can include adding additional information from external sources and utilizing reference data.

Data transformation types and when to use them

There are four types of transformations:

Standalone plans: Define logic to transform the selected input data. Standalone plans have to be run to output the result in a selected Database output.

Best suited for performing one-time or scheduled transformations on specific datasets.
Embedded plans: Define transformation logic that can be reused in standalone plans or transformation rules in the form of Embedded plan steps.

Opt for embedded plans when you have a common transformation logic that needs to be reused across multiple plans.
Transformation rules: Plans you create to predefine how data should be altered, standardized, or converted to meet business requirements. The rules can then be applied to transformation catalog items as needed without having to define any of the logic repeatedly.

Use transformation rules to enforce consistent data standards across your organization.
Transformation catalog item (TCI): Catalog item defined by a transformation plan. The contents of a TCI are dynamic and depend on the catalog item mapped to the input and transformations applied on the Data Flow or standalone plans or transformation rules applied on the Data transformations tab.

Recommended when you require dynamically updated datasets based on some shared transformation logic.

Transformation editor visual canvas

Data transformations are built using a visual canvas that is shared across all available transformation types. It is available in the Transformation editor tab.

An exception to this are transformation catalog items (TCIs), in which case it is found on the Data Flow tab.

Example transformation

This canvas is where you design and build your transformation plans by adding steps and connecting them together.

How are data transformations built?

To build a data transformation, you connect different building blocks called transformation steps. Each transformation plan follows a logical flow that processes your data from input to output.

Transformation steps

There are three types of transformation steps that work together to form a complete data pipeline:

Steps

Input: Defines the data input type of the step.
Transform: Defines the transformation logic used on the input data.
Output: Defines how to output the processed data.

Basic transformation flow

All transformation plans follow this fundamental structure:

Start with input: Begin by adding one or more input steps to bring data into your transformation.
Add transformations: Connect your input to transformation steps such as filter, join, or add attributes to process your data according to business requirements. You can also embed plans within plans using embedded plans.
Define output: Complete your plan by connecting output steps that determine where your transformed data will be stored.
The execution of the plan depends on which type of data transformation you built:
- Standalone plan: Run the transformation from the three dots menu on the Transformation editor tab.
- Embedded plans: Executed as part of other plan types.
- Transformation rules: Results are dynamically updated on the TCI where they are applied.
- TCIs: Dynamically updated based on the applied transformation logic. This can be done through transformation rules, standalone plans, or full transformation built on the Data flow tab.

Next steps

Was this page useful?