Cloud Portal ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
User Community Service Desk Downloads

Filter Step

Filters out all records that don’t pass the ONE expression condition.

This step is available for standalone plans, embedded plans, and transformation catalog items. It is not available for transformation rules.

Overview

The Filter step removes records from your data flow based on a condition. Only records that satisfy the condition continue to downstream steps; all other records are discarded.

Use this step to:

  • Remove invalid or incomplete records before processing.

  • Select a subset of data based on business criteria.

  • Exclude records with null values in required fields.

  • Apply quality thresholds to filter out low-quality data.

  • Limit data to specific categories, regions, or time periods.

Configuration

Define the filter condition

  1. In the step configuration, enter your filter condition using a ONE expression.

  2. The expression must evaluate to a Boolean value (true or false).

  3. Records where the condition evaluates to true are passed through.

  4. Records where the condition evaluates to false are removed.

Think of the filter condition as "keep records where this is true."

Condition examples

Filter by status
status = 'Active'

Keeps only records with Active status.

Filter by date range
order_date >= toDate('2024-01-01', 'yyyy-MM-dd') and order_date < toDate('2025-01-01', 'yyyy-MM-dd')

Keeps orders from 2024 only.

Filter out null values
email is not null and email != ''

Removes records without email addresses.

Filter by numeric threshold
amount > 100

Keeps only records above the threshold.

Filter by multiple conditions
country in {'US', 'CA', 'MX'} and customer_type = 'Business'

Keeps business customers from North America.

Filter using text patterns
matches(@"^[A-Z]{2}-[0-9]{4}$", product_code, false)

Keeps only records matching the expected code format.

Filter valid records
quantity > 0 and price is not null and customer_id is not null

Keeps only complete, valid records.

Common filter patterns

Data quality filters

Remove incomplete records
coalesce(required_field_1, '') != '' and
coalesce(required_field_2, '') != '' and
coalesce(required_field_3, '') != ''

Business logic filters

Active records only
is_deleted = false and status != 'Archived'
Recent data only
modified_date >= dateAdd(now(), -30, 'DAY')
Specific categories
category in {'Electronics', 'Appliances', 'Furniture'}

Filter vs Condition step

Use Filter when you want to keep only matching records and discard the rest.

Use Condition when you need to route records to different processing paths based on a condition (both matching and non-matching records continue processing).

Aspect Filter Condition

Outputs

Single output (matching records only)

Two outputs (out_true and out_false)

Non-matching records

Discarded

Routed to out_false for separate processing

Use case

Remove unwanted data

Route data to different processing branches

Best practices

Filter early

Apply filters early in your transformation plan to reduce the amount of data processed by downstream steps, improving performance.

Be explicit about nulls

Null values can cause unexpected behavior. Explicitly handle nulls in your conditions: field is not null and field = 'value'.

Test edge cases

Verify your filter logic handles edge cases correctly, including null values, empty strings, and boundary conditions.

Document filter purpose

Add a description to explain why records are being filtered. This helps others understand the business logic.

See also

Was this page useful?