When to Use Pushdown for DQ Evaluation
Profiling jobs run in pushdown by default while pushdown for DQ evaluation can be enabled if needed. This allows you to restrict how pushdown for DQ evaluation is used on the connection level—for example, due to cost concerns or compute size limitations, which can impact performance.
Once you have enabled pushdown for DQ evaluation, it’s available for all users working in that connection.
For details about how to enable pushdown for DQ evaluation, see the relevant connection articles:
When to use pushdown for DQ evaluation
We recommend using pushdown for DQ evaluation wherever possible.
You might want to keep pushdown for DQ evaluation turned off in the following scenarios:
-
When compute costs in your source need to be minimized, or where a small compute size can limit performance.
-
If you rely only on rules which are not supported in pushdown for your source. See [source-specific-details] for the list of supported functions and operators.
| Even if you enable pushdown for DQ evaluation at the connection level, you can still turn it off for specific DQ monitors if necessary. See DQ Monitors. |
What rule types are supported
The following rule types are supported in pushdown for DQ evaluation:
-
All rules created using the Condition Builder, as all expression functions available in the condition builder are supported by pushdown. This includes aggregation rules built using the condition builder.
-
Parametric rules.
-
Advanced Expression rules, rules generated using the Ask AI feature, and rules using variables, as long as they include only supported functions and operators. See Supported functions and operators.
If a rule or function isn’t supported in pushdown, ONE detects this automatically and reports it. Unsupported rules fall back to running in Ataccama’s runtime.
What happens if pushdown for DQ evaluation isn’t enabled
If pushdown for DQ evaluation is not configured, data quality jobs run on Ataccama’s runtime, which involves some data transfer between your source and ONE.
You also won’t be able to enable pushdown for specific DQ monitors, as described in Processing settings.
Supported functions and operators
Some rule functions cannot be translated from Ataccama expressions to native functions in Snowflake (including UDF) or Databricks SQL. Ataccama’s runtime handles the translation of these configurations.
The following list contains functions and operators supported by pushdown regardless of your source. Use this list as a reference when creating rules.
If an expression is not present in this list, rules containing the expression aren’t supported and ONE flags it in the application.
Expand to see list
Operators
-
+(concatenate for string, addition for numeric values) -
-(subtraction) -
*(multiplication) -
/(division) -
%(modulo) -
div(division of integer values without a remainder) -
< -
⇐ -
<>,!= -
=,== -
> -
>= -
AND -
NOT -
OR -
XOR -
in -
is -
is in -
is not -
is not in -
is not null -
is null -
like -
not in
Functions
-
avg -
avgif -
containsWord -
count -
countDistinct -
dateAdd -
dateDiff -
datePart -
dateTrunc -
eraseSpacesInNames -
find -
getDate -
iif -
indexOf -
isInFile -
isNumber -
lastIndexOf -
left -
length -
lower -
math.abs -
math.acos -
math.asin -
math.atan -
math.ceil -
math.cos -
math.e -
math.exp -
math.floor -
math.log -
math.log10 -
math.longCeil -
math.longFloor -
math.pi -
math.pow -
math.round -
math.sin -
math.sqr -
math.sqrt -
math.tan -
matches -
maximum -
maximumif -
minimum -
minimumif -
now -
removeAccents -
replace -
right -
squeezeSpaces -
substituteAll -
substr -
sum -
sumif -
toDate -
toDateTime -
toFloat -
toInteger -
toLong -
toString -
today -
trashNonDigits -
trashNonLetters -
trim -
trimLeft -
trimRight -
upper
In Snowflake only, geographic functions are also supported:
-
geo.area -
geo.contains -
geo.coveredBy -
geo.covers -
geo.disjoint -
geo.intersects -
geo.isValid -
geo.length -
geo.perimeter -
geo.within
Why DQ evaluation requires extra storage in your source
Pushdown profiling runs entirely in your source’s compute and doesn’t require any extra setup.
Pushdown for DQ evaluation is different.
When your DQ rules reference catalog items (using conditions like is from Catalog item or is from Reference Data Catalog item), the referenced data must be available in your data source so pushdown SQL can join against it.
That’s why additional storage is required.
In Snowflake, this is stored as CSV files in a stage; in Databricks, as tables in your operational schema.
ONE uploads the data your rules rely on to your source before running DQ evaluation jobs. Ataccama does not export any data read from the tables in your source.
This data is updated when you run DQ evaluation and profiling jobs, and removed automatically after 30 days of inactivity.
In Snowflake, you can recognize such files by filenames containing lookup and the .csv extension.
|
For more about using catalog items in rules, see Evaluate Data Against Catalog Items.
Was this page useful?