Cloud Portal ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
User Community Service Desk Downloads

When to Use Snowflake Pushdown for DQ Evaluation

Profiling jobs run in pushdown by default while using pushdown for DQ evaluation can be enabled if needed. This allows you to restrict how pushdown for DQ evaluation is used on the connection level—for example, due to cost concerns or warehouse size limitations, which can impact performance.

Once you have enabled pushdown for DQ evaluation, it’s available for all users working in your Snowflake connection.

For details about how to enable pushdown for DQ evaluation, see Snowflake Connection > Enable pushdown for DQ evaluation.

When to use pushdown for DQ evaluation

We recommend using pushdown for DQ evaluation wherever possible.

You might want to keep pushdown for DQ evaluation turned off in the following scenarios:

  • When Snowflake compute costs need to be minimized or where a small warehouse size can limit performance.

  • If you rely only on rules which are not supported in Snowflake pushdown. See [What’s supported].

Even if you enable pushdown for DQ evaluation at the connection level, you can still turn it off for specific DQ monitors if necessary. See DQ Monitors.

What’s supported

The following rules are supported in Snowflake pushdown:

  • All rules created using the Condition Builder, as all expression functions available in the condition builder are supported by pushdown. This includes aggregation rules built using the condition builder.

  • Parametric rules.

  • Advanced Expression rules, rules generated using the Ask AI feature, and rules using variables, as long as they include only Supported functions and operators.

Supported functions and operators

Some rule functions cannot be translated from Ataccama expressions to native Snowflake functions or Snowflake UDFs. Ataccama’s cloud runtime handles the translation of these configurations to Snowflake functions.

The following list contains all supported functions and operators. If an expression is not present in this list, rules containing the expression aren’t supported in pushdown.

Expand to see list

Operators

  • + (concatenate)

  • <

  • ⇐

  • <>, !=

  • =, ==

  • >

  • >=

  • AND

  • NOT

  • OR

  • XOR

  • addition

  • division

  • in

  • is

  • is in

  • is not

  • is not in

  • is not null

  • is null

  • like

  • multiplication

  • not in

  • subtraction

Functions

  • avg

  • avgif

  • bitand

  • bitneg

  • bitor

  • bitxor

  • capitalize

  • capitalizeWithException

  • case

  • coding.fromBase64

  • coding.md5, encode.md5

  • coding.toBase64, encode.base64

  • concatenateif

  • containsWord

  • count

  • countDistinct

  • countDistinctIf

  • countNonAsciiLetters

  • countUnique

  • countUniqueif

  • countif

  • cpConvert

  • dateDiff

  • dateAdd

  • datePart

  • dateTrunc

  • decode

  • diceCoefficient

  • distinct

  • doubleMetaphone

  • editDistance

  • eraseSpacesInNames

  • find

  • first

  • firstif

  • frequency

  • getDate

  • getMilliseconds

  • getParameterValue

  • getRuntimeVersion

  • geoDistance

  • hamming

  • iif

  • indexOf

  • isInFile

  • isNotInFile

  • isNumber

  • jaccardCoefficient

  • jaroWinkler

  • last

  • lastIndexOf

  • lastif

  • left

  • length

  • levenshtein

  • lower

  • matches

  • math.sin

  • math.abs

  • math.acos

  • math.asin

  • math.atan

  • math.ceil, math.ceiling

  • math.cos

  • math.e

  • math.exp

  • math.floor

  • math.log

  • math.log10

  • math.longCeil, math.longCeiling

  • math.longFloor

  • math.pi

  • math.pow

  • math.round

  • math.sqrt

  • math.sqr

  • math.tan

  • max

  • maximum

  • maximumif

  • metaphone

  • min

  • minimum

  • minimumif

  • modus

  • modusif

  • ngram

  • now

  • nvl

  • preserveCase

  • random

  • randomUUID

  • removeAccents

  • replace

  • replicate

  • right

  • safeMax

  • safeMin

  • set.contains

  • sortWords

  • soundex

  • squeezeSpaces

  • substr

  • substituteAll

  • substituteMany

  • sum

  • sumif

  • toDate

  • toDateTime

  • toFloat

  • toInteger

  • toLong

  • toString

  • today

  • transliterate

  • trashConsonants

  • trashDiacritics

  • trashNonDigits

  • trashNonLetters

  • trashVowels

  • trim

  • trimLeft

  • trimRight

  • upper

  • word

  • wordCombinations

  • wordCount

What happens if pushdown for DQ evaluation isn’t enabled

If pushdown for DQ evaluation is not configured, data quality jobs run on Ataccama’s cloud runtime, which involves some data transfer between Snowflake and Ataccama.

You also won’t be able to enable pushdown for specific DQ monitors, as described in DQ Monitors > Processing settings.

Why Ataccama uses a Snowflake stage

When your DQ rules reference catalog items (using conditions like is from Catalog item or is from Reference Data Catalog item), this reference data must be available in Snowflake for pushdown execution.

Ataccama uploads the relevant reference data to your Snowflake stage before running DQ evaluation jobs. These files contain only the catalog item data you’ve configured in your rules. Ataccama does not export or stage any data read from your Snowflake tables.

Ataccama uses PUT, LIST, and REMOVE commands to manage these files. They are updated when you run DQ evaluation and profiling jobs, and removed automatically after 30 days of inactivity.

You can recognize these files in Snowflake by filenames containing _lookup_ and the .csv extension.

For more about using catalog items in rules, see Evaluate Data Against Catalog Items.

Was this page useful?