Cloud Portal ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
User Community Service Desk Downloads

When to Use Snowflake Pushdown for DQ Evaluation

Profiling jobs run in pushdown by default, whereas to use pushdown for DQ evaluation requires you need to enable it at the source level. This allows you to restrict the use of the feature if necessary—for example, due to cost concerns or warehouse size limitations, which can impact performance.

Once you have enabled pushdown for DQ evaluation it’s available for all users working in your Snowflake connection.

For our tutorial on how to enable pushdown for DQ evaluation, see Enable Snowflake Pushdown for DQ Evaluation on Source.

When should you use pushdown for DQ evaluation?

It is recommended to use DQ for pushdown wherever possible. You might consider leaving DQ evaluation disabled in the following scenarios:

  • When Snowflake compute costs need to be minimized or where a small warehouse size can limit performance.

  • If you rely only on rules which are not supported in Snowflake pushdown. Keep reading to see whats’s supported?.

Even if you enable pushdown for DQ evaluation at the source level, you can still disable it for specific DQ configurations if necessary. For more information, see Set up DQ Configuration.

What’s supported?

The following rules are supported in Snowflake pushdown:

  • All rules created using the Condition Builder, as all expression functions available in the condition builder are supported by pushdown. This includes aggregation rules built using the condition builder.

  • Parametric rules.

  • Advanced Expression rules, rules generated using the Ask AI feature, and rules using variables, as long as they include only Supported functions and operators.

Supported functions and operators

There are some rule functions which cannot be translated from Ataccama expressions to native Snowflake functions or Snowflake UDFs.

The full list of supported functions and operators can be found here. If an expression is not present in this list, rules containing the expression aren’t supported in pushdown.

Expand to see list

Operators

  • + (concatenate)

  • <

  • < =

  • <>, !=

  • =, ==

  • >

  • > =

  • AND

  • NOT

  • OR

  • XOR

  • addition

  • division

  • in

  • is

  • is in

  • is not

  • is not in

  • is not null

  • is null

  • like

  • multiplication

  • not in

  • subtraction

Functions

  • avg

  • avgif

  • bitand

  • bitneg

  • bitor

  • bitxor

  • capitalize

  • capitalizeWithException

  • case

  • coding.fromBase64

  • coding.md5, encode.md5

  • coding.toBase64, encode.base64

  • concatenateif

  • containsWord

  • count

  • countDistinct

  • countDistinctIf

  • countNonAsciiLetters

  • countUnique

  • countUniqueif

  • countif

  • cpConvert

  • dateDiff

  • dateAdd

  • datePart

  • dateTrunc

  • decode

  • diceCoefficient

  • distinct

  • doubleMetaphone

  • editDistance

  • eraseSpacesInNames

  • find

  • first

  • firstif

  • frequency

  • getDate

  • getMilliseconds

  • getParameterValue

  • getRuntimeVersion

  • geoDistance

  • hamming

  • iif

  • indexOf

  • isInFile

  • isNotInFile

  • isNumber

  • jaccardCoefficient

  • jaroWinkler

  • last

  • lastIndexOf

  • lastif

  • left

  • length

  • levenshtein

  • lower

  • matches

  • math,sin

  • math.abs

  • math.acos

  • math.asin

  • math.atan

  • math.ceil, math.ceiling

  • math.cos

  • math.e

  • math.exp

  • math.floor

  • math.log

  • math.log10

  • math.longCeil, math.longCeiling

  • math.longFloor

  • math.pi

  • math.pow

  • math.round

  • math.sqrt

  • math.srq

  • math.tan

  • max

  • maximum

  • maximumif

  • metaphone

  • min

  • minimum

  • minimumif

  • modus

  • modusif

  • ngram

  • now

  • nvl

  • preserveCase

  • random

  • randomUUID

  • removeAccents

  • replace

  • replicate

  • right

  • safeMax

  • safeMin

  • set.contains

  • sortWords

  • soundex

  • squeezeSpaces

  • substr

  • substituteAll

  • substituteMany

  • sum

  • sumif

  • toDate

  • toDateTime

  • toFloat

  • toInteger

  • toLong

  • toString

  • today

  • transliterate

  • trashConsonants

  • trashDiacritics

  • trashNonDigits

  • trashNonLetters

  • trashVowels

  • trim

  • trimLeft

  • trimRight

  • upper

  • word

  • wordCombinations

  • wordCount

Ataccama DPEs handle the translation of these configurations to Snowflake functions.

What happens if I don’t enable pushdown for DQ evaluation?

If pushdown for DQ evaluation is not enabled, data quality jobs run on Ataccama Data Processing Engines (DPEs). This involves some data transfer between Snowflake and Ataccama, and may require multiple engines depending on the project’s scale. You also won’t be able to enable pushdown for specific DQ configurations, as described in Set up DQ Configuration.

Was this page useful?