Snowflake Pushdown Processing
Overview
Pushdown processing involves delegating data processing tasks (namely profiling and DQ evaluation) to your connected Snowflake warehouse instead of running them on Ataccama DPEs.
In this case, instead of transferring data into ONE for processing, ONE generates SQL queries that run directly within Snowflake, leveraging its computational resources. Only results and invalid samples, if configured, are returned to ONE.
Snowflake pushdown processing is supported for profiling and DQ evaluation jobs.
- Pushdown profiling
-
Profiling runs on pushdown for all Snowflake connections. There are no limitations when running profiling in Snowflake pushdown compared to profiling in ONE.
- Pushdown for DQ evaluation
-
Pushdown processing for DQ evaluation can be enabled or disabled as required, as some limitations exist. See When to Use Snowflake Pushdown for DQ Evaluation.
| Transformation catalog items and SQL catalog items cannot be processed in pushdown. |
Why use pushdown processing?
Using pushdown processing can lead to improvements in performance, security and infrastructure, and subsequently, scalability.
-
Performance. Evaluate millions of records in seconds.
-
Security. Data processing is executed directly in Snowflake, and results (and invalid samples, if configured) are returned. This means there is no transfer of data required or any external services or servers; all processes run on secured and governed Snowflake warehouses.
-
Infrastructure. Utilize your existing infrastructure to evaluate data—no need to set up and manage new processing servers.
How to proceed
-
To decide whether pushdown is the right choice for DQ evaluation, see When to Use Snowflake Pushdown for DQ Evaluation.
-
Ready to enable pushdown for DQ evaluation? Go to the tutorial: Enable Snowflake Pushdown for DQ Evaluation on Source.
Was this page useful?