Export Invalid Records: Deep Dive

This page explains the technical details of how exports work, what data is included, and how records are stored and retained. For instructions on how to set up export of invalid records, see Export Invalid Records.

Storage and retention

Export behavior depends on your processing type:

Non-pushdown DQ evaluation: Records are stored as CSV files in the S3 bucket you configured. Access them using the S3 URI copied from All invalid records on the DQ monitor Records tab.

Retention is controlled by the Delete invalid samples and exported records setting (see Data Quality Retention Settings).
Pushdown DQ evaluation: Records are stored in the table you specify in your data source and never leave it.

Each evaluation replaces the previous results. Once the new records are inserted, the records from earlier runs of the same monitor on the same source table are deleted, so the table holds the most recent run only. This also happens when a run finds no invalid records, which empties the table of that monitor’s earlier results.

Records written by other monitors, or by the same monitor on a different source table, are left alone. A table shared by several monitors therefore holds one run’s records for each of them.

What’s included in the export

The export uses a single-rule-per-line format: each row represents one failed rule for one input attribute. If a record fails multiple rules or has multiple invalid attributes, it appears on multiple rows.

The two export structures contain the same information but differ in column names and ordering.

Non-pushdown exports (S3)

Each row in the exported CSV files includes the following columns:

Column Description

Column	Description
`asset`	Location of the source data. Example: `PUBLIC/SP_DT_CUSTOMERS`.
`invalid_values`	The actual invalid value. When multiple values are involved, they are comma-separated. Example: `minigiftsdistributorsltd.@gmail.com`.
`failed_rule_name`	Name of the failed rule. Example: `Valid Email Format`.
`explanation`	Descriptive text explaining why the rule failed. Example: `Does not match the email format`.
`input_attributes`	Attribute being validated. Example: `EMAIL`.
`criticality`	Rule criticality level.
`additional_attributes`	Any additional attributes you configured, shown as key:value pairs. Example: `customer_id:1,age:29`.
`record_id`	Identifier constructed from the key attributes you selected in the export settings. Contains the key attribute values, comma-separated, without attribute names. A key attribute with no value is left empty. Empty if no key attributes are selected. Example: `1`, or `1,5` for a composite key.
`processing_timestamp`	When the evaluation ran. Example: `2026-05-28T12:48:33.828Z`.
`processing_id`	Unique identifier for the evaluation run. Example: `232ca0c0-0000-8010-8000-0000d5c10015`.
`monitor_id`	ID of the DQ monitor that generated the results. Example: `232ca0c0-0000-7000-0000-0000001500e2`.

asset

Location of the source data.

Example: PUBLIC/SP_DT_CUSTOMERS.

invalid_values

The actual invalid value. When multiple values are involved, they are comma-separated.

Example: minigiftsdistributorsltd.@gmail.com.

failed_rule_name

Name of the failed rule.

Example: Valid Email Format.

explanation

Descriptive text explaining why the rule failed.

Example: Does not match the email format.

input_attributes

Attribute being validated.

Example: EMAIL.

criticality

Rule criticality level.

additional_attributes

Any additional attributes you configured, shown as key:value pairs.

Example: customer_id:1,age:29.

record_id

Empty if no key attributes are selected.

Example: 1, or 1,5 for a composite key.

processing_timestamp

When the evaluation ran.

Example: 2026-05-28T12:48:33.828Z.

processing_id

Unique identifier for the evaluation run.

Example: 232ca0c0-0000-8010-8000-0000d5c10015.

monitor_id

ID of the DQ monitor that generated the results.

Example: 232ca0c0-0000-7000-0000-0000001500e2.

Pushdown exports (Snowflake and Databricks)

Each row in the export table includes the following columns:

Column Description

Column	Description
`table_path`	Location of the source data. Example: `CLOUD_DEMO/BANK_TRANSACTIONS`.
`processing_time`	When the evaluation ran. Example: `2026-07-08T10:16:31.314Z`.
`processing_id`	Unique identifier for the evaluation run. Example: `4e47d5a0-0000-8010-8000-0000543f1be6`.
`configuration_id`	ID of the DQ monitor that generated the results. Example: `4e47d5a0-0000-8010-8000-000021f10056`.
`record_id`	Identifier constructed from the key attributes you selected in the export settings. Contains the key attribute values, comma-separated, without attribute names. A key attribute with no value is left empty. Values are written the way your data source converts them to text, so dates and timestamps can look different from the non-pushdown export. Empty if no key attributes are selected. Example: `1`, or `1,5` for a composite key.
`invalid_value`	The actual invalid value. When multiple values are involved, they are comma-separated. Example: `XX`.
`rule_name`	Name of the failed rule. Example: `Country Code Length Rule`.
`attribute`	Attribute being validated. Example: `COUNTRY_CODE`.
`explanation`	Descriptive text explaining why the rule failed. Example: `Not exactly 3 letters`.
`score`	Rule score. DQ evaluation currently does not produce scores; the column always contains `0`.
`additional_attributes`	Any additional attributes you configured, stored as a JSON object. Example: `{"APPROVAL_STATUS":"Pending","ORDER_AMOUNT":11529.37}`.

table_path

Location of the source data.

Example: CLOUD_DEMO/BANK_TRANSACTIONS.

processing_time

When the evaluation ran.

Example: 2026-07-08T10:16:31.314Z.

processing_id

Unique identifier for the evaluation run.

Example: 4e47d5a0-0000-8010-8000-0000543f1be6.

configuration_id

ID of the DQ monitor that generated the results.

Example: 4e47d5a0-0000-8010-8000-000021f10056.

record_id

Identifier constructed from the key attributes you selected in the export settings. Contains the key attribute values, comma-separated, without attribute names. A key attribute with no value is left empty. Values are written the way your data source converts them to text, so dates and timestamps can look different from the non-pushdown export.

Empty if no key attributes are selected.

Example: 1, or 1,5 for a composite key.

invalid_value

The actual invalid value. When multiple values are involved, they are comma-separated.

Example: XX.

rule_name

Name of the failed rule.

Example: Country Code Length Rule.

attribute

Attribute being validated.

Example: COUNTRY_CODE.

explanation

Descriptive text explaining why the rule failed.

Example: Not exactly 3 letters.

score

Rule score. DQ evaluation currently does not produce scores; the column always contains 0.

additional_attributes

Any additional attributes you configured, stored as a JSON object.

Example: {"APPROVAL_STATUS":"Pending","ORDER_AMOUNT":11529.37}.

Create the pushdown export table yourself

Ataccama ONE creates the export table for you. You only need the statements here if you create the table in advance, for example to use it with the Select from catalog option.

Match the structure exactly. Records are inserted by column position, not by column name, so the table needs all 11 columns in the order given in the preceding table. An extra or missing column fails the insert.

Column names have to match as well, because the cleanup that runs after each insert filters on table_path, processing_time, processing_id, and configuration_id. In Snowflake, name the columns in quoted lowercase as shown. An unquoted table_path becomes TABLE_PATH, and the cleanup then fails.

Table definition for Snowflake

CREATE TABLE IF NOT EXISTS <database>.<schema>.<table> (
  "table_path" VARCHAR,
  "processing_time" TIMESTAMP_TZ,
  "processing_id" VARCHAR,
  "configuration_id" VARCHAR,
  "record_id" VARCHAR,
  "invalid_value" VARCHAR,
  "rule_name" VARCHAR,
  "attribute" VARCHAR,
  "explanation" VARCHAR,
  "score" NUMBER,
  "additional_attributes" VARCHAR
) CLUSTER BY ("table_path", "processing_time", "processing_id", "configuration_id");

Table definition for Databricks with Unity Catalog

CREATE TABLE IF NOT EXISTS <catalog>.<schema>.<table> (
  `table_path` STRING,
  `processing_time` TIMESTAMP,
  `processing_id` STRING,
  `configuration_id` STRING,
  `record_id` STRING,
  `invalid_value` STRING,
  `rule_name` STRING,
  `attribute` STRING,
  `explanation` STRING,
  `score` INT,
  `additional_attributes` STRING
) CLUSTER BY (`table_path`, `processing_time`, `processing_id`, `configuration_id`);

Was this page useful?