Export Invalid Records: Deep Dive

This page explains the technical details of how exports work, what data is included, and how records are stored and retained. For instructions on how to set up export of invalid records, see Export Invalid Records.

What’s included in the export

Each exported record includes:

Technical metadata:

table_path: Location of the source data.
processing_time: When the evaluation ran.
processing_id: Unique identifier for the evaluation run.
monitor_id: ID of the DQ monitor that generated the results.

Invalid record details:

record_id: Unique identifier for the record (if defined in the source data).
rule_name: Name of the failed rule.
attribute: Attribute that contains the invalid value.
invalid_value: The actual invalid value.

Rule context:

score: Score as defined in the rule.
explanation: Descriptive text explaining why the rule failed.

Additional attributes:

Any additional attributes you configured to provide context for investigation.

Storage and retention

Export behavior depends on your processing type:

Pushdown DQ evaluation: Records are stored in the Snowflake table you specify and never leave your Snowflake environment.

New records are appended with timestamps. Previous records are not automatically deleted, so you are responsible for managing table cleanup.

Consider cleaning up old export records when:

Table size impacts storage costs.
Historical records are no longer needed for audit.
Data quality issues have been resolved.

Non-pushdown DQ evaluation: Records are stored in Ataccama’s S3 database and can be downloaded as CSV files.

Retention is controlled by the Delete invalid samples and exported records setting (see Data Quality Retention Settings).

Was this page useful?