Ataccama ONE Agentic Cloud Portal ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
User Community Service Desk Downloads

Export Invalid Records: Deep Dive

This page explains the technical details of how exports work, what data is included, and how records are stored and retained. For instructions on how to set up export of invalid records, see Export Invalid Records.

What’s included in the export

The export uses a single-rule-per-line format: each row represents one failed rule for one input attribute. If a record fails multiple rules or has multiple invalid attributes, it appears on multiple rows.

Each row includes the following columns:

Technical metadata:

  • asset: Location of the source data (for example, connection/db/schema/table).

  • processing_timestamp: When the evaluation ran.

  • processing_id: Unique identifier for the evaluation run.

  • monitor_id: ID of the DQ monitor that generated the results.

Invalid record details:

  • record_id: Identifier constructed from the key attributes you selected in the export settings. Contains key:value pairs (for example, customer_id:1). Empty if no key attributes are configured.

  • failed_rule_name: Name of the failed rule.

  • input_attributes: Attribute being validated.

  • invalid_values: The actual invalid value. When multiple values are involved, they are comma-separated.

Rule context:

  • criticality: Rule criticality level.

  • explanation: Descriptive text explaining why the rule failed.

Additional attributes:

  • additional_attributes: Any additional attributes you configured, shown as key:value pairs (for example, customer_id:1,age:29).

Storage and retention

Export behavior depends on your processing type:

  • Pushdown DQ evaluation: Records are stored in the Snowflake table you specify and never leave your Snowflake environment.

    New records are appended with timestamps. Previous records are not automatically deleted, so you are responsible for managing table cleanup.

Consider cleaning up old export records when:

  • Table size impacts storage costs.

  • Historical records are no longer needed for audit.

  • Data quality issues have been resolved.

  • Non-pushdown DQ evaluation: Records are stored in Ataccama’s S3 database and can be downloaded as CSV files.

    Retention is controlled by the Delete invalid samples and exported records setting (see Data Quality Retention Settings).

Was this page useful?