Export Invalid Records: Deep Dive

This page explains the technical details of how exports work, what data is included, and how records are stored and retained. For instructions on how to set up export of invalid records, see Export Invalid Records.

What’s included in the export

Each exported record includes:

Technical metadata:

table_path: Location of the source data.
processing_time: When the evaluation ran.
processing_id: Unique identifier for the evaluation run.
configuration_id: ID of the monitoring project that generated the results.

Invalid record details:

record_id: Unique identifier for the record (if defined in the source data), or a JSON object of composing columns if the identifier consists of multiple columns.
record_id: Unique identifier for the record (if defined in the source data). If the identifier consists of multiple columns, it is stored as a JSON object serialized to text.
rule_name: Name of the failed rule.
attribute: Name of the attribute that contains the invalid value.
invalid_value: The actual invalid value.

Rule context:

score: Score as defined in the rule.
explanation: Descriptive text explaining why the rule failed.

Additional attributes:

Any additional attributes you configured to provide context for investigation, stored as a JSON object serialized to text.

Storage and retention

Records are stored in the table you specify and never leave your database environment.

New records are appended with timestamps. The records are managed depending on the selected Write strategy: Replace or Append.

Consider cleaning up old export records when:

Table size impacts storage costs.
Historical records are no longer needed for audit.
Data quality issues have been resolved.

Was this page useful?