Export Invalid Records

You can configure DQ monitors to automatically export records that fail validation rules to a table. Each time you run a DQ evaluation, the export updates with the latest results, ensuring you always have current invalid records for analysis and remediation.

Why export invalid records?

Exporting invalid records is useful for sharing results with data owners or stakeholders without access to Ataccama ONE, or for integrating with external remediation workflows.

Invalid record exports vs. samples

Invalid record exports: A complete dataset of all invalid records in an external table. Useful for comprehensive analysis, sharing, and remediation.
Invalid record samples: A limited sample of invalid data on the DQ monitor Invalid records samples tab. Useful for quick investigation within Ataccama ONE.

Set up export of invalid records

To configure automatic export:

Open your DQ monitor.
Select the Settings icon.
In the Export of invalid records section, select Set export of invalid records.
Configure the export settings.
Publish the changes.

The next time you run DQ evaluation, invalid records will be exported to your specified table.

Export settings

Target table

Specify where invalid records should be stored. The target table configuration depends on your processing type.

Non-pushdown DQ evaluation: Invalid records are automatically exported to a CSV table you can download. No target table configuration is required.
Pushdown DQ evaluation: Select a table where invalid records are exported:

Ensure your Snowflake role has the required permissions. See Required Snowflake permissions for details.
- Schema: Select the Snowflake schema where you want to export the table.
- Select in source: Create a new table or use an existing table in your Snowflake database.
  - Name: Enter the table name. If the table does not exist, it will be created automatically with the required structure.
  - Prefix: The prefix is fixed as ATA_INVALID_RECORDS_ for easy identification of export tables.
- Select from catalog: Select an existing catalog item as the export destination.
  
  The catalog item name must start with ATA_INVALID_RECORDS and must have all required columns. To create such catalog item, first create a table with these specifications in Snowflake, and then import the table to catalog.

Record limit

Set the maximum number of records to export per evaluation.

We recommend keeping the limit at 100,000 records or fewer. If you consistently see a high number (millions) of invalid records, this usually indicates misconfigured DQ rules or monitoring settings that require adjustment.

Additional attributes

By default, only the invalid attribute value is exported.

You can select additional attributes to provide context for investigating DQ issues. For example, if an email address fails validation, adding the country code helps you identify whether all issues originate from the same country.

When you select multiple attributes, they appear in a single column in the export table, separated by a delimiter.

View exported records

After DQ evaluation completes, all invalid records are written to your specified table:

For pushdown processing, view records directly in the Snowflake interface:
1. In the monitor Settings sidebar, go to the Export of invalid records section.
2. Select the link to your export table. This opens the table in Snowflake where you can view, query, and download the data.
For non-pushdown processing, download records from Ataccama ONE:
1. In the monitor Settings sidebar, go to the Export of invalid records section.
2. Select the link to download your export table.

Alternatively, select Download all invalid records from the monitor header to download the table.

Manage export configuration

You can modify or pause your export configuration at any time.

Edit the configuration: Changes take effect after you publish and run the next evaluation.
Pause: Temporarily stop exporting without removing the configuration. The next DQ evaluation will not export any new records.
Remove: Delete the export configuration.

Removing the export configuration does not delete the export table or its data from your Snowflake database or S3 storage.

Required Snowflake permissions

Ataccama ONE automatically creates the export table in Snowflake. To enable this, your Snowflake role must have the following permissions:

USAGE on the target database and schema
CREATE TABLE on the target schema
INSERT on the export table

Optionally, grant SELECT on the table if users need to query exported records directly in Snowflake.

Example: Granting required permissions

-- Grant database and schema access
GRANT USAGE ON DATABASE <database_name> TO ROLE <role_name>;
GRANT USAGE ON SCHEMA <database_name>.<schema_name> TO ROLE <role_name>;
-- Allow table creation in the schema
GRANT CREATE TABLE ON SCHEMA <database_name>.<schema_name> TO ROLE <role_name>;
-- Grant INSERT on the export table
GRANT INSERT ON TABLE <database_name>.<schema_name>.<table_name> TO ROLE <role_name>;
-- (Optional) Allow querying exported records
GRANT SELECT ON TABLE <database_name>.<schema_name>.<table_name> TO ROLE <role_name>;

Export invalid records: Deep dive

To learn more about the technical details of how exports work, what data is included, and how records are stored and retained, see Export Invalid Records: Deep Dive.

Was this page useful?