Data Quality Retention Settings
Retention settings control how long Ataccama ONE keeps DQ evaluation and profiling results. You can set limits to comply with data protection regulations, reduce storage usage, or control whether sensitive data like invalid samples is collected or downloaded at all.
Configure retention settings
You need to be an application admin to configure retention settings. See Identity Provider Roles.
To configure retention settings:
-
Go to Global settings > Data retention settings > Data quality and profiling.
To access Global settings, select the Ataccama logo. -
Turn collection and download of invalid samples on or off, and set retention periods.
-
Publish the changes.
Retention isn’t applied the moment you publish. Results past their retention period are deleted at the start of every hour and whenever a DQ evaluation starts.
Available retention settings
Collect invalid samples
Turn collection of invalid record samples on or off across all DQ monitors:
-
When on, individual monitors can be configured to collect samples of invalid records during DQ evaluation.
-
When off, sample collection is unavailable in all monitors.
Collect invalid samples is on by default. Turning collection off doesn’t delete samples collected earlier: the retention settings still apply to them.
Allow invalid sample downloads
Control whether users can download collected invalid samples as a CSV file. The setting applies to all users in your organization. You can’t set it for individual users, roles, or catalog items. You can change it only while Collect invalid samples is on, because there are no samples to download when collection is off. After you publish a change to this setting, it takes effect when users refresh the page.
When the setting is off:
-
Download invalid sample is hidden on the DQ monitor Records tab.
-
Download is hidden in the invalid samples view.
-
Download requests made directly through the API are rejected.
-
Invalid samples can still be viewed in ONE.
Allow invalid sample downloads is off by default.
Keeping the setting off is recommended. To share invalid records outside ONE, use invalid records export instead, which writes the records to a table in your data source or to an S3 bucket you configured.
Delete history of data quality evaluations
Set how long to retain DQ evaluation and profiling results, as a number of days or a number of runs. Runs are counted separately for each DQ monitor.
After the specified number of days or runs, the results are permanently deleted and cannot be recovered.
| This includes invalid record samples and exported invalid records (for non-pushdown processing). If these contain sensitive data, you can set a shorter retention period using Delete invalid samples—whichever period is shorter applies. |
Delete invalid samples
Set a shorter retention period, in days or runs, for invalid record samples and exported invalid records.
These records contain actual field values that failed validation—often the most sensitive data from your source systems. Use this setting if you need to delete them sooner than other DQ results.
When both this setting and Delete history of data quality evaluations are on, invalid samples are deleted as soon as either limit is reached.
After the retention period, data is permanently deleted and cannot be recovered.
| For pushdown processing, exported records remain in your source database and you manage retention there. See Storage and retention for details. |
Best practices for setting retention periods
When setting retention periods, balance compliance, investigation, and storage needs.
Compliance first: Check your data protection regulations before setting retention periods. Some industries require specific retention periods or prohibit storing certain sensitive data.
Investigation workflows: Set retention periods long enough to investigate and resolve typical data quality issues. Consider your average issue resolution timeframe.
Trend analysis: Longer retention enables better trend analysis and helps identify patterns over time. It also improves anomaly detection accuracy.
Storage costs: Balance retention needs with storage budget. Longer retention periods increase storage usage.
Was this page useful?