Ataccama ONE Agentic ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
Downloads

How Freshness Is Detected by Source

Freshness monitoring detects data writes in two ways: source scanning and pipeline monitoring. This page covers source scanning. For pipeline monitoring and how freshness monitoring works overall, see Freshness Monitoring.

During source scanning, Ataccama ONE determines when a table was last written from a source-specific signal that it reads automatically.

System freshness by source

The signal ONE reads, its requirements, and other source-specific behavior vary by technology.

Databricks

System freshness reads lastModified from DESCRIBE DETAIL, which is the time of the table’s latest Delta commit. This reflects data writes and structural changes, and it does not scan the table data.

System freshness requires a Databricks SQL warehouse with Unity Catalog, SELECT on the table, and USE CATALOG and USE SCHEMA on its containers.

Good to know:

  • System freshness applies to Delta tables only.

  • Maintenance operations such as OPTIMIZE and VACUUM create new Delta commits, so under heavy maintenance the reported time is an upper bound on the last data write.

Snowflake

System freshness reads the LAST_ALTERED timestamp from INFORMATION_SCHEMA.TABLES, which reflects the most recent change to the table.

System freshness requires USAGE on the database and schema and SELECT on the table. No extra setup is needed.

Good to know:

  • LAST_ALTERED also advances on structural changes, such as altering a column, and on clone and time-travel operations.

  • For a view, LAST_ALTERED reflects when the view definition last changed, not when the underlying data changed.

PostgreSQL

System freshness reads the latest transaction commit time for the table’s rows from PostgreSQL commit-timestamp tracking.

System freshness requires SELECT on the table and the server setting track_commit_timestamp = on. This setting is off by default, and enabling it requires a server restart. On managed PostgreSQL, such as Amazon RDS or Aurora, Google Cloud SQL, or Azure Database, enable it through the provider’s parameter group and reboot the instance.

Good to know:

  • System freshness reflects inserts and updates, not deletes.

  • A table that has not changed for a long time can stop returning a timestamp, because routine maintenance eventually removes the commit-timestamp data for old rows.

  • Finding the latest commit timestamp scans the table, which can be costly on very large tables.

MS SQL Server

System freshness reads the time of the last write to the table from the engine’s index-usage statistics (sys.dm_db_index_usage_stats.last_user_update). This also applies to Azure SQL.

System freshness requires SELECT on the table, plus permission to read the statistics: VIEW DATABASE STATE on Azure SQL, or VIEW SERVER STATE on-premises.

Good to know:

  • These statistics are held in memory and reset when the instance or database restarts, or when an index is rebuilt. After a restart, a table that has not been written to yet returns no timestamp.

  • System freshness reliably confirms recent activity but cannot date a table that has been idle for a long time.

  • last_user_update advances on any insert, update, or delete that touches the table, including changes that do not affect meaningful data.

  • SQL Server instances can host multiple databases, but freshness is evaluated in the connection’s current database, so configure the connection against the database that holds the tables you want to monitor.

BigQuery

System freshness reads the storage last-modified time from INFORMATION_SCHEMA.TABLE_STORAGE. When project-level metadata access is not granted, ONE falls back to the dataset’s __TABLES__ metadata, so system freshness works either way.

System freshness requires permission to run queries (bigquery.jobs.create) and read access to the dataset (roles/bigquery.dataViewer). Project-level metadata access (roles/bigquery.metadataViewer) is recommended so ONE can use TABLE_STORAGE, but it is optional because of the fallback.

Good to know:

  • The reported time reflects when the table’s stored data was last changed. Rows still in the streaming buffer might not be reflected immediately.

Oracle

System freshness reads the time of the table’s last data change from Oracle’s DBA_TAB_MODIFICATIONS view.

System freshness requires SELECT on the table and SELECT on DBA_TAB_MODIFICATIONS, which is typically granted through the SELECT_CATALOG_ROLE role. Optionally, grant the ANALYZE ANY system privilege so ONE can flush Oracle’s in-memory modification tracking before each scan, which keeps very recent changes up to date.

Good to know:

  • The DBA_TAB_MODIFICATIONS view only covers changes made since table statistics were last gathered, so gathering statistics resets the tracked time.

  • Structural changes such as TRUNCATE are not tracked as data changes.

  • Without the ANALYZE ANY privilege, very recent changes, within the last few minutes, might not be reflected immediately.

  • An Oracle connection targets a single pluggable database (PDB), so configure the connection against the PDB that holds the tables you want to monitor.

See also

Was this page useful?