History Plugin
The history plugin tracks and stores historical versions of records in the MDM hub.
The main features are:
-
Configurable projection of the MDM model and data changes.
-
Only specified entities and columns are stored, for instance entities, only records from the configured source systems are stored.
-
Only changes of specified columns are considered as changes to be stored.
-
-
Independent optimized storage.
-
Historical data can be stored in a separate database (for example, with a different SLA or lower price per storage unit).
-
Data is stored as a compressed BLOB.
-
-
Native services and batch exports including time query support.
How to configure the history plugin
The history plugin is configured in the MDM Server application.properties
file.
To enable the plugin, add the required properties to your application.properties
file.
For the complete list of properties and configuration options, see History plugin configuration.
Detailed configuration of the history model
The default configuration file with the history model is nme-history.xml
, with the following structure:
<?xml version="1.0"?>
<config>
<model>
<entities>
<entity name="party" layer="instance">
<sourceSystems>
<sourceSystem>crm</sourceSystem>
<sourceSystem>life</sourceSystem>
</sourceSystems>
<columns>
<column name="src_first_name" />
<column name="uni_rule_name" traced="false" />
<column name="master_id" />
<!-- more columns -->
</columns>
</entity>
<entity name="party" layer="master" masterView="mas">
<columns>
<column name="cmo_first_name" />
<column name="cmo_sin" />
<!-- more columns -->
</columns>
</entity>
<!-- more entities-->
</entities>
</model>
</config>
The configuration contains the identification of entities and columns from the Model (see Model) and history settings.
Every mentioned entity
must be present in the model and is identified using the following attributes:
Attribute | Description |
---|---|
name |
Name of the entity in the model |
layer |
Type of layer: instance or master |
masterView |
Name of the master layer |
sourceSystems |
(Optional, instance layer only) List of source systems whose records are stored in history; if not specified, records from all source systems are stored. |
Entities contain a list of columns
with the following attributes:
Attribute | Default | Description |
---|---|---|
name |
N/A |
Name of the column in the MDM model. |
traced |
true |
Whether changes to this column trigger the creation of new historical record versions:
|
searchable |
false |
Whether the column is stored separately for efficient filtering and permissions: * |
The relationship between historical records and regular MDM data is through the ID column. The record in the history table takes over its MDM ID from the data table. That is how both instance and master records are linked with history. |
History plugin native services
The History plugin provides native services get<entityName>InstanceHistoryById
and get<entityName>MasterHistoryById
.
These services provide full history for one record identified by id
(MDM internal ID).
get<entityName>InstanceHistoryById
Simple request:
<get:getpartyInstanceHistoryById>
<get:id>91212</get:id>
</get:getpartyInstanceHistoryById>
<getpartyInstanceHistoryByIdResponse xmlns="http://www.ataccama.com/ws/nme/getpartyInstanceHistoryById">
<party>
<metadata>
<id>91212</id>
<origin>life#party#party</origin>
<sourceSystem>life</sourceSystem>
<valid_from>2015-04-10T16:50:32+02:00</valid_from>
<valid_to>2015-04-10T16:50:55+02:00</valid_to>
<active>true</active>
<sourceTimestamp>2015-04-10T16:50:54+02:00</sourceTimestamp>
<lastUpdateTid>123456</lastUpdateTid>
</metadata>
<attributes>
<source_id>300</source_id>
<src_first_name>Jack</src_first_name>
</attributes>
</party>
<party>
<metadata>
<id>91212</id>
<origin>life#party#party</origin>
<sourceSystem>life</sourceSystem>
<valid_from>2015-04-10T16:49:28+02:00</valid_from>
<valid_to>2015-04-10T16:50:32+02:00</valid_to>
<active>true</active>
<sourceTimestamp>2015-04-10T16:50:31+02:00</sourceTimestamp>
<lastUpdateTid>123455</lastUpdateTid>
</metadata>
<attributes>
<source_id>300</source_id>
<src_first_name>Johny</src_first_name>
</attributes>
</party>
<!-- more history versions of this record -->
</getpartyInstanceHistoryByIdResponse>
Instance history can be retrieved via a combination of source_id
and origin
instead of id
:
<get:getpartyInstanceHistoryById>
<get:source_id>300</get:source_id>
<get:origin>life#party#party</get:origin>
</get:getpartyInstanceHistoryById>
To limit returned history, use timeQuery with from and/or to elements:
<get:getpartyInstanceHistoryById>
<get:id>91212</get:id>
<get:timeQuery>
<get:from>2015-04-10T16:50:32+02:00</get:from>
<get:to>2015-04-10T10:50:32+02:00</get:to>
</get:timeQuery>
</get:getpartyInstanceHistoryById>
Returns all record versions that were valid during the specified time period.
To get record history at specific point in time, use timeQuery
with at
element:
<get:getpartyInstanceHistoryById>
<get:id>91212</get:id>
<get:timeQuery>
<get:at>2015-04-10T10:50:32+02:00</get:at>
</get:timeQuery>
</get:getpartyInstanceHistoryById>
The service will return only one history version or nothing if the record did not exist at a given point in time. If there were no relevant record changes, the service will return nothing because the requested version is in the MDM engine operational storage.
History plugin batch export
When the history plugin is active, the following two data providers for batch exports are available:
-
com.ataccama.nme.history.batch.HistoryInstanceEntityDataSource
-
com.ataccama.nme.history.batch.HistoryMasterEntityDataSource
HistoryInstanceEntityDataSource
This data source provides history records of the specified instance entity
.
<dataProvider class="com.ataccama.nme.history.batch.HistoryInstanceEntityDataSource">
<prefix>history_inst_</prefix>
<entity>party</entity>
</dataProvider>
Attribute | Description |
---|---|
prefix |
Prefix of integration inputs in the export plan using this provider. |
entity |
Name of the entity. |
sourceSystem |
Only records from this source system are provided. |
HistoryMasterEntityDataSource
This data source provides history records of the specified master entity.
<dataProvider class="com.ataccama.nme.history.batch.HistoryMasterEntityDataSource">
<prefix>history_master_</prefix>
<entity>party</entity>
<viewName>masters</viewName>
</dataProvider>
Attribute | Description |
---|---|
prefix |
Prefix of integration inputs in the export plan using this provider. |
entity |
Name of the entity. |
viewName |
Name of the master layer. |
Limit export with time query
To limit history by timeQuery
, start a batch export operation with the following parameters:
Parameter | Description |
---|---|
timeQueryFrom |
Only history versions valid from this date. |
timeQueryTo |
Only history versions valid to this date. |
Time format can be either ISO format 2015-04-10T16:50:32+02:00
(for SOAP/native services) or simple format 2015-04-10 16:50:32
.
Available metadata columns
In addition to the columns defined in the history model, the following columns are available in the export plan:
Column | Type | Description | Instance/Master |
---|---|---|---|
id |
LONG_INT |
Internal record identifier. |
I, M |
eng_valid_from |
DATE |
Start date when this record version became valid. |
I, M |
eng_valid_to |
DATE |
End date when this record version was valid. |
I, M |
eng_active |
BOOLEAN |
Whether the record version is active ( |
I, M |
eng_origin |
STRING |
Source where the record originated. |
I |
eng_source_system |
STRING |
Name of the source system that provided the record. |
I |
eng_source_timestamp |
DATE |
Timestamp when the record version was created in the source system. |
I |
eng_nature |
BOOLEAN |
Whether the master record is authored ( |
M |
eng_modified_by |
STRING |
User who last modified the record. |
M |
eng_last_update_tid |
LONG_INT |
Transaction ID of the last update to the record. |
I, M |
How time validity and time query work
Historical records have time validity periods defined by validFrom
and validTo
timestamps.
These create a continuous timeline where each record version is valid for a specific period.
Example timeline:
-
Record v1: Valid from 10:00 to 10:15.
-
Record v2: Valid from 10:15 to 10:30.
-
Record v3: Valid from 10:30 to current.
Each record’s validity period uses a half-open interval [validFrom, validTo)
, meaning the validFrom
time is included but validTo
is not.
In general, which records are returned as a response to a query is determined by the following conditions:
Type of query | Condition |
---|---|
at |
record.validFrom ⇐ query < record.validTo |
from |
record.validTo > query |
to |
record.validFrom ⇐ query |
Was this page useful?