Cloud Portal ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
latest (16.3.0) 16.2.0 16.1.0 16.0.0 15.4.0 15.3.0 15.2.0 15.1.0 14.5.x 13.9.x 13.8.x 13.7.x 13.6.x 13.5.0 13.4.x 13.3.x 13.2.x
User Community Service Desk Downloads

History Plugin

The history plugin tracks and stores historical versions of records in the MDM hub.

The main features are:

  • Configurable projection of the MDM model and data changes.

    • Only specified entities and columns are stored, for instance entities, only records from the configured source systems are stored.

    • Only changes of specified columns are considered as changes to be stored.

  • Independent optimized storage.

    • Historical data can be stored in a separate database (for example, with a different SLA or lower price per storage unit).

    • Data is stored as a compressed BLOB.

  • Native services and batch exports including time query support.

How to configure the history plugin

The history plugin is configured in the MDM Server application.properties file.

To enable the plugin, add the required properties to your application.properties file. For the complete list of properties and configuration options, see History plugin configuration.

Detailed configuration of the history model

The default configuration file with the history model is nme-history.xml, with the following structure:

<?xml version="1.0"?>
<config>
    <model>
        <entities>
            <entity name="party" layer="instance">
                <sourceSystems>
                    <sourceSystem>crm</sourceSystem>
                    <sourceSystem>life</sourceSystem>
                </sourceSystems>
                <columns>
                    <column name="src_first_name" />
                    <column name="uni_rule_name" traced="false" />
                    <column name="master_id" />
                    <!-- more columns -->
                </columns>
            </entity>
            <entity name="party" layer="master" masterView="mas">
            <columns>
                    <column name="cmo_first_name" />
                    <column name="cmo_sin" />
                    <!-- more columns -->
                </columns>
            </entity>
            <!-- more entities-->
        </entities>
    </model>
</config>

The configuration contains the identification of entities and columns from the Model (see Model) and history settings.

Every mentioned entity must be present in the model and is identified using the following attributes:

Attribute Description

name

Name of the entity in the model

layer

Type of layer: instance or master

masterView

Name of the master layer

sourceSystems

(Optional, instance layer only) List of source systems whose records are stored in history; if not specified, records from all source systems are stored.

Entities contain a list of columns with the following attributes:

Attribute Default Description

name

N/A

Name of the column in the MDM model.

traced

true

Whether changes to this column trigger the creation of new historical record versions:

  • true - Column changes create new record versions.

  • false - Column is stored only when other traced columns change.

searchable

false

Whether the column is stored separately for efficient filtering and permissions:

* true - Column is stored as a separate persistence column to support filtering. * false - Column is stored in compressed BLOB with other non-searchable columns (efficient filtering not possible).
The relationship between historical records and regular MDM data is through the ID column. The record in the history table takes over its MDM ID from the data table. That is how both instance and master records are linked with history.

History plugin native services

The History plugin provides native services get<entityName>InstanceHistoryById and get<entityName>MasterHistoryById. These services provide full history for one record identified by id (MDM internal ID).

get<entityName>InstanceHistoryById

Simple request:

Request
<get:getpartyInstanceHistoryById>
    <get:id>91212</get:id>
</get:getpartyInstanceHistoryById>
Response
<getpartyInstanceHistoryByIdResponse xmlns="http://www.ataccama.com/ws/nme/getpartyInstanceHistoryById">
    <party>
        <metadata>
            <id>91212</id>
            <origin>life#party#party</origin>
            <sourceSystem>life</sourceSystem>
            <valid_from>2015-04-10T16:50:32+02:00</valid_from>
            <valid_to>2015-04-10T16:50:55+02:00</valid_to>
            <active>true</active>
            <sourceTimestamp>2015-04-10T16:50:54+02:00</sourceTimestamp>
            <lastUpdateTid>123456</lastUpdateTid>
        </metadata>
        <attributes>
            <source_id>300</source_id>
            <src_first_name>Jack</src_first_name>
        </attributes>
    </party>
    <party>
        <metadata>
            <id>91212</id>
            <origin>life#party#party</origin>
            <sourceSystem>life</sourceSystem>
            <valid_from>2015-04-10T16:49:28+02:00</valid_from>
            <valid_to>2015-04-10T16:50:32+02:00</valid_to>
            <active>true</active>
            <sourceTimestamp>2015-04-10T16:50:31+02:00</sourceTimestamp>
            <lastUpdateTid>123455</lastUpdateTid>
        </metadata>
        <attributes>
            <source_id>300</source_id>
            <src_first_name>Johny</src_first_name>
        </attributes>
    </party>
    <!-- more history versions of this record -->
</getpartyInstanceHistoryByIdResponse>

Instance history can be retrieved via a combination of source_id and origin instead of id:

Request
<get:getpartyInstanceHistoryById>
    <get:source_id>300</get:source_id>
    <get:origin>life#party#party</get:origin>
</get:getpartyInstanceHistoryById>

To limit returned history, use timeQuery with from and/or to elements:

Request
<get:getpartyInstanceHistoryById>
    <get:id>91212</get:id>
    <get:timeQuery>
        <get:from>2015-04-10T16:50:32+02:00</get:from>
        <get:to>2015-04-10T10:50:32+02:00</get:to>
    </get:timeQuery>
</get:getpartyInstanceHistoryById>

Returns all record versions that were valid during the specified time period.

To get record history at specific point in time, use timeQuery with at element:

Request
<get:getpartyInstanceHistoryById>
    <get:id>91212</get:id>
    <get:timeQuery>
        <get:at>2015-04-10T10:50:32+02:00</get:at>
    </get:timeQuery>
</get:getpartyInstanceHistoryById>

The service will return only one history version or nothing if the record did not exist at a given point in time. If there were no relevant record changes, the service will return nothing because the requested version is in the MDM engine operational storage.

get<entityName>MasterHistoryById

This native service is similar to get<entityName>InstanceHistoryById. The only difference is that only id can be used to get record history (no source_id and origin). Returned metadata do not contain origin, sourceSystem, and sourceTimestamp.

History plugin batch export

When the history plugin is active, the following two data providers for batch exports are available:

  • com.ataccama.nme.history.batch.HistoryInstanceEntityDataSource

  • com.ataccama.nme.history.batch.HistoryMasterEntityDataSource

HistoryInstanceEntityDataSource

This data source provides history records of the specified instance entity.

Example configuration
<dataProvider class="com.ataccama.nme.history.batch.HistoryInstanceEntityDataSource">
    <prefix>history_inst_</prefix>
    <entity>party</entity>
</dataProvider>
Attribute Description

prefix

Prefix of integration inputs in the export plan using this provider.

entity

Name of the entity.

sourceSystem

Only records from this source system are provided.

HistoryMasterEntityDataSource

This data source provides history records of the specified master entity.

Example configuration
<dataProvider class="com.ataccama.nme.history.batch.HistoryMasterEntityDataSource">
    <prefix>history_master_</prefix>
    <entity>party</entity>
    <viewName>masters</viewName>
</dataProvider>
Attribute Description

prefix

Prefix of integration inputs in the export plan using this provider.

entity

Name of the entity.

viewName

Name of the master layer.

Limit export with time query

To limit history by timeQuery, start a batch export operation with the following parameters:

Parameter Description

timeQueryFrom

Only history versions valid from this date.

timeQueryTo

Only history versions valid to this date.

Time format can be either ISO format 2015-04-10T16:50:32+02:00 (for SOAP/native services) or simple format 2015-04-10 16:50:32.

Available metadata columns

In addition to the columns defined in the history model, the following columns are available in the export plan:

Column Type Description Instance/Master

id

LONG_INT

Internal record identifier.

I, M

eng_valid_from

DATE

Start date when this record version became valid.

I, M

eng_valid_to

DATE

End date when this record version was valid.

I, M

eng_active

BOOLEAN

Whether the record version is active (true) or inactive (false).

I, M

eng_origin

STRING

Source where the record originated.

I

eng_source_system

STRING

Name of the source system that provided the record.

I

eng_source_timestamp

DATE

Timestamp when the record version was created in the source system.

I

eng_nature

BOOLEAN

Whether the master record is authored (true) or consolidated (false).

M

eng_modified_by

STRING

User who last modified the record.

M

eng_last_update_tid

LONG_INT

Transaction ID of the last update to the record.

I, M

How time validity and time query work

Historical records have time validity periods defined by validFrom and validTo timestamps. These create a continuous timeline where each record version is valid for a specific period.

Example timeline:

  • Record v1: Valid from 10:00 to 10:15.

  • Record v2: Valid from 10:15 to 10:30.

  • Record v3: Valid from 10:30 to current.

Each record’s validity period uses a half-open interval [validFrom, validTo), meaning the validFrom time is included but validTo is not.

In general, which records are returned as a response to a query is determined by the following conditions:

Type of query Condition

at

record.validFrom ⇐ query < record.validTo

from

record.validTo > query

to

record.validFrom ⇐ query

Was this page useful?