Ataccama ONE Agentic ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
latest (17.1.0) 17.0.0 16.3.x 16.2.0 16.1.0 16.0.0 15.4.0 15.3.0 15.2.0 15.1.0 14.5.x 13.9.x 13.8.x 13.7.x 13.6.x 13.5.0 13.4.x 13.3.x 13.2.x
User Community Service Desk Downloads

Internal Columns

This topic covers the columns not defined in the Model and created or interpreted by the engine.

Database columns

Database (DB) columns are part of each instance or master table maintained by the engine. The column names (both database columns and pseudocolumns) cannot be renamed or used for a different purpose than specified. Moreover, their values cannot be changed directly, but only by the engine processes or features.

The DB columns are added by the engine automatically. Some of the columns are also available in ONE plans serving as transitions under either database column name (DB) or audit pseudocolumn name (AP) - see Available in a plan as DB/AP column in the table below. There are also Audit pseudocolumns, which are (usually) not related to any database column, and they are available in ONE plans only.

There are no database types shown in the following table because they are dependent on the particular DB used.

Database column Instance table Master table Description Available in a plan as DB/AP Data type

id

x

  • Unique id automatically assigned to each incoming instance record by the MDM engine.

  • Serves as PK when DB persistence is used (see Persistence Layer, section DB Persistence).

DB

long_int (3)

id

x

  • Mapped in merge plan manually from master_id as a result of representative creator step.

  • Serves as PK when DB persistence is used (see Persistence Layer, section DB Persistence).

DB

long_int (3)

id

  • Id automatically assigned to each imported or created record.

long_int (3)

source_id (1)

x

+

  • Unique id of source system record; mapped in load plan/process delta service.

DB

String

master_id

x

+

  • Used to store the result of default matching; filled in match plan as a result of the unification step.

  • Also serves as a foreign key to the corresponding master entity.

DB

long_int (3)

match_isolate

x

  • Support for manual match operations (split_isolate).

DB

Boolean

eng_origin

x

  • Stores the origin id of data origin, defined in [prefix]_ORIGIN table, CODE column (number).

  • Related AP contains a textual representation of the origin_id; equal to origin mapped in a load plan/process delta service.

AP

String

duplicate_id

x

  • An engine column on a master entity.

  • Populated automatically when creating an authored record that, based on the master match rules, potentially matches an existing master record (consolidated or authored).

eng_key

eng_active

x

x

AP

Boolean

eng_existing

  • In general, when a record is deleted, it has eng_existing=false. These records are not deleted physically because they are used for the purposes of exporting.

  • Determines if the record is reported from the source system as existing (when using sourceDeletionStaregy=DELETE). If the record is deleted, eng_existing = false.

N/A

Boolean

eng_existing

x

  • In general, when a record is deleted, it has eng_existing=false.

  • These records are not deleted physically, because they have to be there for export purpose.

  • Determines if the record is reported as deleted. It happens when two groups are merged into the one and eng_existing=false is set for the master record which was terminated.

N/A

Boolean

eng_existing

  • In general, when a record is deleted, it has eng_existing=false.

  • Determines if the record is reported as deleted.

eng_modified_by

x

Stores the effective execution identity of the operation that last modified the record. The value reflects the identity context under which the modifying operation ran and is not always a human user.

Possible values:

  • NmeEwfIdentity: The operation was executed via an EWF workflow integration, including workflow runs triggered by the scheduler.

  • NmeStreamIdentity: The operation was executed via a streaming interface.

  • <account_name>: The operation was executed under a request identity (web application, API, or service call). The account can be either a human user or a technical/service account.

String

eng_nature

x

  • Shows whether the record is authored or consolidated (data type).

Boolean

eng_source_timestamp

x

see below

AP

Datetime

eng_source_system

x

see below

AP

String

eng_creation_tid

x

x

see below

AP (2)

long_int (3)

eng_creation_date

x

x

see below

AP

Datetime

eng_deletion_tid

x

x

see below

AP (2)

long_int (3)

eng_deletion_date

x

x

see below

AP

Datetime

eng_last_update_tid

see below

AP (2)

long_int (3)

eng_last_update_date

x

x

see below

AP

Datetime

eng_last_source_update_tid

x

see below

AP (2)

long_int (3)

eng_last_source_update_date

x

see below

AP

Datetime

eng_activation_tid

x

x

see below

AP (2)

long_int (3)

eng_activation_date

x

x

see below

AP

Datetime

eng_deactivation_tid

x

x

see below

AP (2)

long_int (3)

eng_deactivation_date

x

x

see below

AP

Datetime

xid (4)

x

x

primary key of each table

N/A

long_int (3)

xctid (4)

x

x

internal technical column

N/A

long_int (3)

(1) Now defined as a string with size 1000, mapped to DB as varchar(2000). This is hardcoded in the MDM source code now. The maximum allowed content is 1000 chars.

(2) Technically, the column might be available in ONE plan. In general, various TID/DATE appearances in ONE plans depend on a particular MMD definition.

(3) Currently used in MDM only. It refers to a data type longer than integer and shorter than long, with its size limited to 8 bytes due to performance reasons (max value is 264, roughly 18 exabytes).

(4) Present only when VLDB Persistence is used, see Persistence Layer, section VLDB Persistence.

Audit pseudocolumns

The following pseudocolumns are available in various places across engine configuration, for example, ONE plans and WebServices. DB columns are described above.

Batch load pseudocolumns

Appear only in batch load plans, and then they are remapped into an eng_ prefixed columns and persisted as described above.

Pseudocolumn Data type Description

origin

String

Serves to map already defined origin, that is equal to textual representation eng_origin.

change_type

String

Serves as input to the delta detection process. It’s mapped into eng_change_type.

_source_timestamp

Datetime

Serves to provide external timestamp to the MDM. It’s mapped into eng_source_timestamp.

Transition plans pseudocolumns

The allowing pseudocolumns are available in ONE plans serving as transitions in MDM and in export plans.

Status pseudocolumns

Pseudocolumn Data type Description

eng_active

Boolean

  • Instance records: Determines if the record is reported from the source system as existing (when using sourceDeletionStaregy=DEACTIVATE).

  • Master records: Determines if at least one instance record corresponding to this master has eng_active=true.

eng_origin

String

Instance records only.

Contains the textual representation of the record origin. It is stored as origin_id column in a database.

eng_source_system

String

Instance records only.

Contains the textual representation of the system corresponding to the record origin. Database column name is eng_system.

eng_source_timestamp

Datetime

Instance records only.

The aim of this column is to prevent entering and overwriting the most recent record by the older ones based on the timestamp provided.

Modification date pseudocolumns

A set of pseudocolumn pairs (date + transaction ID) describing when a particular record-level event took place.

Date columns are of DATETIME type, transaction IDs are of LONG type. Only the date-related pseudocolumns are available in all transition and export plans, but all the columns are persisted in a database as described.

The column values for new records are set by default during the Change Detection phase, however, for expanded records (for example CopyColumns, or aggregation) the values will be set during the Committing phase. This behavior means that expanded records always show previous values in the transition plans, and these values will be different from what you find in the MDM storage once the transaction completes.
Transaction ID column Date column Event

eng_creation_tid

eng_creation_date

Record creation. Always non-null.

eng_deletion_tid

eng_deletion_date

Record deletion. Null for existing records.

Only makes sense in internal plans and delta exports.

eng_last_update_tid

eng_last_update_date

Last update of the record. Will be changed every time record is changed, including record I/U/D from input or loaded during processing. Always non-null.

eng_last_source_update_tid

eng_last_source_update_date

Only for instance records; the last update of the record in delta detection phase (excludes internal calculations). That is, the last time the record was sent by source system and change was detected in the delta detection phase.

Always non-null.

eng_activation_tid

eng_activation_date

(Last) record activation. Always non-null.

eng_deactivation_tid

eng_deactivation_date

(Last) record deactivation. Null for active records.

Inner plan pseudocolumns

The following pseudocolumns are not persisted in the database - they are only available in transition plans (computed for each loading operation):

Pseudocolumn Data type Description

eng_change_type

Char

Determines if the record was changed in the current load. The change type is evaluated against the eng_existing column.

  • N - No change (this typically appears only during retrofit operations).

  • I - Newly added record.

  • D - Deleted record.

  • U - An existing record that was modified (no insert/delete).

eng_activity_change_type

Char

Similar to eng_change_type, determines if the record was changed in the current load with respect to eng_active. See Status pseudocolumns for reference.

  • N - No change (this typically appears only during retrofit operations).

  • I - Newly activated record.

  • D - Deactivated or inactive record.

  • U - An active record that was modified (no activation/deactivation).

eng_external_record

Boolean

true if the record is not an actual record in the hub, for example, a record sent in by the Identify Service (see Services).

Delta export pseudocolumns

Only appear during the delta data exports. They are not persisted in the database (but the information allowing to determine the state is preserved).

Pseudocolumn Data type Description

eng_change_type

Char

Determines the change between the state in reference time and the current state.

  • I - The record has been newly inserted since the reference time (present at the moment).

  • D - The record has been deleted since the reference time (not present at the moment).

  • U - The record has been modified since the reference time. No insert or delete operation took place.

  • If the record was deleted and then inserted (or vice versa) since the reference time, I (resp. D) will be reported - depending on whether the record is (I) or is not (D) present currently.

eng_activity_change_type

Char

Determines the change of the eng_active pseudocolumn.

  • I - The record has been newly activated since the reference time (active at the moment).

  • D - The record has been deactivated since the reference time (not active at the moment).

  • X - The record was inactive at the reference time, hasn’t been activated in the period. Only appears when the scope is EXISTING.

  • U - The record has been modified since the reference time. No change on eng_active flag took place. This also means that eng_active = true, as otherwise an X would be reported.

  • If the record was deactivated and then activated (or vice versa) since the reference time, I (resp. D) will be reported - depending on whether the record is (I) or is not (D) active currently.

Web services pseudocolumns

Appear only in web services.

Request pseudocolumns

Pseudocolumn Data type Description

sourceId

String

Used to map instance primary key to the source_id attribute for traversal/genericTraversal (see Services).

origin

String

Serves as identification of requested record (together with sourceId) for some of the MDM Native Services (see Services).

change_type

String

Serves as input to the delta detection process for RW web service only. It’s mapped into eng_change_type.

source_timestamp

Datetime

Serves as input to the change detection process for RW web service only. It’s mapped into eng_source_timestamp.

Was this page useful?