Lead your team forward
OCT 24 / 9AM ET Register nowModel
The MDM model is the definition of logical layers (instance and master), event handlers, and settings.
Models, and their database connections and persistence types, are defined in nme-config.xml
(see MDM Configuration).
<config>
...
<model>
<models>
<consolidation>
<persistenceLayer class="com.ataccama.nme.persistence.database.DbPersistenceFactory">
<dataSource>mdc_db</dataSource>
<prefix>C_</prefix>
</persistenceLayer>
<configFile>nme-model.gen.xml</configFile>
<eventHandlersConfigFile>nme-event_handler.gen.xml</eventHandlersConfigFile>
<executionPlanConfigFile>nme-consolidation-plan.xml</executionPlanConfigFile>
</consolidation>
</models>
</model>
...
</config>
Configuration Parameters
A full model configuration consists of the following parameters:
Parameter | Required | Description |
---|---|---|
persistenceLayer |
Y |
The type of persistence layer to be used for this model (the |
persistenceLayer/dataSource |
Y |
The name of the database connection in runtime configuration (see Runtime Configuration). |
persistenceLayer/prefix |
N |
This prefix will be added to all tables created for this model. |
configFile |
Y |
The path to the main configuration file of the model. Contains the definition of the logical models: entities and their relationships, transformation operations, and other settings. |
eventHandlersConfigFile |
N |
The path to the event handler configuration for this model type (see Event Handler). Each model type has a separate configuration. |
executionPlanConfigFile |
N |
Optional, used only for the Consolidation model. The path to the execution (operation) plan (see Operation Plan). |
Model Definition
The configured model - nme-model.gen.xml
- is located in Files/etc
.
The file is generated when at least the instance layer is configured.
The main purpose is to define:
-
Instance layer
-
instance entities and relationships among them
-
copy columns
-
operations
-
source system definition and mapping to MDM instance entities
-
-
Master layers
-
master layer name
-
master entities and relationships
-
relationships among master and instance entities
-
entity behavior and operations
-
external entities and relationships
-
-
Basic structure of nme-model.gen.xml
is the following:
<?xml version='1.0' encoding='UTF-8'?>
<model>
<instanceLayer>
<entities>
...
</entities>
<sourceSystems>
</sourceSystems>
</instanceLayer>
<masterLayers>
<masterLayer name="Master">
<entities>
...
</entities>
<externalEntities>
...
</externalEntities>
</masterLayer>
<masterLayer name="...">
...
</masterLayer>
</masterLayers>
</model>
As shown here, it is possible to define one instance layer including entities and source system definitions, and several master layers and their entities. The file also contains a link to the event handler configuration (see Event Handler).
Instance Layer
Instance Entities
Defines entities and their behavior on the instance layer. An entity is defined by columns, relationships and operations as shown in the following example.
<entities>
<entity name="party">
<overridable data="true" matching="true" activity="true"/>
<columns>
<column name="src_first_name" origin="source" type="string" size="100"/>
<column name="std_first_name" origin="clean" type="string" size="100"/>
..
</columns>
<relationships/>
<cleansingOperation>
..
</cleansingOperation>
<matchingOperation>
..
</matchingOperation>
<aggregatingOperation>
..
</aggregatingOperation>
</entity>
<entity name="rd_gender" usage="DICTIONARY">
<overridable data="true" matching="true" activity="true"/>
<columns>
<column name="master_code" origin="source" size="1000" type="string"/>
<column name="master_name" origin="source" size="1000" type="string"/>
</columns>
</entity>
</entities>
Parameters:
-
name
: the name of the entity. -
usage
: usage of the entity.-
DATA_ENTITY
: default, standard data entity. -
DICTIONARY
: entity with a relatively small number of records used for storing dictionary values, no indices will be generated on relationships from data entity to dictionary.
-
-
<overridable>
: defines ifdata
,matching
, andactivity
overrides are enabled for the instance entities.
Column
Column definition for instance entities is done this way:
<columns>
<column name="src_name" origin="source" type="string" size="100" overridable="true"/>
<column name="std_name" origin="clean" type="string" size="100"/>
<column name="src_ssn" origin="source" type="string" size="100" overridable="true"/>
<column name="std_ssn" origin="clean" type="string" size="100"/>
<column name="uni_can_id" origin="match" type="long"/>
<column name="agg_group_count" origin="aggregate" type="string" size="1000"/>
</columns>
Parameters:
-
name
: the name of the column. -
origin
(column definition context - instance columns only): defines a place in a plan where the column appears.-
source
: source value, appears in cleanse plan input. -
clean
: cleansed, appears as a cleanse plan output. -
match
: match value, appears as a match plan output. -
aggregate
: aggregation output, appears as an aggregation plan output.
-
-
type
: data type. -
size
: column length (used as a database column size) that is mandatory forstring
data type. -
overridable
: specifies if data overrides are allowed on the column level.
Copy Columns
Copy column functionality is necessary for copying an attribute from one entity to another. To define a copy column, there must exist a relationship definition between the entities. After it is defined, it is possible to define the direction of copy columns: parent to child or child to parent.
<column name="mrg_addresses" origin="COPY_CLEAN" type="STRING" size="100">
<valueSource relationshipName="addresses" columnName="std_value">
<aggregation class="com.ataccama.nme.engine.model.ConcatenateDistinct" separator=";" maxLength="100"/>
<filterExpression>source.std_address_type = "R"</filterExpression>
</valueSource>
</column>
Parameters:
-
name
,type
andsize
: see the column definition example in this section. -
origin
(copy column context): defines which columns are taken and where the copied columns appear.-
copy_source
: a source column added to a cleansing plan output. -
copy_clean
: a cleansed result column added to a match plan input. -
copy_match
: a matched result column added to a merge plan input.
-
-
valueSource
:-
columnName
: the source column for the copy. -
relationshipName
: the relationship definition and the direction to be used for the copy.
-
-
aggregation class
:-
ConcatenateDistinct
: used to perform child to parent copy column. Because of the 1:M relationship, there is a concatenate distinct functionality to store all the values related to a single attribute on parent entity.-
separator
: the values separator that is used for concatenation. -
maxLength
: the length of the concatenated outcome.
-
-
MinValue
: used to perform child to parent copy column. It takes the minimum value from all related entities (because of 1:M relationship). -
MaxValue
: used to perform child to parent copy column. It takes the maximum value from all related entities (because of 1:M relationship). -
FirstValue
: used to perform parent to child copy column. It might be used for child to parent copy column, but there is no deterministic selection.
-
-
isFk
(boolean): if true, this copy column is used as a foreign key for this relationship. -
expression
: the expression that is used to fill copy columns. -
<filterExpression>
(boolean): optional; provides an ability to determine when column copying should be performed using a filter expression. There are two pseudoinputs available:source
(origin of copy) andtarget
.
History Collector
Enables to keep historical values of a given column and stores concatenated values to a newly defined column. Such functionality might be useful in a match process.
<column name="std_uir_adr_id_h" origin="CLEAN" type="STRING" size="40000">
<historyCollector columnName="std_uir_adr_id" separator="^~" maxCount="20"/>
</column>
Parameters:
-
columnName
: the name of the source column. -
separator
: the values separator definition for concatenation. -
maxCount
: the number of values kept in history.
OldValueProvider
OldValueProvider
is a way to access old value of instanceEntity
column in transition plan.
<column name="std_expiration_date" origin="clean" type="datetime" size="100">
<oldValueProvider columnName="std_expiration_date_old" />
</column>
Parameters:
-
columnName
: defines the target column witholdValue
, which is then available in transition plans.
oldValueProvider
can be defined for columns with following origins:
Origin of column | oldValue column is available in transition plan |
---|---|
SOURCE |
clean, match, merge |
COPY_SOURCE |
clean, match, merge |
CLEAN |
clean, match, merge |
COPY_CLEAN |
match, merge |
MATCH |
match, merge |
COPY_MATCH |
merge |
Do not use oldValue-based expressions after the Extended Unification step in a match plan. Some additional records might be expanded from unification repository during the match process and result will not be exact. Instead, move this computation to the merge plan, where all the relevant data and their oldValues are available. |
Instance Relationship
Relationship definition is always a part of a child entity definition and it points to a parent entity.
<relationships>
<rel reverseName="addresses" parentEntity="party" parentColumn="source_id" name="party" foreignKeyColumn="pty_source_id">
<type class="com.ataccama.nme.engine.model.SameSystemRelationship"/>
</rel>
</relationships>
Parameters:
-
rel
: main relationship definition with the following attributes.-
name
: relationship name; used as a reference name for copy columns. -
reverseName
: reverse relationship name; used as a reference name for copy columns. -
parentEntity
: name of the parent entity. -
parentColumn
: primary key column of the parent entity. -
foreignKeyColumn
: foreign key column of the child entity.
-
-
type
: type of relationship defined in theclass
attribute.-
SameSystemRelationship
(default): relationship is created only between identifiers having the same origin, that is, within a single system. -
CrossSystemRelationship
: enables to create a relationship between two entities regardless the origin. Useful for code books and dictionaries. Same definition as an instance relationship, excepttype
is not defined here since there is no longer a source system reference. Therefore, the behavior is the same as described forSameSystemRelationship
. -
SameSystemExtendedRelationship
: relationship is created only between identifiers having the same origin, that is, within a single system. However, it is possible to define where information about record origin is stored (for the purpose of evaluating this relationship only):-
record source system might be stored in any column (type string, column origin source) as source system name (same name as defined in source system definition), for example, "crm".
-
either
parentSystemColumnName
orchildSystemColumnName
or both must be defined:-
parentSystemColumnName
: name of column in the parent entity containing source system name of the parent record. -
childSystemColumnName
: name of column in the child entity containing source system name of the child record.
-
-
if the defined column contains
null
, the record origin is used; if defined column contains string that does not match any of the defined source systems, an exception is thrown (during the MDM consolidation process in CopyColumns action, during web service call with preloadedRelationships or traversal)
-
-
<relationships>
<rel reverseName="related_parties"
parentEntity="party"
parentColumn="id"
name="parent_party"
foreignKeyColumn="parent_pty_id">
<type class="com.ataccama.nme.engine.model.SameSystemExtendedRelationship">
<parentSystemColumnName>src_party_system</parentSystemColumnName>
<childSystemColumnName>src_parent_pty_system</childSystemColumnName>
</type>
</rel>
</relationships>
Instance Operation
Cleanse Operation
Defines a cleanse logic for a given entity which is represented by a ONE plan.
<cleansingOperation class="com.ataccama.nme.dqc.operations.CleansingPlan" planFileName="../trans/party/party_clean.comp"/>
Match Operation
Defines a match logic for a given entity, which is represented by a ONE plan. Currently there is only one implementation of matchingOperation available and it is compatible only with the Matching step.
<matchingOperation planFileName="../engine/trans/party/party_match.comp" class="com.ataccama.nme.dqc.mdu.MduMatchingPlan"/>
Multiple Matching steps in one matching plan
It is possible to have multiple Matching steps in one matching plan to achieve some extremely complex matching scenarios. In such case every Matching step must have filled different Repository Name and should have different bindings. Moreover, the matching plan itself requires more configuration:
<matchingOperation planFileName="../engine/trans/party/party_match.comp" class="com.ataccama.nme.dqc.mdu.MduMatchingPlan">
<matchings>
<!-- One matching element for every Matching step in plan. Name here must refer to Repository Name in step. -->
<matching name="nm">
<masterIdColumn>master_id_nm</masterIdColumn>
<matchRuleNameColumn>match_rule_name_nm</matchRuleNameColumn>
<matchQualityColumn>match_quality_nm</matchQualityColumn>
<matchRelatedIdColumn>match_related_id_nm</matchRelatedIdColumn>
<isolateFlagColumn>isolate_flag_nm</isolateFlagColumn>
<rematchFlagColumn>rematch_flag_nm</rematchFlagColumn>
<proposals enabled="true"/>
</matching>
<matching name="sin">
<masterIdColumn>master_id_sin</masterIdColumn>
<matchRuleNameColumn>match_rule_name_sin</matchRuleNameColumn>
<matchQualityColumn>match_quality_sin</matchQualityColumn>
<matchRelatedIdColumn>match_related_id_sin</matchRelatedIdColumn>
<isolateFlagColumn>isolate_flag_sin</isolateFlagColumn>
<rematchFlagColumn>rematch_flag_sin</rematchFlagColumn>
<proposals enabled="false"/>
</matching>
</matchings>
</matchingOperation>
Every Matching step must have a corresponding definition in the matching operation in the model configuration file. The name of the matching must refer to the step using its Repository Name. Key column bindings must be configured for every matching:
Name in the Model Configuration File | Name in the Matching Step | Type | Default column name | Description |
---|---|---|---|---|
masterIdColumn |
Master Id Column |
LONG_INTEGER |
master_id |
Where the master group identifier is stored. |
matchRuleNameColumn |
Match Rule Name Column |
STRING, 100 |
match_rule_name |
Name of the matching rule that matched this record to another or null if the record is the only one in the master group. |
matchQualityColumn |
Match Quality Column |
FLOAT |
match_quality |
Quality of match, 0 being the best, computed from matching tests (such as editDistance), if a simple Boolean expression is used, 0 represents a match. |
matchRelatedIdColumn |
Match Related Id Column |
LONG_INTEGER |
match_related_id |
If a record is matched to another record, its instance record identifier is stored here. |
isolateFlagColumn |
Isolate Flag |
BOOLEAN |
isolate_flag |
True if a record is isolated from matching with other records. |
rematchFlagColumn |
Rematch |
BOOLEAN |
rematch_flag |
Not persisted. If true, the record is rematched and might change its master group identifier. |
rematchIfChangedColumns |
rematch If Changed Columns |
BOOLEAN |
/ |
Defines columns ( |
proposals |
Proposals |
BOOLEAN |
/ |
If enabled, another integration output in the matching plan will be generated to accept the pairs of proposed records (based on the defined rules). |
These columns must not be present in the entity as the engine adds them automatically with correct types and origin.
If there is only one Matching step, default values are used and no configuration is necessary.
Default matching name (and Repository Name) is k
.
Matching proposals and AI Matching
Defines entities that are used for working with matching proposals and AI Matching.
<proposals enabled="true">
<defaultMasterEntityView view="masters" name="party"/>
<entitiesForAiMatching>
<entityForAiMatching view="norm" name="person"/>
<entityForAiMatching view="norm" name="organization"/>
</entitiesForAiMatching>
</proposals>
Parameters:
-
proposals
:-
enabled
: enables matching proposals.
-
-
defaultMasterEntity
: If specified, proposals will be accessible in MDM Web App. The values should correspond with matching operation.-
view
: name of the master layer. -
name
: name of the entity.
-
-
entitiesForAiMatching
: optional list of entities for AI Matching.-
view
: name of the master layer. -
name
: name of the entity.
-
Aggregate Operation
An optional operation that defines aggregation logic for a given entity, which is represented by a ONE plan.
<aggregatingOperation class="com.ataccama.nme.dqc.operations.AggregatingPlan" planFileName="../engine/trans/party/party_aggregate.comp" groupingColumn="master_id" />
Parameters:
-
groupingColumn
: specifies the grouping column used to collect all needed records into a group of records being processed. Column type has to be one of LONG_INT, INTEGER, LONG, STRING.
Source System Definition
Defines a list of source systems which are connected to the MDM engine.
<sourceSystems>
<system name="crm">
<instanceMappings>
<originId name="crm_customer#Party" entity="party"/>
<originId name="crm_address#Address" entity="address"/>
</instanceMappings>
</system>
<system name="life">
<instanceMappings>
<originId name="life_contract#Party" entity="party"/>
<originId name="life_contract#Address" entity="address"/>
</instanceMappings>
</system>
</sourceSystems>
Parameters:
-
<system name>
: the source system name. It is available in eng_source_system column in ONE plans. -
<instanceMappings>
-
name
: a descriptive name that provides text information about record origin. It is available as eng_origin attribute in ONE plans.-
<source_system>_<source_entity>#<target_MDC_entity>
, for example,life_contract#Party
. -
<source_company>##<source_system>##<source_entity>
, for example,ATACCAMA##LIFE##contracta
.
-
-
Master Layers
On master layers, there are two types of entities: master entities (<entities>
) and external entities (<externalEntities>
).
Master Entities
Defines master entities and their behavior on the master layer. A master entity can behave in one of the defined ways: consolidation, authoring, or mixed (both options enabled at the same time).
-
Consolidation: records are read from external systems, preprocessed on instance layer and a master record is created based on the merge operation.
-
Authoring: records can be created on the master layer without having an instance layer available.
-
Mixed: the combination of consolidation and authoring.
Depending on the master entity behavior type, specific entity settings (that is, attributes and operations that can be performed) vary.
<entities>
<entity class="com.ataccama.nme.engine.model.PersistentMasterEntity" dedicatedIds="false" name="party">
<columns>
..
</columns>
<relationships/>
<consolidation instanceEntity="party" groupingColumn="master_id">
<overridable data="true" activity="true"/>
<mergingOperation>
..
</mergingOperation>
<overrides>
..
</overrides>
</consolidation>
<authoring>
<masterMatching>
..
</masterMatching>
</authoring>
</entity>
<entity class="com.ataccama.nme.engine.model.VirtualInstanceEntity" instanceEntity="party" name="party_instance" >
<relationships>
..
</relationships>
</entity>
</entities>
On the master layer, entities can be of two class
types:
-
PersistentMasterEntity
: defines a physical table to be stored in a database. -
VirtualInstanceEntity
: defines a virtual entity which is required to create a master-to-instance relationship.
Parameters:
-
name
: the name of the entity. -
usage
(only for PersistentMasterEntity): specifies usage of the entity.-
DATA_ENTITY
: a default, standard data entity. -
DICTIONARY
: an entity with a relatively small number of records used for storing dictionary values, no indices will be generated on relationships from data entity to dictionary.
-
-
dedicatedIds
(only for PersistentMasterEntity): specifies whether a certain entity uses its own sequence for themaster_id
and not from the shared sequence across the whole MDM solution. This is especially useful when there are limitations on the consuming system. -
<consolidation>
: enables consolidated records.-
instanceEntity
: a corresponding entity on the instance layer. -
groupingColumn
(only for PersistentMasterEntity): specifies the grouping column used to create a golden record. Column has to be of LONG_INT data type. Default column name ismaster_id
.
-
-
<authoring>
: enables authored records.
Column
Column definition for master entities is done this way:
<columns>
<column name="cmo_first_name" type="string" size="100"/>
<column name="cmo_last_name" type="string" size="100"/>
</columns>
Parameters:
-
name
: the name of the column. -
origin
(column definition context - instance columns only): defines a place in a plan where the column appears.-
source
: a source value, appears in cleanse plan input. -
clean
: cleansed, appears as a cleanse plan output. -
match
: a match value, appears as a match plan output. -
aggregate
: an aggregation output, appears as an aggregation plan output
-
-
type
: the data type. -
size
: column length (used as a database column size) that is mandatory forstring
data type. -
overridable
: data overrides on column level.
Copy Columns
<column name="cmo_contact_comp" origin="copy_merge" type="string" size="4000">
<valueSource columnName="cmo_contact_comp" relationshipName="contacts">
<aggregation class="com.ataccama.nme.engine.model.ConcatenateDistinct" maxLength="4000" separator="^~"/>
</valueSource>
</column>
Parameters:
-
name
,type
andsize
: see the column definition example in this section. -
origin
(copy column context): defines which columns are taken and a place, where the copied columns appear.-
copy_merge
: source column added to a matching plan input.
-
-
valueSource
-
columnName
: source column for the copy. -
relationshipName
: relationship definition and direction to be used for the copy.
-
-
aggregation class
:-
ConcatenateDistinct
: used to perform child to parent copy column. Because the 1:M relationship, there is a concatenate distinct functionality to store all the values related to a single attribute on parent entity.-
separator
: values separator definition for concatenation. -
maxLength
: length of concatenated outcome.
-
-
MinValue
: used to perform child to parent copy column. It takes minimum value from all related entities (because of 1:M relationship). -
MaxValue
: used to perform child to parent copy column. It takes maximum value from all related entities (because of 1:M relationship). -
FirstValue
: used to perform parent to child copy column. It might be used for child to parent copy column, but there is no deterministic selection.
-
-
isFk
: (Boolean) if true, this copy column will be used as the foreign key for this relationship. -
expression
: the expression that is used to fill copy columns. -
<filterExpression>
: (boolean) optional; provides an ability determine when column copying should be performed using a filter expression. There are two pseudoinputs available:source
(origin of copy) andtarget
to construct the expression.
Master Relationship
Relationship definition is always a part of a child entity definition and it points to a parent entity.
<relationships>
<rel foreignKeyColumn="master_id" name="master" parentColumn="id" parentEntity="contact" reverseName="instances"/>
<rel foreignKeyColumn="src_type" name="fk_dic_type_contact" parentColumn="source_code" parentEntity="rd_cont_type_trans" reverseName="rev_fk_dic_type_contact">
<type class="com.ataccama.nme.engine.model.SameSystemExtendedRelationship">
<parentSystemColumnName>source_system</parentSystemColumnName>
</type>
</rel>
<rel foreignKeyColumn="std_type" name="std_type_cont_type_master_details" parentColumn="master_code" parentEntity="rd_cont_type" reverseName="master_codes_std_type_cont_type_contact"/>
</relationships>
Parameters:
-
rel
: the main relationship definition with the following attributes.-
name
: the name of the relationship; used as a reference name for copy columns. -
reverseName
: reverse relationship name; used as a reference name for copy columns. -
parentEntity
: the name of the parent entity. -
parentColumn
: the primary key column of the parent entity. -
foreignKeyColumn
: the foreign key column of the child entity.
-
-
type
: type of relationship defined in theclass
attribute.-
CrossSystemRelationship
: has to be used for a master-to-master relationship and master-to-instance relationship. It might be used betweenVirtualInstanceEntity
entities especially for code books and dictionaries. If<type>
is not defined,CrossSystemRelationship
is the default behavior of the relationship where one of the entities isPersistentMasterEntity
. -
SameSystemRelationship
: can only be used betweenVirtualInstanceEntity
entities. Iftype
is not defined,SameSystemRelationship
is the default behavior of the relationship betweenVirtualInstanceEntity
entities. -
SameSystemExtendedRelationship
: can only be used betweenVirtualInstanceEntity
entities.
-
For` PersistentMasterEntity` relatedRecordBehavior
can be defined.
It specifies how a change in the parent record affects its child records.
For more information, see Mixed Style in MDM, section Related Record Behavior.
-
onAuthoredParentDelete
: specifies options that can be enabled when an authored parent record is deleted.-
deleteAuthoredRecord
-
followSurvivor
-
loseRecordOwnership
-
-
onParentLoseOwnership
: specifies options that can be enabled when an authored parent loses its authored status.-
deleteAuthoredRecord
-
loseRecordOwnership
-
-
onConsolidatedParentDelete
: specifies options that can be enabled when a consolidated parent record is deleted due to matching.-
allInstancesDeleted
-
followerExist
-
followerMissing
-
<relationships>
<rel foreignKeyColumn="party_id" name="party" parentColumn="id" parentEntity="party" reverseName="notes">
<relatedRecordBehavior>
<onAuthoredParentDelete deleteAuthoredRecord="true" followSurvivor="true" loseRecordOwnership="true"/>
<onParentLoseOwnership deleteAuthoredRecord="false" loseRecordOwnership="false"/>
<onConsolidatedParentDelete allInstancesDeleted="KEEP" followerExist="FOLLOW" followerMissing="KEEP"/>
</relatedRecordBehavior>
</rel>
</relationships>
Validate Operation
Validation layer for the master data represented by a ONE plan.
<validatingOperation class="com.ataccama.nme.dqc.operations.ValidatingPlan" planFileName="../engine/trans/party/masters_party_validate.comp"/>
Operations and other settings available vary depending on the behavior of master entity: consolidation, authored, or mixed.
Consolidation Settings
Consolidation settings that can be defined are as follows:
<consolidation groupingColumn="master_id" instanceEntity="party">
<overridable activity="true" data="true"/>
<mergingOperation class="com.ataccama.nme.dqc.operations.MergingPlan" customActivityTracking="false" planFileName="../engine/trans/party/masters_party_merge.comp"/>
<overrides>
<enable>true</enable>
<lifeCycle class="com.ataccama.nme.engine.model.lifecycle.RemoveOverrides">
<columns>
<column name="cmo_first_name" ifSame="true"/>
<column name="cmo_last_name" ifSame="true"/>
<column name="cmo_birth_date" ifSame="true"/>
</columns>
</lifeCycle>
</overrides>
</consolidation>
Parameters:
-
<overridable>
: defines ifdata
andactivity
overrides are enabled or disabled.
Merge Operation
Operation defines a link between an MDM plan representing a merge operation and master entity definition.
Parameters:
-
customActivityTracking
(Boolean): use custom rules for activity calculation, that is, overwriteeng_active
value in the merge plan.false
by default. -
<recordFilterExpression>
(Boolean, optional): enables to apply a filter to the merge plan record input. The expression cannot contain thesequence()
function.
Overrides Lifecycle
The parameter defines if overrides are removed automatically and upon which conditions.
The <enable>
parameter enables overrides:
-
true
: overrides are considered 'temporary' and are removed automatically if certain conditions are met. -
false
: overrides are considered 'permanent' and can only be removed manually.
Parameters:
-
class
: when overrides are enabled, the class isRemoveOverrides
. -
<column>
:-
name
: the name of the column whose values are compared to the incoming values. -
types of conditions under which overrides are removed automatically:
-
ifSame
: if the override value is equal to the incoming value (regardless of the original value). -
ifChanged
: if the original value is different to the incoming value (regardless of the override value). -
expression
: if an expression condition is satisfied.
-
-
Authored Settings
Authored settings that can be defined are as follows:
<authoring>
<masterMatching class="com.ataccama.nme.engine.model.NmeMasterMatchingWithConsolidation">
<matchingColumns>
<column name="mat_party_type" expression="upper(cmo_type)"/>
<column name="mat_gender" expression="upper(cmo_gender)"/>
<column name="mat_first_name" expression="upper(squeezeSpaces(cmo_first_name))"/>
<column name="mat_last_name" expression="upper(squeezeSpaces(cmo_last_name))"/>
<column name="mat_full_name" expression="upper(squeezeSpaces(cmo_first_name + ' ' + cmo_last_name))"/>
<column name="mat_initials" expression="upper(squeezeSpaces(sortWords(set.mapExp(cmo_first_name + ' ' + cmo_last_name, ' ', (x) { left(x,1) }))))"/>
<column name="mat_person_id" expression="upper(squeezeSpaces(cmo_pid))"/>
<column name="mat_birth_date" expression="cmo_birth_date"/>
<column name="mat_contact_set" expression="upper(cmo_contact_comp)"/>
</matchingColumns>
</masterMatching>
<lifeCycle class="com.ataccama.nme.engine.model.lifecycle.LoseOwnership">
<conditions>
<condition class="com.ataccama.nme.engine.model.lifecycle.ExpressionCondition">
<expression>current.cmo_pid = new.cmo_pid and new.cmo_pid is not null</expression>
</condition>
<condition class="com.ataccama.nme.engine.model.lifecycle.ColumnsEqualCondition">
<columns>
<column>cmo_first_name</column>
<column>cmo_last_name</column>
</columns>
</condition>
</conditions>
</lifeCycle>
</authoring>
Master Match Operation
This operation defines the way instances are matched to authored master records. For more information, see Mixed Style in MDM, section Master Matching.
Parameters:
-
<matchingColumns>
: defines instance entity columns that need to be matched with master entity columns.-
name
: name of the source column. -
expression
: matching expression.
-
Lifecycle
The parameter defines changes that are made to the record based on set conditions.
Parameters:
-
<conditions>
: condition that need to be met for entity to lose ownership. All conditions listed in this section must be fulfilled. -
class
: defines the type of the condition that needs to be met.-
ExpressionCondition
: condition based on an expression.-
<expression>
: specifies the expression usingcurrent
andnew
dot source inputs.
-
-
ColumnsEqualCondition
: condition based on equality of columns.-
<column>
: defines columns that need to be the same in current and new records for the condition to be met.
-
-
External Entities
Defines external entities and their behavior on the master layer.
<entities>
<externalEntities>
<externalEntity name="audit">
<columns>
..
</columns>
<relationships>
..
</relationships>
</externalEntity>
</externalEntities>
Column
Column definition for master entities is done this way:
<columns>
<column name="event_type" type="string" size="50"/>
<column name="event_timestamp" type="datetime"/>
<column name="event_user" type="string" size="50"/>
</columns>
Parameters:
-
name
: the name of the column -
type
: data type -
size
: column length (used as a database column size) that is mandatory forstring
data type
External Relationship
The external relationship specifies the source of the external data and the way the external entity is mapped to other entities.
<relationships>
<rel name="insured_events" parentEntity="party" reverseName="insured_events">
<dataProvider class="com.ataccama.nme.dqc.external.ExternalDataProviderConfig" planFileName="../engine/external/masters_insured_event/insured_events.comp">
<columns>
<column name="cmo_pid"/>
</columns>
<relationships/>
</dataProvider>
</rel>
</relationships>
nme-model.gen.xml Example
The following configuration does not show all configuration possibilities. |
<?xml version='1.0' encoding='UTF-8'?>
<model>
<instanceLayer>
<entities>
<entity name="party">
<columns>
<column name="src_name" origin="source" type="string" size="100"/>
<column name="std_name" origin="clean" type="string" size="100"/>
<column name="exp_name" origin="clean" type="string" size="500"/>
<column name="sco_name" origin="clean" type="integer" size="10"/>
<column name="std_first_name" origin="clean" type="string" size="100"/>
<column name="std_last_name" origin="clean" type="string" size="100"/>
<column name="src_ssn" origin="source" type="string" size="100"/>
<column name="std_ssn" origin="clean" type="string" size="100"/>
<column name="exp_ssn" origin="clean" type="string" size="500"/>
<column name="sco_ssn" origin="clean" type="integer" size="10"/>
<column name="uni_can_id" origin="match" type="long" size="20"/>
<column name="uni_rule_name" origin="match" type="string" size="100"/>
<column name="uni_role" origin="match" type="string" size="10"/>
<column name="add_std_zip" origin="copy_clean" type="string" size="1000">
<valueSource columnName="std_zip" relationshipName="pat_add">
<aggregation maxLength="1000" class="com.ataccama.nme.engine.model.ConcatenateDistinct" separator="~"/>
</valueSource>
</column>
<column name="agg_group_count" origin="aggregate" type="string" size="1000"/>
</columns>
<relationships/>
<matchingOperation class="com.ataccama.nme.dqc.operations.MatchingPlan" planFileName="../trans/party/party_match.comp"/>
<cleansingOperation class="com.ataccama.nme.dqc.operations.CleansingPlan" planFileName="../trans/party/party_clean.comp"/>
<aggregatingOperation class="com.ataccama.nme.dqc.operations.AggregatingPlan" groupingColumn="master_id" planFileName="../engine/trans/party/party_aggregate.comp"/>
</entity>
<entity name="address">
<columns>
<column name="pty_source_id" origin="source" type="string" size="100"/>
<column name="src_street" origin="source" type="string" size="100"/>
<column name="std_street" origin="clean" type="string" size="100"/>
<column name="src_city" origin="source" type="string" size="100"/>
<column name="std_city" origin="clean" type="string" size="100"/>
<column name="src_state" origin="source" type="string" size="100"/>
<column name="std_state" origin="clean" type="string" size="100"/>
<column name="src_zip" origin="source" type="string" size="100"/>
<column name="std_zip" origin="clean" type="string" size="100"/>
<column name="exp_address" origin="clean" type="string" size="500"/>
<column name="sco_address" origin="clean" type="integer" size="10"/>
<column name="pty_master_id" origin="copy_clean" type="long_int" size="30">
<valueSource columnName="master_id" relationshipName="party">
<aggregation class="com.ataccama.nme.engine.model.FirstValue"/>
</valueSource>
</column>
</columns>
<relationships>
<rel reverseName="addresses" parentEntity="party" parentColumn="source_id" name="party" foreignKeyColumn="pty_source_id">
<type class="com.ataccama.nme.engine.model.SameSystemRelationship"/>
</rel>
</relationships>
<matchingOperation class="com.ataccama.nme.dqc.operations.MatchingPlan" planFileName="../trans/address/address_match.comp"/>
<cleansingOperation class="com.ataccama.nme.dqc.operations.CleansingPlan" planFileName="../trans/address/address_clean.comp"/>
</entity>
</entities>
<sourceSystems>
<system name="crm">
<instanceMappings>
<originId name="crm_customer#Party" entity="party" code="10"/>
<originId name="crm_address#Address" entity="address" code="11"/>
</instanceMappings>
</system>
<system name="life">
<instanceMappings>
<originId name="life_contract#Party" entity="party" code="20"/>
<originId name="life_contract#Address" entity="address" code="21"/>
</instanceMappings>
</system>
</sourceSystems>
</instanceLayer>
<masterLayers>
<masterLayer name="Master">
<entities>
<entity name="party" class="com.ataccama.nme.engine.model.PersistentMasterEntity">
<columns>
<column name="cmo_first_name" type="string" size="100"/>
<column name="cmo_last_name" type="string" size="100"/>
<column name="cmo_id_card" type="string" size="30"/>
<column name="cmo_ssn" type="string" size="30"/>
</columns>
<relationships/>
<consolidation instanceEntity="party" groupingColumn="master_id">
<mergingOperation customActivityTracking="false" class="com.ataccama.nme.dqc.operations.MergingPlan" planFileName="../trans/party/Master_party_merge.comp"/>
<recordFilterExpression>eng_active=true</recordFilterExpression>
</mergingOperation>
</consolidation>
<authoring/>
</entity>
<entity name="address" class="com.ataccama.nme.engine.model.PersistentMasterEntity">
<columns>
<column name="pty_id" type="long_int" size="30"/>
<column name="cmo_street" type="string" size="100"/>
<column name="cmo_city" type="string" size="100"/>
<column name="cmo_state" type="string" size="100"/>
<column name="cmo_zip" type="string" size="100"/>
</columns>
<relationships>
<rel reverseName="addresses" parentEntity="party" parentColumn="id" name="party" foreignKeyColumn="pty_id"/>
</relationships>
<consolidation instanceEntity="address" groupingColumn="master_id">
<mergingOperation customActivityTracking="false" class="com.ataccama.nme.dqc.operations.MergingPlan" planFileName="../trans/party/Master_address_merge.comp"/>
<recordFilterExpression>eng_active=true</recordFilterExpression>
</mergingOperation>
</consolidation>
<authoring/>
</entity>
<entity name="party_instance" instanceEntity="party" class="com.ataccama.nme.engine.model.VirtualInstanceEntity" >
<relationships>
<rel parentEntity="party" parentColumn="id" name="fk_party_inst" foreignKeyColumn="master_id"/>
</relationships>
</entity>
<externalEntities>
<externalEntity name="audit_drafts">
<columns>
<column name="event_type" type="string" size="50"/>
<column name="event_timestamp" type="datetime"/>
<column name="event_user" type="string" size="50"/>
<column name="comment" type="string"/>
</columns>
<relationships>
<rel name="party_has_drafts" parentEntity="party" reverseName="party_has_drafts">
<dataProvider class="com.ataccama.nme.dqc.external.ExternalDataProviderConfig" planFileName="../engine/external/provider_audit_drafts/party_has_drafts.comp">
<columns>
<column name="id"/>
</columns>
<relationships/>
</dataProvider>
</rel>
</relationships>
</externalEntity>
</entities>
</masterLayer>
</masterLayers>
<eventHandlersConfigFile>nme-event_handler.gen.xml</eventHandlersConfigFile>
</model>
Was this page useful?