Matching Step

A partition is the first division of records in the matching step. There may be one or more partitions. A partition can contain all records in the data flow. Records from partition A can only be matched to records in Partition A. They are evaluated from top to bottom and if no expression is given for it, TRUE is implied. If a use-case requires the use of the TRUE condition (matching without partitioning), that partition should be last in the list of key rules.

Key rules are used to make key groups. Key groups are divisions within partitions. If a record links two groups, they are unified and assigned the same master ID. Key Groups consist of records whose Component columns match.

Matching rules are the reason for group creation. Each Key Group can have multiple Matching Rules. Matching Groups are made by Matching Rules. One Matching Group can be made of several Matching Rules:

Tests are quantifiable checks to determine whether two records match. They are defined in the Match Functions section of the Matching Step. Match Functions use 'x' and 'y' to represent the attributes being tested. 'x' is the tested attribute from one record while 'y' is the attribute from the other record. Tests work in conjunction with Matching Rules. For 2 records to match, they must be connected by a matching rule and pass all tests.

During matching, the records in a matching group are constantly rearranged due to the addition or deletion of records, as well as change of record keys or other attributes. The Matching Step tries to keep already used group IDs as much as possible. Only one record of each group is called the Merge survivor, or in other words, the carrier of the group ID - this is the ID Keeper Record.

Constraint rules can be used to prevent records from being matched to the same Matching Group (if a certain condition is met) which would otherwise be matched based on Key Rules and Matching Rules.

When an attribute is used as a Constraint Rule, all records in one Matching Group must have the same value in that attribute (or null values in that attribute). In other words, records that differ in the Constraint Rule value are divided out into different Matching Groups regardless of the Key and Matching Rules.

Records with null Constraint Rule value are match to any other record with null or non-null Constraint Rule value if Key Rules and Matching Rules tests and conditions are satisfied.

Records that match based on the configured proposal rules are sent through the matching step’s optional output. They are often integrated with MDM Web App for use as records to be manually matched by data stewards.

Isolate flag is a binary column, which if True means the record will not match with any others in any case.

Pivot and Candidate are the names given to records while they are in the matching process. Accessing a particular attribute in the record can be done with pivot.record_name or candidate.record_name.

Standalone mode is only needed when running the Matching Step outside of an MDM project (that is, in a plan file that is not an entity_merge.comp component).

Data enters the Matching Step and is divided into user-defined Partitions. Partitions allow user to split the input records into different groups of records which should not be matched together: typically for splitting a Party entity into records representing persons and companies. A record can only be matched to another record if both records are in the same partition.

Records are put in partitions sequentially:

Within a partition, there are one or many Key Rules. Key rules are made up of Key Components, which are references to fields in the input data. If records have matching values in their Key Components, they are put into the same Key Rule group.

Records are put in the Key Rule groups sequentially:

The first record in the first Key Rule group becomes the pivot record. The second becomes the candidate record. The pivot and candidate roles are used by Matching Rules to match records together.

The Matching Rule used to match a pivot and candidate together is checked sequentially:

Once a pivot and candidate have checked all Matching Rules, the candidate role is changed to the next record in the Key Rule group. Finally, Proposal Rules are run on the leftover, unmatched records. These proposed matches exit via the Matching Step’s Proposals output.

In the left navigation menu, select Partitions. Define your partitions expressions: records that match one expression are put into one partition. If only a single partition is required, you can make one partition whose expression is true. The order of the partitions matters - if a record can go into many partitions, it goes into the first eligible candidate in the list. Leftover records are not matched.

Key Rules are stored inside each partition. To configure key rules, define the following: * Name: The name of the rule. * No Key Cond: If a record follows this condition, it is not put into a key rule group. * Components: The fields in the data that are included into the rule. Records with identical key rule components are put into the same key rule group. Behave as Set: If checked, the component value is assumed as a list of string values. If two sets have at least one intersecting value, then they are considered identical (for the purpose of key group components). Separator: Characters used as word separator when Behave as Set is checked. The default separator is one space. * *Expression: Expression calculating one key component value. Non-string values are converted into string. * No Empty Component: If checked, all components have to be valid (non-null) to conform a valid key. If not checked and at least one component is non-null, the key is valid and will be used. * Matching Rules: It is possible to configure one or many matching rules within each key rule. If no matching rules are defined for a key rule, then records put into that key rule are all considered matched, and all receive the same master_id. The order of key rules matters: if a record can go into many key rule groups, it goes into the first eligible candidate in the list. To know more about configuring matching rules, see [Matching rules].

Matching rules determine which records are finally matched together by the step. The order of matching rules matters: if a record can go into many matching rule groups, it goes into the first eligible candidate in your list.

To configure matching rules, define the following:

Proposal rules are configured in a similar way to matching rules, however, they exit the proposals output of the Matching Step.