If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Exploring MDM Features

Welcome to the Master Data Management tutorial. In the following chapters you will learn more about MDM and its configuration options demonstrated using a fully functional sample solution. The sample solution is based on a fictional customer scenario, which emulates real customer issues. Below you will find a walk-through of the sample solution, which gives an overview of the main MDM features.

Prerequisites:

Knowledge of working with projects (see Projects).
Basic knowledge of the Matching step (see Matching Step).
Basic knowledge of Entity Relationship Modeling.

Background Story

A large Canadian insurance company with branches across Canada is looking for a new MDM tool in order to help the company improve data quality and achieve its data governance goals. The company started to recognize the need for better data after observing record duplication and bearing the risk of client misidentification due to storing client and contract data in multiple databases and systems.

Problems Identified

Client data stored in multiple systems - different versions of the truth (CRM system, LIFE legacy system, and INVEST system containing corporate client data).
Low data quality of existing records and those entering the sources systems (client addresses often incomplete or misspelled).

Sample Solution Specifications

Supports MDM data standardization processes - cleansing, matching and mastering of records.
Processes data from all company systems and provides the Single Customer View.
Verifies the quality of new input data (Data Quality Firewall).
Offers online data providing web services and batch interfaces.
Provide a view of master data via a web application.

MDM Sample Solution Walk-through

The built-in sample solution in the MDM product is based on the General MDM Model Project and shows a Customer Data Integration (CDI) project. The solution is based on the story and problems identified in the model company. The sample solution is a fully functioning example of integrating three different source systems into an MDM Hub with a pre-defined MDM logical model, pre-configured cleansing and matching components, and both batch and online output interfaces set up. The solution uses an embedded Derby database. No additional changes are required to run the example out of the box.

Open a New Sample Project

Open the ONE Desktop and navigate to File > New > Model Project, or select the New Model Project Wizard in the Model Explorer Panel.

Select Use template > General MDM project - CDI example and select Finish.

This will open a pre-configured project to solve the problems and meet the company requirements described above.

General MDM Project Overview

The application configuration is always provided in one encapsulating solution - called a project. The model project configuration contains several areas (as shown below).

For the purpose of this tutorial, all nodes of the project have been fully configured, so you will not need to perform any further configuration. The node descriptions below are provided to explain the MDM project configuration in MDM. To start the overview of features, skip to the next section.

Preferences. Contains global system settings like Ignored columns, Record deactivation strategy, AME model validation schema (forcing DB length naming restrictions).
Connected Systems. Holds configuration of systems that the MDM Hub will process in the batch and/or online mode. Each connected system has its own source model that is mapped to the MDM Hub canonical model. Furthermore, each system can have different load operations configured. Of course, if a system provides only online web service interfaces, no batch load operations are defined.
Logical Model. Used to set up almost all "real-world" objects in the MDM Hub, which are then generated from the metadata description of all entities, relationships, and settings. It consists of several models describing particular data layers. The number of layers depends on project design and requirements.
Output Interfaces. Used to configure batch exports from the MDM Hub (exports are fully processed solely by the MDM Engine) and Data Changes Event Handler and Publisher.
Services. Defines out-of-the-box native services to be used, together with endpoints where native services are available.
Streaming. This is where you define stream consumers, which process messages from JMS queues and transform them into input data for MDM processing.
GUI Configuration. Everything related to general GUI MDM Web App settings: search definitions, validations, workflows, and data type formatting settings.
Transformations. A transformation in the MDM Hub context is an auto-generated plan file that has to be completed by developers, who provide some application logic. The plan is used to traverse two neighboring MDM layers. Between layers L1 and L2 is the cleansing plan for each entity in the layer. The transformation in general is called Cleanse or Cleansing. Between layers L2 and L3 are matching plans and between L3 and L4 are merge plans. In different layers it is common to use prefixed attributes based on the function and phase in data processing (for example, src_, std_, exp_, sco_, …).
Preview. There are several views displayed in the Preview node: Canonical Model, Physical DB model, and MDM Web App GUI labels. Editing is not supported; use this item only for browsing.
Documentation. Automatically generated project documentation displays content of the current MDM configuration. It lists all entities, attributes, relationships. The documentation content is automatically recompiled with model changes.
Advanced Features. Defines advanced operations like entity reprocessing and execution status monitoring persistence export.
Files. Represents the project configuration files. It contains all generated plans, components, configurations, data, lookups, and more.

Cleanse, Match, and Merge

Now we want to process data in the initial load from the three source systems: CRM, Life, and Invest. This can be done from the administration Web Console provided by the MDM server at the localhost:8051/admin endpoint. After opening the console, click the Load Operations link under MD Interfaces as seen in the following figure. To start processing, click Run next to the full load operations for the three systems, plus reference data: crm_full, life___full, invest_full, and hub_reference_data_full. MDM will start processing data, displaying the RUNNING or WAITING status.

The input source data for the three systems is embedded in the Files > data > in folder. The data set is limited to a few records demonstrating various cleansing, matching, and mastering examples. The CRM system is loaded from the XML extract; however, the data is also available in the CSV format. The Life and Invest systems are loaded from a set of CSV files.

Check Batch Processing Results

For active monitoring of all MDM batch processes, go to Execution Status under MD Process Monitoring. The Execution Status screen displays the list of running or finished operations (loads or exports).

Click the Id or Name of the operation to see its details: the phases of the batch load and tasks for each phase.

MDM performs batch loads in these phases:

Data Acquisition
Change Detection
Master Data Consolidation
Committing

Events

The load operation produces Data Changes (also called events or event data changes). An event is produced when a change in an attribute in one record is detected. MDM provides event Publishers (see note for details). The event output structure can be configured using Transformers - both custom and predefined. An event can consist of MDM engine metadata and entity attributes with new and old values (optional). Simple XML Transformer sample event output is displayed below - something that you will see after running batch load operations. This functionality is configured in the Output Interfaces > Event Handler Settings node.

Transformer - configuration which allows converting an MDM event format to an arbitrary string (custom transformation) or predefined XML format using Simple XML Transformer.

Publisher - configuration which allows propagating a transformed MDM event to the events consumer system, for example, JMS Publisher, SQL Publisher, HTTP SOAP Publisher.

Data Change Event Sample Record

party(layer: instance):
<event>
    <metadata>
        <id>2139</id>
        <event_type>INSERT</event_type>
        <entity>party</entity>
        <activation_date>2014-08-05 17:04:12</activation_date>
        <activation_tid>2100</activation_tid>
        <creation_date>2014-08-05 17:04:12</creation_date>
        <creation_tid>2100</creation_tid>
        <last_update_date>2014-08-05 17:04:12</last_update_date>
        <last_update_tid>2100</last_update_tid>
        <last_source_update_date>2014-08-05 17:04:12</last_source_update_date>
        <last_source_update_tid>2100</last_source_update_tid>
        <layer>instance</layer>
        <origin>life#party#party</origin>
        <source_system>life</source_system>
    </metadata>
    <attributes>
        <source_id>312</source_id>
        <master_id>50</master_id>
        <src_company_name>SILVERGREEN</src_company_name>
        <src_business_number>653216317 RD 0010</src_business_number>
        <eng_active>true</eng_active>
    </attributes>
    <oldAttributes>
        <source_id/>
        <master_id/>
        <src_company_name/>
        <src_business_number/>
        <eng_active>false</eng_active>
    </oldAttributes>
</event>

Single Customer View with MDM Web App

MDM propagates all the data to the MDM web application in order to allow data browsing and correction in screens defined in the model project.

Open localhost:8050 in your web browser and log in with admin as the name and password.

See Exploring MDM Web App for the overview of the main use cases in the MDM Web App.

DQ Firewall and Online Services

In the sample solution, Data Quality Firewall is realized using a pre-configured email-validating DQC plan, which is published as an online service.

To see how it works, navigate to Files > test > ws_test > soapServices in ONE Desktop _Model Explorer view and open dqfEmail.wstest, which will open the screen below.

When you click Send Request, the MDM server will receive a SOAP request and process it. The result is mapped from the email-cleansing plan, which stays behind the service, to the SOAP format and returned.

In the example below, the input email string "info [ at ] example .com" is processed, and the component suggests the best parsed email as "info@example.com". The other output attributes are the "explanation" and "score" columns.

MDM allows using the free Ataccama cleansing and validation components, which can be used as part of validation plans.

DQ Firewall request and response examples

Native Services

Besides the online services published from plans, MDM provides out-of-the-box native services based on the entity model and configuration. The sample solution includes such services in Files > test > ws_test. Notable examples (in the complex_read subfolder) include:

genericmastersTraversal.wstest - example of a powerful Traversal native service, which retrieves data from an entity and its related entities through a relationship defined in the model.
identifyParty.wstest - example of a unique service providing information about the hypothetical matching result calculated using the same cleansing and matching rules as in the batch load operations. In other words, it answers the question "What would be the master record of the record sent in the WS request?" (see example below)

Identify Web Service

There is an sample web service request provided in the solution. Try it out for yourself and navigate to Files > test > ws_test in MDM Model Explorer view and open identifyParty.wstest, which will open the screen below. By clicking Send Request the MDM server will receive a SOAP request and process it.

In the example below, the input request contains first name ("J"), last name ("Jordan"), and a social insurance number ("163-679-111") processed using the same cleansing and matching & merging components. Because we have already loaded some test data in previous steps, the service will try to match the records as shown in the picture, otherwise it would return an empty response. The request can be extended and more entities or attributes can be added.

ProcessDelta Web Service

There is a sample web service request provided in the solution. Try it out for yourself by navigating to Files > test > ws_test in the MDM Model Explorer view and open processDelta.wstest, which will open the screen below. By clicking Send Request the MDM server will receive a SOAP request and process it.

In the example below, the input request contains new information about a customer from the CRM system with the source ID 1023. MDM will process this input data through cleansing, matching & merging, and commit data to its storage. Afterwards this new information will be available in other web services, MDM Web App, and batch exports. The request can be extended and more entities or attributes can be added.

ProcessDelta request and response example