User Community Service Desk Downloads
If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Migrating Data in MDM

Overview

Migration of data in the MDM engine, using the Migration Export and Migration Load operations, is the most comprehensive way to accommodate complex changes to your MDM solution. This is necessary, for example, when upgrading between major MDM versions, where additional business logic has been added.

The key principle is create a complete snapshot of all relevant data data using the Migration Export operation and when all changes are applied, reload the MDM storage into the now empty database using Migration Load operation.

As part of the migration process, data structure can be changed as needed by transformations. It is also possible to perform initial load of a newly connected system as part of the migration.

You must stop data processing between the export and migration load, otherwise you lose any changes made to the data.
When performing data migration, the stability of all metadata (eng_*date & eng*_tid), id, source_id, previous distinct values, inactive records, and matching IDsis ensured. The list of columns needed depends on migration load operation settings.

Configure the migration export

Use the Migration Export feature to prepare a complete snapshot of your data.

  1. Open your current MDM project.

  2. Open Advanced Features > Migration.

  3. Under Migration Export, select Enable. Select Tasks and draft export and Manual Match Decision Load if you need to migrate tasks and drafts and manual match decisions respectively.

    Migration settings tab

    Included in the Migration Export operation are:

    • All instances.

    • Masters (all master layers and entities where authoring is enabled).

    • Instance and master override tables.

    This means that whichever load you require --Migration Load, Authored Master Records, or both-- Migration Export must be enabled and all will be incorporated into the same plan: migration_export.comp.

    Configuration is generated to nme-batch.gen.xml, where every entity defined generates a dataProvider class.

    History Migration Export is configured separately (see Configure history migration load).

  4. Stop loading data by switching to the read-only mode (for example, by using the switchReadWriteMode native service, see RW-RO Mode Switching Services).

  5. Wait for any running load to finish.

    If VLDB persistence is in use, you need to wait until I_TREG is empty. To see when this is the case, check the Persistence Status section in the Admin Center (when 'number of known transactions' = 0 and 'LTC runnning' = false). For more information, see MD Process Monitoring.

    Persistence status tab
  6. Right-click Migration and select Generate.

    Generating a migration plan
To view the preconfigured migration plan, right-click Migration and select Open Migration Export Plan.

Configure history migration export

To enable export of entities defined in the history plugin (see History Plugin), configure a history export under Output Interfaces > Export Operations. See Complex Export Operation.

History export

Configure the migration load

Configure the Migration Load to be able to load all exported data back to the MDM solution.

MDM assigns all IDs from a single sequence so all IDs are unique across the whole solution.

If you are migrating external data, there could be an issue with duplicate IDs (for example, master party might have the same ID as master address). We recommend ensuring that all IDs are unique before performing the migration load.

  1. Open Advanced Features > Migration.

  2. Select Migration Load composite element. Under Migration Load, keep all options selected and select the required Matching Migration Mode.

    Migration load general settings
    • Preserve and Match (default): All provided group IDs are preserved as they are. New rules are applied and therefore the matching ID is only assigned if the group ID is empty on the migration load input. The relationship is defined across more Connected Systems.

      This is the suggested option for reference data, dictionaries, or manual matching exceptions.

    • Preserve and Assign: Matching is not performed at all. Records without group IDs are considered as having no key, and a unique group ID is assigned to each record.

      We recommend performing reprocess and rematch right after the migration to correctly match the records without previous results (with unique group ID assigned during the migration process).

    • Reload: All provided group IDs are both recomputed and reused, that is, the previous matching result is preserved only if it is confirmed by the new rules.

  3. Add the matching columns you want to load in the Selected Matching Columns section. The default columns (master_id, keeper flag, and isolate flag) are imported if the Load Matching Columns option is selected, and other columns are defined in this section.

  4. In Load Additional Columns, define columns for migration which are not Engine columns, Previous Distinct Vales columns, or Matching columns.

  5. Right-click Migration and select Generate.

  6. Right-click Migration and select Open Migration Load Plan. A preconfigured plan opens.

  7. Add data sources (for example, TXT files from the migration export or another export operation) for each exported entity.

  8. Perform column mappings.

    As a general rule for this step, check the errors in the Integrated Output step and add all missing columns to make sure the plan is valid.
    • Perform the mapping for each entity (attributes should be mostly the same); some default mapping is part of the load plan.

      Mapping columns

Use an additional Alter Format step to be able to add missing columns (solution-dependent) or perform some additional transformations, for example:

  1. eng_source_timestamp (DATETIME): Timestamp from the source system. If not available, eng_last_source_update_date can be used instead.

  2. isolate_flag (BOOLEAN): Stays empty.

Configure overrides migration load

  1. Open Advanced Features > Migration.

  2. Select Migration Load composite element and select the Overrides tab.

    Load overrides tab
To configure overrides migration, Load Id must be selected on the Migration tab. For engine columns to be loaded, Load Engine Columns must be selected on the Migration tab.

Migrate overrides on the instance layer

  1. Enable overrides migration on the Instance Layer by selecting Enable.

  2. To migrate overrides on all available entities, select All Entities.

  3. Otherwise, select individual entities within the Entities section. Double-click the asterisk (*) to begin.

    Press Ctrl+Space to open a list of possible values when adding entities.

The new load plan is generated in Files > engine > load > migration_load_ovr_instance.comp.

Migrate overrides on the master layer

Load Matching Columns must be selected on the Migration tab and the PRESERVE Matching Migration Mode is recommended.
  1. Enable overrides migration on the Instance Layer by selecting Enable.

  2. Select the mater layer for which you want to configure the override migration load.

    Migration is only possible on one master layer at a time. Select the name of the required layer using the dropdown.
  3. To migrate overrides on all available entities, select All Entities.

  4. Otherwise, select individual entities within the Entities section. Double-click the asterisk (*) to begin.

    Press Ctrl+Space to open a list of possible values when adding entities.

The new load plan is generated in Files > engine > load > migration_load_ovr_master.comp.

Configure authored master records migration load

  1. Open Advanced Features > Migration.

    Migration settings tab
  2. Select the Authored Master Records composite element.

    Authored master record settings tab
  3. Enable authored record migration by selecting Enable.

  4. Enable Load Master ID, Duplicate Id, and Load Engine Columns.

  5. Select the Master Layer for which the load plan is relevant. Only one can be selected.

  6. To migrate overrides on all available entities, select All Entities.

  7. Otherwise, select individual entities within the Entities section. Double-click the asterisk (*) to begin.

    Press Ctrl+Space to open a list of possible values when adding entities.
Matching Migration Mode is inherited from the Migration Load configuration.

The new load plan is generated in Files > engine > load > migration_load_authored.comp.

Configure task migration load

  1. Under Advanced Features > Migration > Task Migration Load, select composite element.

    Task Migration Load tab
  2. To enable task migration, select Enable.

  3. To also migrate drafts, select Migrate drafts. This adds drafts integration outputs to the plan and is not selected by default.

  4. To override null or empty input values by task template defaults, select Use task defaults. Do not select this when migrating existing tasks, otherwise you lose all the task details.

The new load plan is generated in Files > engine > load > migration_load_task.comp.

Configure manual match decision load

  1. Under Advanced Features > Migration > Manual Match Decision Load, select composite element.

    Manual Match Decision Load tab
  2. Select Enable to enable match decision load.

  3. To be able to overwrite eng_ columns, select Load Engine Columns.

Configure history migration load

To load history entities defined in the history plugin (see History Plugin), go to Advanced Features > History Migration Load and select Enable.

The history load plan is generated in Files > engine > migration > migration_history_load.comp.

Prepare for the migration load

Perform the following steps before doing the upgrade.

Generic steps

  1. Stop the current MDM server and back up the whole solution.

  2. Ideally, copy the previous schema for the MDM storage and keep the old one as backup. You don’t need to copy data tables, for example, [prefix][entity]_I, [prefix][entity]_[master layer], and all [prefix][entity]_K_[matching repository name].

  3. Delete the [prefix]_STATS table, otherwise the migration load does not start. Data statistics are filled in after the migration is finished.

  4. Check that the following exist (prefix = I_ for Internal):

    1. Tables (updated):

      • AC__EXPORT_REG - If you want to preserve the continuity of incremental instance exports, keep this table as it contains information about the data that was exported previously. Otherwise you can drop it.

        Check that Load Id and Load Engine Columns are selected in the Migration Load settings.
      • I__TASKS

      • I__TASKS_EH

    2. Sequences:

      • I_ID_SEQ

      • I_TASK_SEQ

      • I__TASKS_EH_SEQ

      • Optional sequences:

        • Any processUpsert service sequence I_PU_[entity_name], for example, I_PU_PARTY.

        • MDA sequence I_MDA_SEQ (the MDA part corresponds to the GUI System Name setting in Logical Model > Master Data Layers > [master data layer] > GUI Configuration.

          Indexes are not migrated.
  5. Upload the new configuration and start the new MDM server.

  6. Perform the Migration Load.

Adjusting [prefix]_ID_SEQ

If Load Id or Load Matching Columns are enabled, the following steps have to be performed. Otherwise, the sequence can be dropped and recreated by the MDM engine during the startup procedure.

The id and master_id are now assigned from one sequence, so it might be necessary to preserve and/or adjust the sequence.

  1. Find out the value that is available from the databse sequence [prefix]_ID_SEQ.

    -- Example of how to get the current value from an Oracle database
    SELECT last_number FROM user_sequences WHERE sequence_name = 'A_ID_SEQ';
  2. Find out the highest id or master_id assigned to any entity, divide it by 1000, and round it up.

    To find out the highest number, use an SQL query or profile the exported files.
    The ID sequence in the database has to be increased before the migration to be greater than the maximum (group ID or record ID).
  3. If the database sequence is lower than the highest id used, increase the database sequence to have both values equal. Otherwise, continue to the next step.

    -- Example of how to increase the sequence as required in an Oracle database
    -- Note: Oracle does not let you change the value of a sequence. If you need to change its value, you should recreate the sequence. On the other hand, there is a workaround allowing you to change the value of a sequence without recreating it. For example, if the sequence value is 1000 and you want to set it to 1100, you could do the following:
    -- Increase the sequence increment
    ALTER SEQUENCE A_ID_SEQ INCREMENT BY +100;
    -- Set the required value
    SELECT A_ID_SEQ.NEXTVAL FROM dual;
    -- Revert the sequence increment to 1
    ALTER SEQUENCE A_ID_SEQ INCREMENT BY 1;
    -- Check the result
    SELECT last_number FROM user_sequences WHERE sequence_name = 'A_ID_SEQ';

Prepare history migration load

If you are migrating history tables, you need to deactivate the History Event Handler in the Admin Center.

Deactivating the history event handler

Perform the migration load

  1. Make sure all Instance, Master data tables, and data statistics tables in the MDM storage are empty, otherwise the migration load fails. The tables are:

    • [prefix][entity]_I

    • [prefix][entity]_[master layer]

    • [prefix]X[entity]_I

    • [prefix]X[entity]_[master layer]

    • [prefix]_STATS

    • [prefix]X_STATS

      This also applies to any history tables you are migrating (prefixed by [hist_prefix]).
  2. Start the server using the latest runtime.

    Make sure the MDM server is configured to run in the read-write mode.
  3. Perform the migration load in the following order:

    • Authored master records migration load

    • Migration load

    • Manual match decision load

    • Overrides migration load

    • Task migration load

  4. Optionally, perform history migration load. If you are migrating history tables, reactivate the History Event Handler afterwards.

  5. Check the consistency of migrated data.

When the migration load is finished, although the engine metadata columns are preserved, the master layer is computed from scratch. This means all incremental exports provide a full set of the data in the first run after the migration.

Instance increment exports are preserved when engine metadata are preserved as well.

Was this page useful?