ONE Desktop ONE DQG ONE MDM ONE RDM ONE Runtime Server
latest (15.1.0) 14.5.x 13.9.x 13.8.x 13.7.x 13.6.x 13.5.0 13.4.x
User Community Service Desk Downloads
If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Performance Tuning

The following parameters are expert settings only. Only very advanced users can decide the right value for most settings and there is no guarantee of improved performance when a particular parameter is used.

JVM parameters

The most relevant Java Virtual Machine (JVM) settings are described in JVM configuration.

XMX memory settings

MDM uses both heap space (XMX memory) and non-heap space. The size of XMX memory should not exceed 70% of physical memory.

Runtime and steps performance tuning

It is possible to pass various runtime parameters in bulk in a runtime properties file or create a performance file for any plan.

In case of MDM, take note of the specific parameters used to configure parallelism of the Matching step, as described in the following section.

Matching step performance tuning

The Matching step can have the parallelism set on three levels -partition, key, and matching rule- with the following parameters:

  • mduPartitionParallelism: Partitions defined in the Matching step are processed in parallel.

  • mduKeyParallelism: Key rules defined within each partition are processed in parallel.

  • mduMatchParallelism: Matching rules defined within each key rule are processed in parallel.

Allowed values:

  • 0 - Parallelism disabled (default).

  • 1 - Parallelism enabled.

runtime.properties
...
mduPartitionParallelism=1
mduKeyParallelism=1
mduMatchParallelism=1
...

Once the parallelism is enabled, it is the actual number of Partitions, Key Rules, and Match Rules in the Matching step configuration that determines the resulting level of parallelism.

For example, if there are two partitions, five keys, and ten matching rules configured, and the parallelism is set on all 3 levels, as in the example settings (with all 3 parameters set to 1), the overall parallelism will require 100 threads.

Setting parallelism on step level

While it is possible to specify a global parallelism value that will be used by any Matching step, it is also possible to specify each parallelism setting separately for each Matching step with a specific notation.

For example, the following setting will be applied only on the Matching step whose ID is Party Matching.

Party\ Matching.mduPartitionParallelism=1
Spaces must be escaped using a backward slash (\). For example, Party Matching step has to be written as Party\ Matching.

It is also possible to combine both global and step-specific settings.

runtime.properties
...
### Global Matching Parallelism Settings ###
mduPartitionParallelism=1
mduKeyParallelism=1
mduMatchParallelism=1
...
### Party Matching ###
Party\ Matching.mduPartitionParallelism=1

MDM Server performance tuning

For the list of various parameters that influence the performance of MDM server, see Runtime Parameters.

To use some of these parameters, add them as a Java parameter (for example, -Dnme.copyColumns.skipUnchanged=true) or define them in the runtime properties file, as in the following example.

The values of runtime parameters set during the engine startup are available in the Admin Center under Runtime Parameters (see MD Process Monitoring, section Runtime parameters).

MDM Runtime parameters in runtime.properties file
...
nme.parallel.strategy=full
nme.delta.parallel=1
nme.commit.trimTooLongStrings=true
...

MDM database hints

MDM allows hinting for all SQL queries used internally. SQL queries are available in the <ONE_HOME>/vldb-commands_<DB>.xml and the file is referenced in the MDM configuration file.

Currently, only Oracle hints are pre-prepared: see the vldb-commands_oracle.xml file.

The ability to hint database queries was previously implemented using the runtime parameters. This method still works without any manual interventions.

Persistence layer debug logging

Logging for persistence layer SELECT operations can be set up in the MDM Server application properties (see MDM Server Application Properties, section Logging):

Example of application.properties
logging.level.com.ataccama.mdm.persistence=INFO
ataccama.one.mdm.db.logging.enabled=true
ataccama.one.mdm.db.logging.long-running-operation=10s
ataccama.one.mdm.db.logging.show-parameters=false

Query tuning

Query tuning is only available for PostgreSQL databases.

MDM Web App allows you to monitor and optimize performance of SQL queries from the Admin Center by downloading the CSV file containing performance settings and adjusting it as needed.

To identify a specific SQL query, the selector is used. It typically includes multiple fields, some of which might remain unfilled. When a more detailed selector is defined, it takes priority over a less specific one.

The selector includes the following fields:

  • callId: Typically, the call identifier within the application code, often in the format <ClassName>.<MethodName>.

  • planFilename: The name of the plan file if the query originates from a specific execution plan or file.

  • layer: The persistence layer name relevant to the SQL query, if applicable.

  • instanceEntity: The name of the instance entity if the query relates to a specific instance entity in the application.

  • masterEntity: The name of the master entity if the query is associated with a master entity in the application.

A sample CSV file looks as follows:

Sample performance settings file
"layer";"callId";"planFilename";"masterEntity";"instanceEntity";"subselectOrder";"fetchSize";"inClause";"analyzeResults";"expectedDataSize"
"C_";"PlanMergingOperation.collectBatchOutput";"../engine/trans/rel_party2party/norm_rel_org_org_merge.comp";"rel_org_org.master.norm";"rel_party2party.instance";"ASC";;"OR";"false";"LARGE"
"C_";"NmeHistoryEventProc.loadValidFromRecords";;;;;;"OR";"false";"LARGE"
"C_";"PlanMergingOperation.collectBatchOutput";"../engine/trans/provider_credentials/provider_provider_credentials_merge.comp";"provider_credentials.master.provider";"provider_credentials.instance";"ASC";;"OR";"false";"LARGE"
"C_";"PlanMergingOperation.deleteEmptyGroups";;;;;;"OR";"false";"LARGE"
"C_";"NmeModelStatistics.sumAndCleanup";;;;;;"OR";"false";"SMALL"
"C_";"NmeInstanceAccessor.createUnorderedInList.2";;;"party";;;"OR";"false";"SMALL"
"C_";"FullLoadBatchProc.processEntity";;;;"ASC";;"OR";"false";"LARGE"
"C_";"PlanMergingOperation.collectBatchOutput";"../engine/trans/party/norm_person_merge.comp";"person.master.norm";"party.instance";"ASC";;"OR";"false";"LARGE"
"C_";"VldbTableAccessor.findFirst";;;;;;"OR";"false";"SMALL"
"C_";"MduAccessor.addOldKeys";;;;;;"OR";"false";"LARGE"
"C_";"PlanMergingOperation.insertBatchRecords";;;;;;"OR";"false";"LARGE"
"C_";"PlanMergingOperation.collectBatchOutput";"../engine/trans/party/provider_party_merge.comp";"party.master.provider";"party.instance";"ASC";;"OR";"false";"LARGE"
"C_";"MduAccessorBase.fetchMasterRecords";;;;;;"OR";"false";"LARGE"
"C_";"MduMasterMatchingOperation.recalculateDuplicateIds";;;;;;"OR";"false";"LARGE"

Some of possible optimization settings include:

  • Insert sorting into subqueries: Sorts subquery results for improved performance.

  • Set fetch size: Controls the number of rows retrieved at once, impacting the performance.

  • Change the expected data size: Adjusts the estimated data size.

Query tuning parameters are stored in a database table named querytunning. To define the data source for this table, use the ataccama.one.mdm.performance.datasource property.

Data is loaded from this table during application startup and saved when new calls are registered. Keep in mind that manual changes to this table might be overwritten during runtime.

Query tuning parameters can be exported and imported into a CSV file, enabling easy modifications and sharing across different application installations.

Database statistics

This component monitors and records statistics for each SQL query used in the application. The statistics include the following:

  • Total Time: The cumulative time spent executing this query.

  • Min Time: The shortest duration for the query execution.

  • Max Time: The longest execution time for the query execution.

  • Request Count: The number of times the query was executed.

  • Last Finished Time: The date and time of the most recent completed execution.

  • Last Unfinished Time: The date and time of the last incomplete execution.

  • Completion Time: The date and time when the SQL query was finished.

A sample statistics CSV file looks as follows:

Sample performance statistics file
"layer";"callId";"planFilename";"masterEntity";"instanceEntity";"totalTime";"minTime";"maxTime";"requestCount";"lastFinishedCallTime";"lastUnfinishedCallTime"
"C_";"NmeModelStatistics.getInstanceStatistics";;;;"4";"0";"3";"2";"2023-09-20T11:10:32.142115612Z";"2023-09-20T11:10:32.139954465Z"
"C_";"VldbUnorderedResultIteratorDirect.ensureNextRecord";;;;"0";"0";"0";"0";;"2023-09-20T11:10:33.917376469Z"
"C_";"NmeModelStatistics.sumAndCleanup";;;;"7";"0";"7";"1";"2023-09-20T11:06:29.523252194Z";"2023-09-20T11:06:29.516591436Z"
"C_";"VldbTableAccessor.findFirst";;;;"10";"0";"2";"21";"2023-09-20T11:10:33.654213860Z";"2023-09-20T11:10:33.654027005Z"

Statistics are stored in a dedicated table called sqlcallstatistic. To define the data source for this table, use the ataccama.one.mdm.performance.datasource property (see MDM Server Application Properties, section Database Performance.

The statistics are updated every 20 seconds, providing real-time query performance monitoring. To facilitate query performance analysis, you can export these statistics to a CSV file using the designated endpoint /api/performance/statistic/download to facilitate query performance analysis.

Was this page useful?