If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Runtime Parameters

The following parameters are expert settings only. There is usually only an expert way to decide the right value and therefore there is no guarantee of improved performance when a parameter is used.

Where are runtime parameters configured?

We differentiate between MDM-specific runtime parameters and general runtime properties.

You can recognize MDM-specific runtime parameters by the nme. prefix. These parameters are stored in the MDM database and updated through the MDM Web App Admin Center.

They are loaded to the database automatically on the initial server startup, or manually, for example, after an external configuration change or after resetting your environment. For details about how and when to run the migration manually, see MDM Web App Admin Center > Server Dashboard.

After editing certain parameters in the Admin Center, you need to restart the MDM Server for the changes to take effect. If required, this is mentioned in the parameter description and indicated in the web application.

General runtime parameters, such as those configuring hardware resources available to the MDM Server, are provided to the JVM when starting the MDM Server (for example, -Dnme.copyColumns.skipUnchanged=true) or included in the runtime.properties file.

By default, the file is located in <ataccama.one.mdm.config.config-folder>/etc (alternatively, the file is found in <ataccama.one.mdm.config.runtime-folder>). To verify the actual location, check the values of the ataccama.one.mdm.config.config-folder and ataccama.one.mdm.config.runtime-folder application properties.

Runtime parameters values set during engine startup are available in ONE Runtime Server Admin.

Even though they start with the nme. prefix, nme.matching.* properties are configured and managed the same way as general runtime parameters: through the runtime.properties file. This way, you can configure them separately for each instance instead of having a shared configuration for all matching steps.

Edit permissions for runtime parameters

Runtime parameters are edited from the MDM Web App Admin Center > Runtime Parameters screen. For details, see MDM Web App Admin Center.

To edit runtime parameters, you need to be assigned a role defined in the ataccama.one.mdm.admin-center.manager.required-roles application property. By default, this is the MDM_server_admin role.

To learn more about the relevant application properties, see MDM Admin Center Permissions.

Process control

Name Default value Description

nme.copyColumns.skipUnchanged

false

If true, a CopyColumns operation is skipped in certain situations when the data is unchanged in such a way that recomputing the aggregation should return the same result. This happens when sourceEntity has no changes and targetEntity has only UPDATE changes (no INSERT or DELETE).

Note that DELETE is produced only by load operation with SourceDeletionStrategy DELETE or by calling RW native services with changeType X. SourceDeletionStrategy DEACTIVATE and calling RW native services with changeType D are considered as UPDATE.

Must be turned off for reprocessing if the aggregation logic has changed.

nme.match.rematchIf.removeManualMatch

false

If true, rematchIfColumnsChanged is prioritized over manual match overrides. If false, manual match overrides are prioritized over rematchIfColumnsChanged.

Performance

Name Default value Description

nme.clean.parallelism

1

Default parallelism setting for cleansing plans. Can be overridden by .perf files.

nme.validate.parallelism

1

Default parallelism setting for master validation.

nme.multiLoad.parallel

1

Number of delta load operations that are executed in parallel during a M`ultiLoad` operation.

nme.delta.parallel

1

Number of threads that are used to load entities in delta load operations.

nme.delta.parallel.threshold

10 000

Minimum number of incoming records (all entities together) required to trigger parallel processing of delta load operation (used only if nme.delta.parallel is greater than 1).

nme.consolidation.parallel

1

Number of threads (and database connections) that are used in the consolidation phase.

nme.consolidation.parallel.threshold

10 000

Minimum number of modified records (all entities together) after delta detection required to trigger parallel consolidation (used only if nme.consolidation.parallel is greater than 1).

nme.commit.parallel

1

Number of threads (workers) available during the committing phase. Workers can simultaneously insert entities into the repository.

nme.commit.parallel.threshold

10 000

Minimum number of modified records (all entities together) required to trigger parallel committing (only if nme.commit.parallel is greater than 1).

nme.commit.parallel.entity.maxWorkers

1

Maximum number of threads (workers) that can be assigned to an entity during the committing phase. Enables parallel committing to a single entity.

nme.commit.parallel.sort

weight

A metric used to prioritize entities for parallel committing. Allowed values:

count - Number of records.
weight - Number of records times the number of columns.

nme.commit.trimTooLongStrings

false

If true, string values exceeding the column size are automatically trimmed during committing. When the processing finishes, the number of violations and the maximum string length are reported for each entity and column.

nme.matching.partition.parallel

0

Number of threads used for processing partitions in the Matching step. Can also be configured for specific entities or specific matching within the entity, see Matching step performance tuning.

Configured in the runtime.properties file.

nme.matching.key.parallel

0

Number of threads used for processing key rules in the Matching step. Can also be configured for specific entities or specific matching within the entity, see Matching step performance tuning.

Configured in the runtime.properties file.

nme.matching.rule.parallel

0

Number of threads used for processing matching rules in the Matching step. Can also be configured for specific entities or specific matching within the entity, see Matching step performance tuning.

Configured in the runtime.properties file.

nme.matching.parallel.threshold

10000

Minimum amount of data belonging to a partition or key rule required to enable parallel processing (only if the corresponding parallelism setting is greater than 1). Can also be configured for specific entities or specific matching within the entity, see Matching step performance tuning.

Configured in the runtime.properties file.

nme.matching.parallel.group.threshold

100

Minimum size of key groups necessary for processing matching rules in parallel (only if the key rule parallelism is greater than 1). Can also be configured for specific entities or specific matching within the entity, see Matching step performance tuning.

Configured in the runtime.properties file.

nme.matching.parallel.statistics

false

Enables metrics tracking of the amount of data processed in each part (both serially and in parallel). For key groups, reports the number of key groups and the average group size.

Configured in the runtime.properties file.

nme.taskExecutor.parallel

20

Number of tasks (loads, exports) executed by Task Executor in parallel (provided that synchronization conditions allow that).

After editing the value, restart the MDM server to apply the change.

nme.export.read.parallel

1

Number of data sources that are read in parallel in a single export operation.

nme.export.read.parallel.nameFilter

List of comma-separated names of export operations that will support parallel retrieval of data sources. Using the asterisk (*) wildcard is supported.

nme.io.storage.memSize

10 000

Number of records kept in memory in temporary storages. Any records exceeding that number are written to the hard disk and deserialized when needed.

This is a shared parameter and is applied to each entity from the Instance and Master models.

After editing the value, restart the MDM server to apply the change.

nme.io.storage.compress

none

Temporary files storing processed records can be compressed. Possible values:

none - (default) No compression.
compress - Uses the built-in Java deflate compression algorithm.

After editing the value, restart the MDM server to apply the change.

nme.checkPoint.directory

same as java.io.tmpdir

Base folder for storing MDM checkpoint data for a resumable initial load operation.

If the parameter isn’t specified, internally it still uses the same base folder as defined in the java.io.tmpdir JVM flag. In addition, a NULL value is displayed in that case in the Admin Center.

nme.checkPoint.restore.parallel

4

Number of threads used to restore the checkpoint.

nme.consolidation.pool.minSize

1

Minimum size of the pool of plan runners for cleanse, match, and merge plans.

After editing the value, restart the MDM server to apply the change.

nme.consolidation.pool.maxSize

5

Maximum size of the pool of plan runners for cleanse, match, and merge plans.

After editing the value, restart the MDM server to apply the change.

nme.parallel.strategy

NONE

The parallelism mode of RW operations:

NONE - No parallelism, RW operations are serialized.
FULL - Full parallelism, several RW operations can run at the same time. However, only one batch export of one typeId can run in parallel, the only limitation is the threadPool size of NME Executor (parameter nme.taskExecutor.parallel).
SINGLE_BATCH - Batch loads are serialized, with online requests running in parallel.
EXCLUSIVE_BATCH - Any batch load stops online requests and is exclusive with everything (that is, only one batch can run at same time and nothing else).

After editing the value, restart the MDM server to apply the change.

nme.parallel.coordinator.key.strategy

none

The strategy for key processing to optimize memory usage when large source IDs are used.

Possible values:

PLAIN - (default) No key processing.
COMPRESS - Keys are compressed using gzip, resulting in reduced memory usage with minimal CPU impact and no key conflicts.
SHA256 - Keys are hashed using SHA-256, providing the most efficient memory usage. In specific cases, possible key collision might lead to increased CPU usage.

After editing the value, restart the MDM server to apply the change.

nme.parallel.restart.wait

10000

Delay (in milliseconds) in a restart loop while the server waits for all conflicting transactions to finish.

NOTE: After editing the value, restart the MDM server to apply the change.

Other

Name Default value Description

nme.taskExecutor.taskLogLevel

DETAIL_ALL

By default, the start and finish of all operations (load, export, reprocess, processDelta) and their subtasks are logged to the standard logger (usually StdOut).

To switch this off, set the value to NO_DETAIL. In that case, only the start and finish of the whole operation are logged.

There is another, hybrid option: DETAIL_BATCH. When used, all details are logged only for batch loads and exports, and only the start and finish are logged for RW online services.

nme.statistics.maxDirtyCount

10

Data statistics are computed in an incremental way, creating partial counts that needs to be summed up to get the correct total value. If the number of partial counts is greater than this parameter, partial counts are summed up and replaced by a single total count.

nme.eventHandler.active

true

If true, after engine startup, event handlers are active, that is, data change events are automatically published.

If false, data change events are stored in the configured storage directory but not published.

nme.eventHandler.publish.batchSize

100

Data change events are (for performance reasons) published in batches. This parameter sets the number of events in one batch.

nme.stream.consumers.active

false

Stream consumers are stopped on server start. This option allows starting all stream consumers on server startup.

nme.services.range.maxCount

100

Maximum allowed value of the count element in the range expression in list and search services. If the range expression is not given in the request, the default value is used.

After editing the value, restart the MDM server to apply the change.

nme.services.values.returnNulls

false

If true, the NME REST API returns NULL values in JSON responses, in the format { "src_column" : null }. With the default false value, NULL values are omitted.

nme.tmpStorage.checkDuplicates

false

If true, the data written in each execution step during the master data consolidation phase is checked for duplicate values in the ID column.

This additional check leads to reduced performance, which is why it is set to false by default. However, it can be useful for troubleshooting.

nme.normalize.emptyStringIsNull

true

If true, empty strings ("") are normalized to NULL everywhere:

Input from services, load plans, and streaming.
Output columns of transition plans.
Other inputs like search queries in services or conditional exports.

By default, the parameter is set to true to ensure consistent processing of empty strings on all databases and prevent redundant overprocessing of input records with empty elements.

nme.model.source_id.length

200

Size of the source_id column.

Keep in mind that this column is indexed, and some databases have a size limit for indexable varchar columns. This means that setting this parameter to a large enough value might result in a "max index size breached" error, depending on your database.

After editing the value, restart the MDM server to apply the change.

Database and storage

VLDB persistence parameters

The following parameters apply only when VLDB persistence is used.

Name Default value Description

nme.vldb.batchSize

100

Batch size for writing data in the committing phase.

nme.vldb.commitSize

10 000

Commit size for writing data in the committing phase.

nme.readonly

false

If true, the hub starts in read-only mode. That mode can be later changed via the native service switchReadWriteMode.

In read-only mode, the transactions cannot change data in the repository. In cluster installations, where several NME instances are running over the same database storage, this flag should be set to true on all read-only nodes. At most one node can have this parameter set to false.

After editing the value, restart the MDM server to apply the change.

nme.vldb.smallDataThreshold

200 000

Threshold for the number of inList records determining if LARGE or SMALL data hints should be used.

After editing the value, restart the MDM server to apply the change.

nme.vldb.tinyDataThreshold

10

Threshold for the number of inList records. Numbers under this threshold are considered tiny and a different access method is used (direct access instead of inList).

nme.vldb.prefetch.large

If set, the value of this property is used as the prefetch size for SELECT statements for LARGE expected data sizes.

nme.vldb.prefetch.small

If set, the value of this property is used as the prefetch size for SELECT statements for SMALL expected data sizes.

nme.vldb.alterTableLevel

addColumns,addIndexes,alterColumnsAlter

Defines what alter table commands are executed if a table in the database does not match the model. Allowed values:

addColumns: Add missing columns.
alterColumnsAlter: Alter existing columns that can be altered by DB DDL command, only extending varchar size and on some databases altering to clob is supported, that is, it won’t change from small to large numbers (INTEGER to LONG). If the database does not support alteration via DDL command, a warning is written into the log.
addIndexes: Add missing indexes.

After editing the value, restart the MDM server to apply the change.

nme.vldb.visibilityResolver.tmpTableThreshold

100

Threshold for the number of logical transactions present in the TREG to switch engine queries using global temporary tables.

After editing the value, restart the MDM server to apply the change.

nme.vldb.xidMemory.size

10

Number of records in RW transactions (XIDs) that are stored on transaction for faster LTC. This limits the size of one transaction and accessor (entity).

After editing the value, restart the MDM server to apply the change.

nme.vldb.xidMemory.totalSize

1000000

The highest number of records in RW transactions (XIDs) that are stored for faster LTC.

NOTE: After editing the value, restart the MDM server to apply the change.

nme.vldb.heartbeat.interval

60

How often (in seconds) the main database connection executes commits during a write operation which runs with multiple database connections. This prevents the main connection from running too long without a commit and helps avoid some issues related to long-running transactions (for instance, it limits the size of database logs).

NOTE: After editing the value, restart the MDM server to apply the change.

Data collector (LTC) parameters

The following settings apply to an LTC feature related to VLDB persistence.

The LTC stands for Logical Transaction Collector, which is a component responsible for deleting obsolete records. In a way, it is similar to Java garbage collector but it deals with data related to obsolete transactions.

Name Default value Description

nme.vldb.collector.enabled

true

If true, the LTC process is enabled.

After editing the value, restart the MDM server to apply the change.

nme.vldb.collector.batchSize

100

Batch size (the number of deleted records in one execute() call) for the LTC process.

nme.vldb.collector.batchWait

0

Delay (in milliseconds) between executing individual deletion batches in the LTC process.

nme.vldb.collector.commitSize

10 000

Commit size (the number of deleted records in deletion commit sets) for the LTC process.

nme.vldb.collector.commitWait

100

Delay (in milliseconds) between processing individual deletion commit sets in the LTC process.

Hint parameters

The following parameters make it possible to apply hints for some special computations that are done during the processing.

The hint examples given can only be used in Oracle databases version 11 and later.

After editing, restart the MDM server to apply the changes.

Name Default value Description

nme.vldb.unorderedInList.match.small

/*+ ORDERED USE_NL(l d x) FULL(l) INDEX(d) INDEX(x) */

Hint for retrieving matched records for SMALL inLists in unordered lists.

nme.vldb.unorderedInList.match.large

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving matched records for LARGE inLists in unordered lists.

nme.vldb.unorderedInList.unmatch

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving unmatched records for inLists in unordered lists.

For unmatch operations, there is no difference between SMALL and LARGE inputs.

nme.vldb.orderedInList.match.small

/*+ ORDERED USE_NL(l d x) FULL(l) INDEX(d) INDEX(x) */

Hint for retrieving matched records for SMALL inLists in ordered lists.

nme.vldb.orderedInList.match.large

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving matched records for LARGE inLists in ordered lists.

nme.vldb.tableAccessor.find.small

/*+ORDERED USE_NL(x d) INDEX(x) */

Hint for retrieving data via SELECT WHERE for SMALL datasets.

nme.vldb.tableAccessor.find.large

/*+ORDERED USE_HASH(x d) FULL(d) FULL(x) SWAP_JOIN_INPUTS(x) */

Hint for retrieving data via SELECT WHERE for LARGE datasets.

nme.vldb.tableAccessor.find.main.small

/*+ ORDERED USE_NL(m j) */

Hint for retrieving data via SELECT LEFT JOIN for SMALL datasets.

nme.vldb.tableAccessor.find.main.large

/*+ ORDERED USE_HASH(m j) */

Hint for retrieving data via SELECT LEFT JOIN for LARGE datasets.

nme.vldb.tableAccessor.count.accuracy.mode

HIGH

Hint for controlling the accuracy of record counting.

If set to HIGH, an exact count of records is provided using a complex query that joins a temporary table with the main table. If set to LOW, an approximate count of records is provided using a simpler and faster query on the main table.

Database persistence parameters

The following parameters can be applied only when database persistence is used.

Name Default value Description

nme.db.inList.batchSize

1 000

Batch size for inList input data.

nme.db.inList.commitSize

infinity

Commit size for inList input data.

nme.db.batchSize

100

Batch size for writing data in the committing phase.

nme.db.commitSize

infinity

Commit size for writing data in the committing phase.

nme.db.smallDataThreshold

10 000

Threshold for the number of records deciding if LARGE or SMALL data hints should be used.

nme.db.tinyDataThreshold

10

Threshold for the number of records. Numbers under this threshold are considered tiny and a different access method is used (direct access instead of inList).

nme.db.alterTableLevel

addColumns,addIndexes,alterColumnsAlter

Defines what alter table commands are executed if a table in the database does not match the model. Allowed values:

addColumns: Add missing columns.
alterColumnsAlter: Alter existing columns that can be altered by DB DDL command, only extending varchar size and on some databases altering to clob is supported, that is, it won’t change from small to large numbers (INTEGER to LONG). If the database does not support alteration via DDL command, a warning is written into the log.
addIndexes: Add missing indexes.

After editing the value, restart the MDM server to apply the change.

Hint parameters

The following parameters make it possible to apply hints for some special computations that are done during the processing.

The hint examples given can only be used in Oracle databases version 11 and later.

After editing, restart the MDM server to apply the changes.

Name Default value Description

Name	Default value	Description
nme.db.unorderedInList.match	`/+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) /`	Hint for retrieving matched records for `inLists` in unordered lists.
nme.db.unorderedInList.unmatch	`/+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) /`	Hint for retrieving unmatched records for `inLists` in unordered lists.
nme.db.unorderedInList.match	`/+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) /`	Hint for retrieving matched records for `inLists` in ordered lists.
nme.db.unorderedInList.unmatch	`/+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) /`	Hint for retrieving unmatched records for `inLists` in ordered lists.
nme.db.inList.main.hint	`/+ ORDERED USE_HASH(m j) /`	Hint for retrieving data via SELECT LEFT JOIN.

nme.db.unorderedInList.match

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving matched records for inLists in unordered lists.

nme.db.unorderedInList.unmatch

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving unmatched records for inLists in unordered lists.

nme.db.unorderedInList.match

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving matched records for inLists in ordered lists.

nme.db.unorderedInList.unmatch

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving unmatched records for inLists in ordered lists.

nme.db.inList.main.hint

/*+ ORDERED USE_HASH(m j) */

Hint for retrieving data via SELECT LEFT JOIN.

Was this page useful?