If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Runtime Parameters

The following parameters are expert settings only. There is usually only an expert way to decide the right value and therefore there is no guarantee of improved performance when a parameter is used.

To use some of these parameters, just add them as a java parameter, for example, -Dnme.copyColumns.skipUnchanged=true or put them in the runtime.properties file. Runtime parameters values set during engine startup are available in ONE Runtime Server Admin.

Process Control

Name Default Value Description

nme.copyColumns.skipUnchanged

false

If true, a CopyColumns operation will skip its execution in certain situations when the data is unchanged in a way that recomputing the aggregation should yield the same result. Skip happens when sourceEntity has no changes and targetEntity has only UPDATE (no INSERT or DELETE).

Please note that DELETE is produced only by load operation with SourceDeletionStrategy DELETE or by calling RW native services with changeType X. SourceDeletionStrategy DEACTIVATE and calling RW native services with changeType D are considered as UPDATE.

Must be turned off for reprocessing if aggregation logic has changed.

nme.match.rematchIf.removeManualMatch

false

If true, rematchIfColumnsChanged is stronger than manual match overrides. If false, manual match overrides are stronger than rematchIfColumnsChanged.

Performance

Name Default Value Description

Name	Default Value	Description
nme.clean.parallelism	`1`	Default parallelism setting for cleansing plans. Can be overridden by `.perf` files.
nme.validate.parallelism	`1`	Default parallelism setting for master validation.
nme.multiLoad.parallel	`1`	Number of delta load operations that will be executed in parallel during a MultiLoad operation.
nme.delta.parallel	`1`	Number of threads that will be used to load entities in delta load operations.
nme.delta.parallel.threshold	`10 000`	Minimum number of incoming records (all entities together) to trigger parallel processing of delta load operation (only if nme.delta.parallel > 1).
nme.consolidation.parallel	`1`	Number of threads (and database connections) that will be used in consolidation phase.
nme.consolidation.parallel.threshold	`10 000`	Minimum number of modified records (all entities together) after delta detection to trigger parallel consolidation (only if nme.consolidation.parallel > 1).
nme.commit.parallel	`1`	Number of threads (workers) available during the Committing phase. Workers can simultaneously insert entities into repository.
nme.commit.parallel.threshold	`10 000`	Minimum number of modified records (all entities together) to trigger parallel committing (only if nme.commit.parallel > 1).
nme.commit.parallel.entity.maxWorkers	`1`	Maximum number of threads (workers) assignable to an entity during the committing phase. Enables parallel committing to a single entity.
nme.commit.parallel.sort	`weight`	A metric to prioritize entities for parallel committing. Possible values: count - Number of records. weight - Number of records * number of columns.
nme.commit.trimTooLongStrings	`false`	If `true`, String values exceeding column size will be automatically trimmed during committing. At the end of the processing a number of violations and maximum String length will be reported for each entity and column.
nme.taskExecutor.parallel	`20`	Number of tasks (loads, exports) being executed by Task Executor in parallel (as long as synchronization conditions allow that).
nme.export.read.parallel	`1`	The number of data sources that will be read in parallel in a single export operation.
nme.export.read.parallel.nameFilter	\	List of comma-separated names of export operations that will support parallel retrieval of its data sources (asterisk wildcard is supported).
nme.io.storage.memSize	`10 000`	Number of records kept in memory in temporary storages. Any excessive records are written to the hard disk and deserialized when needed. This is a shared parameter and will be applied to each entity from the Instance & Master models.
nme.io.storage.compress	`none`	Temporary files storing processed records can be compressed. Possible values: none - (default) No compression. compress - Uses java built-in deflate compression algorithm.
nme.checkPoint.directory	same as `java.io.tmpdir`	Base folder to store MDM checkpoint data for a resumable initial load operation. Keep in mind that the parameter can display NULL value if not specified by a user, however, internally it still uses the same base folder as defined in `the java.io.tmpdir` JVM flag.
nme.checkPoint.restore.parallel	`4`	The number of threads that will be used to restore the checkpoint.
nme.consolidation.pool.minSize	`1`	Minimum size of the pool of plan runners for Cleanse, Match and Merge plans.
nme.consolidation.pool.maxSize	`5`	Maximum size of the pool of plan runners for Cleanse, Match and Merge plans.
nme.parallel.strategy	`NONE`	Parallelism mode of RW operations: NONE - No parallelism, RW operations are serialized. FULL - Full parallelism, several RW operations can run at the same time (however only one batch export of one typeId can run in parallel), the only limitation is the threadPool size of NME Executor (parameter `nme.taskExecutor.parallel`). SINGLE_BATCH - Batch loads are serialized, with online requests running in parallel. EXCLUSIVE_BATCH - Any batch load stops online requests and is exclusive with everything, that is, only one batch can run at same time and nothing else.
nme.parallel.coordinator.key.strategy	`none`	Strategy for key processing to optimize memory usage when large source IDs are used. Possible values: none - (default) No key processing. compress - Keys are compressed using gzip, resulting in reduced memory usage with minimal CPU impact and no key conflicts. hash - Keys are hashed using SHA-256, providing the most efficient memory usage. In specific cases, possible key collision might lead to increased CPU usage.

nme.clean.parallelism

1

Default parallelism setting for cleansing plans. Can be overridden by .perf files.

nme.validate.parallelism

1

Default parallelism setting for master validation.

nme.multiLoad.parallel

1

Number of delta load operations that will be executed in parallel during a MultiLoad operation.

nme.delta.parallel

1

Number of threads that will be used to load entities in delta load operations.

nme.delta.parallel.threshold

10 000

Minimum number of incoming records (all entities together) to trigger parallel processing of delta load operation (only if nme.delta.parallel > 1).

nme.consolidation.parallel

1

Number of threads (and database connections) that will be used in consolidation phase.

nme.consolidation.parallel.threshold

10 000

Minimum number of modified records (all entities together) after delta detection to trigger parallel consolidation (only if nme.consolidation.parallel > 1).

nme.commit.parallel

1

Number of threads (workers) available during the Committing phase. Workers can simultaneously insert entities into repository.

nme.commit.parallel.threshold

10 000

Minimum number of modified records (all entities together) to trigger parallel committing (only if nme.commit.parallel > 1).

nme.commit.parallel.entity.maxWorkers

1

Maximum number of threads (workers) assignable to an entity during the committing phase. Enables parallel committing to a single entity.

nme.commit.parallel.sort

weight

A metric to prioritize entities for parallel committing. Possible values:

count - Number of records.
weight - Number of records * number of columns.

nme.commit.trimTooLongStrings

false

If true, String values exceeding column size will be automatically trimmed during committing. At the end of the processing a number of violations and maximum String length will be reported for each entity and column.

nme.taskExecutor.parallel

20

Number of tasks (loads, exports) being executed by Task Executor in parallel (as long as synchronization conditions allow that).

nme.export.read.parallel

1

The number of data sources that will be read in parallel in a single export operation.

nme.export.read.parallel.nameFilter

List of comma-separated names of export operations that will support parallel retrieval of its data sources (asterisk wildcard is supported).

nme.io.storage.memSize

10 000

Number of records kept in memory in temporary storages. Any excessive records are written to the hard disk and deserialized when needed. This is a shared parameter and will be applied to each entity from the Instance & Master models.

nme.io.storage.compress

none

Temporary files storing processed records can be compressed. Possible values:

none - (default) No compression.
compress - Uses java built-in deflate compression algorithm.

nme.checkPoint.directory

same as java.io.tmpdir

Base folder to store MDM checkpoint data for a resumable initial load operation. Keep in mind that the parameter can display NULL value if not specified by a user, however, internally it still uses the same base folder as defined in the java.io.tmpdir JVM flag.

nme.checkPoint.restore.parallel

4

The number of threads that will be used to restore the checkpoint.

nme.consolidation.pool.minSize

1

Minimum size of the pool of plan runners for Cleanse, Match and Merge plans.

nme.consolidation.pool.maxSize

5

Maximum size of the pool of plan runners for Cleanse, Match and Merge plans.

nme.parallel.strategy

NONE

Parallelism mode of RW operations:

NONE - No parallelism, RW operations are serialized.
FULL - Full parallelism, several RW operations can run at the same time (however only one batch export of one typeId can run in parallel), the only limitation is the threadPool size of NME Executor (parameter nme.taskExecutor.parallel).
SINGLE_BATCH - Batch loads are serialized, with online requests running in parallel.
EXCLUSIVE_BATCH - Any batch load stops online requests and is exclusive with everything, that is, only one batch can run at same time and nothing else.

nme.parallel.coordinator.key.strategy

none

Strategy for key processing to optimize memory usage when large source IDs are used.

Possible values:

none - (default) No key processing.
compress - Keys are compressed using gzip, resulting in reduced memory usage with minimal CPU impact and no key conflicts.
hash - Keys are hashed using SHA-256, providing the most efficient memory usage. In specific cases, possible key collision might lead to increased CPU usage.

Other

Name Default Value Description

Name	Default Value	Description
nme.taskExecutor.taskLogLevel	`DETAIL_ALL`	By default start and finish of all operations (load, export, reprocess, processDelta) and their subtasks are logged to standard logger (usually StdOut). This can be switched off by setting this parameter to `NO_DETAIL` - in that case only start and finish of the whole operation will be logged. `DETAIL_BATCH` is a hybrid option: all details will be logged only for batch loads and exports, only start and finish will be logged for RW online services.
nme.taskExecutor.taskPersistLevel	`PERSIST_ALL`	If set to `PERSIST_ALL`, batch operations and processDelta monitoring (/console/nmeMonitoring) is persisted between server restarts. If set to `NO_PERSIST` this monitoring is kept only in memory and lost with server restart.
nme.statistics.maxDirtyCount	`10`	Data statistics are computed in incremental way, creating partial counts that needs to be summed to get a correct total value. If the number of partial counts is greater than this parameter, partial counts are summed and replaced by a single total count.
nme.eventHandler.active	`true`	If `true`, after engine startup, event handlers are active, that is, data change events are automatically published. If `false`, data change events are stored in the configured storage directory but not published.
nme.eventHandler.publish.batchSize	`100`	Data change events are (for performance reasons) published in batches, this parameter sets the number of events in one batch.
nme.stream.consumers.active	`false`	Stream Consumers are stopped on server start. This option allows starting all stream consumers on server startup.
nme.services.range.maxCount	`100`	Maximum allowed value of count element in range expression in list and search services. If range expression is not given in request it is the default value.
nme.services.values.returnNulls	`false`	If `true`, the NME REST API returns null values in JSON responses, in format `{ "src_column" : null }`. With the default `false` setting, null values are omitted.
nme.tmpStorage.checkDuplicates	`false`	If `true`, the data written in each execution step during the Master Data Consolidation phase will be checked for duplicate values in the ID column. This additional checking will cause performance degradation (hence `false` by default) but may be useful when hunting internal problems.
nme.normalize.emptyStringIsNull	`true`	If `true`, MDM normalizes empty string "" to null everywhere: Input from services, load plans, and streaming. Output columns of transition plans. Other inputs like search queries in services or conditional exports. The parameter is set to `true` by default to deliver consistent processing of empty strings on all databases and prevent redundant overprocessing of input records with empty elements.
nme.model.source_id.length	`200`	Size of the source_id column. Keep in mind that this column is indexed, and some databases (Sybase ASE, MySQL) have a size limit for indexable varchar columns, which that setting this parameter to a large enough value might result in a "max index size breached" error.
nme.tasks.migration.apply-automatically	`true`	If set to `true`, all active drafts are migrated to tasks automatically when upgrading to the version 14.2.0.

nme.taskExecutor.taskLogLevel

DETAIL_ALL

By default start and finish of all operations (load, export, reprocess, processDelta) and their subtasks are logged to standard logger (usually StdOut).

This can be switched off by setting this parameter to NO_DETAIL - in that case only start and finish of the whole operation will be logged.

DETAIL_BATCH is a hybrid option: all details will be logged only for batch loads and exports, only start and finish will be logged for RW online services.

nme.taskExecutor.taskPersistLevel

PERSIST_ALL

If set to PERSIST_ALL, batch operations and processDelta monitoring (/console/nmeMonitoring) is persisted between server restarts.

If set to NO_PERSIST this monitoring is kept only in memory and lost with server restart.

nme.statistics.maxDirtyCount

10

Data statistics are computed in incremental way, creating partial counts that needs to be summed to get a correct total value. If the number of partial counts is greater than this parameter, partial counts are summed and replaced by a single total count.

nme.eventHandler.active

true

If true, after engine startup, event handlers are active, that is, data change events are automatically published.

If false, data change events are stored in the configured storage directory but not published.

nme.eventHandler.publish.batchSize

100

Data change events are (for performance reasons) published in batches, this parameter sets the number of events in one batch.

nme.stream.consumers.active

false

Stream Consumers are stopped on server start. This option allows starting all stream consumers on server startup.

nme.services.range.maxCount

100

Maximum allowed value of count element in range expression in list and search services. If range expression is not given in request it is the default value.

nme.services.values.returnNulls

false

If true, the NME REST API returns null values in JSON responses, in format { "src_column" : null }. With the default false setting, null values are omitted.

nme.tmpStorage.checkDuplicates

false

If true, the data written in each execution step during the Master Data Consolidation phase will be checked for duplicate values in the ID column. This additional checking will cause performance degradation (hence false by default) but may be useful when hunting internal problems.

nme.normalize.emptyStringIsNull

true

If true, MDM normalizes empty string "" to null everywhere:

Input from services, load plans, and streaming.
Output columns of transition plans.
Other inputs like search queries in services or conditional exports.

The parameter is set to true by default to deliver consistent processing of empty strings on all databases and prevent redundant overprocessing of input records with empty elements.

nme.model.source_id.length

200

Size of the source_id column. Keep in mind that this column is indexed, and some databases (Sybase ASE, MySQL) have a size limit for indexable varchar columns, which that setting this parameter to a large enough value might result in a "max index size breached" error.

nme.tasks.migration.apply-automatically

true

If set to true, all active drafts are migrated to tasks automatically when upgrading to the version 14.2.0.

DB / Storage

VLDB Persistence Parameters

The following parameters are usable only when VLDB Persistence is used (see Persistence Layer, section VLDB Persistence).

Name Default Value Description

nme.vldb.inList.batchSize

Removed in version 15.3.0 and later.

1 000

Batch size for inList input data.

nme.vldb.inList.commitSize

Removed in version 15.3.0 and later.

10 000

Commit size for inList input data.

nme.vldb.batchSize

100

Batch size for writing data in the Committing phase.

nme.vldb.commitSize

10 000

Commit size for writing data in the Committing phase.

nme.vldb.readonly

false

If true, hub starts in read-only mode. That mode can be later changed via native service switchReadWriteMode, see Native Services, section RW/ROModeSwitchingServices.

In read-only mode, the transactions cannot change the data in repository. In cluster installations (several NME instances are running over same DB storage), this flag should be set to true on all read-only nodes. At most one node may have this parameter set to false.

nme.vldb.smallDataThreshold

200 000

Threshold for number of inList records determining if LARGE or SMALL data hints should be used.

nme.vldb.tinyDataThreshold

10

Threshold for number of inList records, numbers below are considered tiny and different access method is used - direct access instead of inList.

nme.vldb.prefetch.large

If set, the value of this argument will be used as prefetch size for SELECT statements for LARGE expected data sizes.

nme.vldb.prefetch.small

If set, the value of this argument will be used as prefetch size for SELECT statements for SMALL expected data sizes.

nme.vldb.alterTableLevel

addColumns,alterColumnsAlter

Defines what alter table commands will be executed if a table in database does not match the model. This is list of permitted values:

addColumns. Add missing columns.
alterColumnsAlter. Alter existing columns that can be altered by DB DDL command, only extending varchar size and on some databases altering to clob is supported, that is, it won’t change from small to large numbers (INTEGER → LONG). If database does not support alteration via DDL command, warning is written into log.

nme.vldb.visibilityResolver.tmpTableThreshold

100

Threshold for number of Logical Transactions present in the TREG to switch engine queries using global temporary tables.

nme.vldb.xidMemory.size

10

Number of records in RW transactions (XIDs) that are stored on transaction for faster LTC - per one transaction and accessor (entity).

nme.vldb.xidMemory.totalSize

1000000

Number of records in RW transactions (XIDs) that are stored for faster LTC - total maximum.

Data Collector (LTC) Parameters

The following settings belong to an LTC feature related to the VLDB Persistence (see Persistence Layer, section VLDB Persistence). The LTC stands for Logical Transaction Collector, and it is responsible for deleting obsolete records (for example similar to Java garbage collector but dealing with obsolete transactions' data).

Name Default Value Description

Name	Default Value	Description
nme.vldb.collector.enabled	`true`	If `true` LTC process is enabled.
nme.vldb.collector.batchSize	`100`	Batch size (#deleted records in one execute() call) for the LTC process.
nme.vldb.collector.batchWait	`0`	Delay (in ms) between executing individual deletion batches in the LTC process.
nme.vldb.collector.commitSize	`10 000`	Commit size (#deleted records in deletion commit sets) for the LTC process.
nme.vldb.collector.commitWait	`100`	Delay (in ms) between processing individual deletion commit sets in the LTC process.

nme.vldb.collector.enabled

true

If true LTC process is enabled.

nme.vldb.collector.batchSize

100

Batch size (#deleted records in one execute() call) for the LTC process.

nme.vldb.collector.batchWait

0

Delay (in ms) between executing individual deletion batches in the LTC process.

nme.vldb.collector.commitSize

10 000

Commit size (#deleted records in deletion commit sets) for the LTC process.

nme.vldb.collector.commitWait

100

Delay (in ms) between processing individual deletion commit sets in the LTC process.

Hint Parameters

The parameters below make it possible to apply hints for some special computations being done during the processing.

The following hint examples are usable for Oracle 11+ database only.

Name Default Value Description

VldbUnorderedInList.match.small

/*+ ORDERED USE_NL(l d x) FULL(l) INDEX(d) INDEX(x) */

Hint for retrieving matched records for SMALL inLists in unordered lists.

VldbUnorderedInList.match.large

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving matched records for LARGE inLists in unordered lists.

VldbUnorderedInList.unmatch

`/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */ `

Hint for retrieving unmatched records for inLists in unordered lists.

For unmatch operations, there is no difference between SMALL and LARGE inputs.

VldbOrderedInList.match.small

/*+ ORDERED USE_NL(l d x) FULL(l) INDEX(d) INDEX(x) */

Hint for retrieving matched records for SMALL inLists in ordered lists.

VldbOrderedInList.match.large

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving matched records for LARGE inLists in ordered lists.

VldbOrderedInList.unmatch

`/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */ `

Hint for retrieving unmatched records for inLists in ordered lists.

For unmatch operations, there is no difference between SMALL and LARGE inputs.

VldbTableAccessor.find.small

/*+ORDERED USE_NL(x d) INDEX(x) */

Hint for retrieving data via SELECT WHERE for SMALL datasets.

VldbTableAccessor.find.large

/*+ORDERED USE_HASH(x d) FULL(d) FULL(x) SWAP_JOIN_INPUTS(x) */

Hint for retrieving data via SELECT WHERE for LARGE datasets.

DB Persistence Parameters

The following parameters are usable only when DB Persistence (Persistence Layer, section DB Persistence) is used.

Name Default Value Description

Name	Default Value	Description
nme.db.inList.batchSize	`1 000`	Batch size for inList input data.
nme.db.inList.commitSize	`infinity`	Commit size for inList input data.
nme.db.batchSize	`100`	Batch size for writing data in the Committing phase.
nme.db.commitSize	`infinity`	Commit size for writing data in the Committing phase.
nme.db.smallDataThreshold	`10 000`	Threshold for the number of records deciding if LARGE or SMALL data hints should be used.
nme.db.tinyDataThreshold	`10`	Threshold for number of records, numbers below are considered tiny and different access method is used - direct access instead of inList.
nme.db.alterTableLevel	`addColumns,alterColumnsAlter`	Defines what alter table commands will be executed if a table in database does not match the model. This is list of values, permitted values: addColumns. Add missing columns alterColumnsAlter. Alter existing columns that can be altered by DB DDL command, only extending varchar size and on some databases altering to clob is supported, that is it won’t change from small to large numbers (INTEGER → LONG). If database does not support alteration via DDL command, warning is written into log.

nme.db.inList.batchSize

1 000

Batch size for inList input data.

nme.db.inList.commitSize

infinity

Commit size for inList input data.

nme.db.batchSize

100

Batch size for writing data in the Committing phase.

nme.db.commitSize

infinity

Commit size for writing data in the Committing phase.

nme.db.smallDataThreshold

10 000

Threshold for the number of records deciding if LARGE or SMALL data hints should be used.

nme.db.tinyDataThreshold

10

Threshold for number of records, numbers below are considered tiny and different access method is used - direct access instead of inList.

nme.db.alterTableLevel

addColumns,alterColumnsAlter

Defines what alter table commands will be executed if a table in database does not match the model. This is list of values, permitted values:

addColumns. Add missing columns
alterColumnsAlter. Alter existing columns that can be altered by DB DDL command, only extending varchar size and on some databases altering to clob is supported, that is it won’t change from small to large numbers (INTEGER → LONG). If database does not support alteration via DDL command, warning is written into log.

Hint Parameters

The parameters below make it possible to apply hints for some special computations being done during the processing.

The following hint examples are usable for Oracle 11+ database only.

Name	Default Value	Description
DbUnorderedInList.match	/+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) /	Hint for retrieving matched records for inLists in unordered lists.
DbUnorderedInList.unmatch	/+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) /	Hint for retrieving unmatched records for inLists in unordered lists.
DbOrderedInList.match	/+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) /	Hint for retrieving matched records for inLists in ordered lists.
DbOrderedInList.unmatch	/+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) /	Hint for retrieving unmatched records for inLists in ordered lists.

Name

Default Value

Description

DbUnorderedInList.match

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving matched records for inLists in unordered lists.

DbUnorderedInList.unmatch

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving unmatched records for inLists in unordered lists.

DbOrderedInList.match

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving matched records for inLists in ordered lists.

DbOrderedInList.unmatch

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving unmatched records for inLists in ordered lists.

Was this page useful?