User Community Service Desk Downloads
If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Runtime Parameters

The following parameters are expert settings only. There is usually only an expert way to decide the right value and therefore there is no guarantee of improved performance when a parameter is used.

To use some of these parameters, just add them as a java parameter, for example, -Dnme.copyColumns.skipUnchanged=true or put them in the runtime.properties file. Runtime parameters values set during engine startup are available in ONE Runtime Server Admin.

Process Control

Name Default Value Description

nme.copyColumns.skipUnchanged

false

If true, a CopyColumns operation will skip its execution in certain situations when the data is unchanged in a way that recomputing the aggregation should yield the same result. Skip happens when sourceEntity has no changes and targetEntity has only UPDATE (no INSERT or DELETE).

Please note that DELETE is produced only by load operation with SourceDeletionStrategy DELETE or by calling RW native services with changeType X. SourceDeletionStrategy DEACTIVATE and calling RW native services with changeType D are considered as UPDATE.
Must be turned off for reprocessing if aggregation logic has changed.

nme.match.rematchIf.removeManualMatch

false

If true, rematchIfColumnsChanged is stronger than manual match overrides. If false, manual match overrides are stronger than rematchIfColumnsChanged.

Performance

Name Default Value Description

nme.clean.parallelism

1

Default parallelism setting for cleansing plans. Can be overridden by .perf files.

nme.validate.parallelism

1

Default parallelism setting for master validation.

nme.multiLoad.parallel

1

Number of delta load operations that will be executed in parallel during a MultiLoad operation.

nme.delta.parallel

1

Number of threads that will be used to load entities in delta load operations.

nme.delta.parallel.threshold

10 000

Minimum number of incoming records (all entities together) to trigger parallel processing of delta load operation (only if nme.delta.parallel > 1).

nme.consolidation.parallel

1

Number of threads (and database connections) that will be used in consolidation phase.

nme.consolidation.parallel.threshold

10 000

Minimum number of modified records (all entities together) after delta detection to trigger parallel consolidation (only if nme.consolidation.parallel > 1).

nme.commit.parallel

1

Number of threads (workers) available during the Committing phase. Workers can simultaneously insert entities into repository.

nme.commit.parallel.threshold

10 000

Minimum number of modified records (all entities together) to trigger parallel committing (only if nme.commit.parallel > 1).

nme.commit.parallel.entity.maxWorkers

1

Maximum number of threads (workers) assignable to an entity during the committing phase. Enables parallel committing to a single entity.

nme.commit.parallel.sort

weight

A metric to prioritize entities for parallel committing. Possible values:

  • count - Number of records.

  • weight - Number of records * number of columns.

nme.commit.trimTooLongStrings

false

If true, String values exceeding column size will be automatically trimmed during committing. At the end of the processing a number of violations and maximum String length will be reported for each entity and column.

nme.matching.partition.parallel

0

Number of threads used for processing partitions in the Matching step. Can also be configured for specific entities or specific matching within the entity, see Performance Tuning, section Matching step performance tuning.

nme.matching.key.parallel

0

Number of threads used for processing key rules in the Matching step. Can also be configured for specific entities or specific matching within the entity, see Performance Tuning, section Matching step performance tuning.

nme.matching.rule.parallel

0

Number of threads used for processing matching rules in the Matching step. Can also be configured for specific entities or specific matching within the entity, see Performance Tuning, section Matching step performance tuning.

nme.matching.parallel.threshold

10000

Minimum amount of data belonging to a partition or key rule required to enable parallel processing (only if the corresponding parallelism setting is greater than 1). Can also be configured for specific entities or specific matching within the entity, see Performance Tuning, section Matching step performance tuning.

nme.matching.parallel.group.threshold

100

Minimum size of key groups necessary for processing matching rules in parallel (only if the key rule parallelism is greater than 1). Can also be configured for specific entities or specific matching within the entity, see Performance Tuning, section Matching step performance tuning.

nme.matching.parallel.statistics

false

Enables metrics tracking of the amount processed in each part (both serially and in parallel). For key groups, reports the number of key groups and the average group size.

nme.taskExecutor.parallel

20

Number of tasks (loads, exports) being executed by Task Executor in parallel (as long as synchronization conditions allow that).

nme.export.read.parallel

1

The number of data sources that will be read in parallel in a single export operation.

nme.export.read.parallel.nameFilter

\

List of comma-separated names of export operations that will support parallel retrieval of its data sources (asterisk wildcard is supported).

nme.io.storage.memSize

10 000

Number of records kept in memory in temporary storages. Any excessive records are written to the hard disk and deserialized when needed. This is a shared parameter and will be applied to each entity from the Instance & Master models.

nme.io.storage.compress

none

Temporary files storing processed records can be compressed. Possible values:

  • none - (default) No compression.

  • compress - Uses java built-in deflate compression algorithm.

nme.checkPoint.directory

same as java.io.tmpdir

Base folder to store MDM checkpoint data for a resumable initial load operation. Keep in mind that the parameter can display NULL value if not specified by a user, however, internally it still uses the same base folder as defined in the java.io.tmpdir JVM flag.

nme.checkPoint.restore.parallel

4

The number of threads that will be used to restore the checkpoint.

nme.consolidation.pool.minSize

1

Minimum size of the pool of plan runners for Cleanse, Match and Merge plans.

nme.consolidation.pool.maxSize

5

Maximum size of the pool of plan runners for Cleanse, Match and Merge plans.

nme.parallel.strategy

NONE

Parallelism mode of RW operations:

  • NONE - No parallelism, RW operations are serialized.

  • FULL - Full parallelism, several RW operations can run at the same time (however only one batch export of one typeId can run in parallel), the only limitation is the threadPool size of NME Executor (parameter nme.taskExecutor.parallel).

  • SINGLE_BATCH - Batch loads are serialized, with online requests running in parallel.

  • EXCLUSIVE_BATCH - Any batch load stops online requests and is exclusive with everything, that is, only one batch can run at same time and nothing else.

nme.parallel.coordinator.key.strategy

none

Strategy for key processing to optimize memory usage when large source IDs are used.

Possible values:

  • none - (default) No key processing.

  • compress - Keys are compressed using gzip, resulting in reduced memory usage with minimal CPU impact and no key conflicts.

  • hash - Keys are hashed using SHA-256, providing the most efficient memory usage. In specific cases, possible key collision might lead to increased CPU usage.

Other

Name Default Value Description

nme.taskExecutor.taskLogLevel

DETAIL_ALL

By default start and finish of all operations (load, export, reprocess, processDelta) and their subtasks are logged to standard logger (usually StdOut).

This can be switched off by setting this parameter to NO_DETAIL - in that case only start and finish of the whole operation will be logged.

DETAIL_BATCH is a hybrid option: all details will be logged only for batch loads and exports, only start and finish will be logged for RW online services.

nme.taskExecutor.taskPersistLevel

PERSIST_ALL

If set to PERSIST_ALL, batch operations and processDelta monitoring (/console/nmeMonitoring) is persisted between server restarts.

If set to NO_PERSIST this monitoring is kept only in memory and lost with server restart.

nme.statistics.maxDirtyCount

10

Data statistics are computed in incremental way, creating partial counts that needs to be summed to get a correct total value. If the number of partial counts is greater than this parameter, partial counts are summed and replaced by a single total count.

nme.eventHandler.active

true

If true, after engine startup, event handlers are active, that is, data change events are automatically published.

If false, data change events are stored in the configured storage directory but not published.

nme.eventHandler.publish.batchSize

100

Data change events are (for performance reasons) published in batches, this parameter sets the number of events in one batch.

nme.stream.consumers.active

false

Stream Consumers are stopped on server start. This option allows starting all stream consumers on server startup.

nme.services.range.maxCount

100

Maximum allowed value of count element in range expression in list and search services. If range expression is not given in request it is the default value.

nme.services.values.returnNulls

false

If true, the NME REST API returns null values in JSON responses, in format { "src_column" : null }. With the default false setting, null values are omitted.

nme.tmpStorage.checkDuplicates

false

If true, the data written in each execution step during the Master Data Consolidation phase will be checked for duplicate values in the ID column. This additional checking will cause performance degradation (hence false by default) but may be useful when hunting internal problems.

nme.normalize.emptyStringIsNull

true

If true, MDM normalizes empty string "" to null everywhere:

  • Input from services, load plans, and streaming.

  • Output columns of transition plans.

  • Other inputs like search queries in services or conditional exports.

The parameter is set to true by default to deliver consistent processing of empty strings on all databases and prevent redundant overprocessing of input records with empty elements.

nme.model.source_id.length

200

Size of the source_id column. Keep in mind that this column is indexed, and some databases (Sybase ASE, MySQL) have a size limit for indexable varchar columns, which that setting this parameter to a large enough value might result in a "max index size breached" error.

nme.tasks.migration.apply-automatically

true

If set to true, all active drafts are migrated to tasks automatically when upgrading to the version 14.2.0.

DB / Storage

VLDB Persistence Parameters

The following parameters are usable only when VLDB Persistence is used (see Persistence Layer, section VLDB Persistence).

Name Default Value Description

nme.vldb.inList.batchSize

Removed in version 15.3.0 and later.

1 000

Batch size for inList input data.

nme.vldb.inList.commitSize

Removed in version 15.3.0 and later.

10 000

Commit size for inList input data.

nme.vldb.batchSize

100

Batch size for writing data in the Committing phase.

nme.vldb.commitSize

10 000

Commit size for writing data in the Committing phase.

nme.vldb.readonly

false

If true, hub starts in read-only mode. That mode can be later changed via native service switchReadWriteMode, see Native Services, section RW/ROModeSwitchingServices.

In read-only mode, the transactions cannot change the data in repository. In cluster installations (several NME instances are running over same DB storage), this flag should be set to true on all read-only nodes. At most one node may have this parameter set to false.

nme.vldb.smallDataThreshold

200 000

Threshold for number of inList records determining if LARGE or SMALL data hints should be used.

nme.vldb.tinyDataThreshold

10

Threshold for number of inList records, numbers below are considered tiny and different access method is used - direct access instead of inList.

nme.vldb.prefetch.large

/

If set, the value of this argument will be used as prefetch size for SELECT statements for LARGE expected data sizes.

nme.vldb.prefetch.small

/

If set, the value of this argument will be used as prefetch size for SELECT statements for SMALL expected data sizes.

nme.vldb.alterTableLevel

addColumns,alterColumnsAlter

Defines what alter table commands will be executed if a table in database does not match the model. This is list of permitted values:

  • addColumns. Add missing columns.

  • alterColumnsAlter. Alter existing columns that can be altered by DB DDL command, only extending varchar size and on some databases altering to clob is supported, that is, it won’t change from small to large numbers (INTEGER → LONG). If database does not support alteration via DDL command, warning is written into log.

nme.vldb.visibilityResolver.tmpTableThreshold

100

Threshold for number of Logical Transactions present in the TREG to switch engine queries using global temporary tables.

nme.vldb.xidMemory.size

10

Number of records in RW transactions (XIDs) that are stored on transaction for faster LTC - per one transaction and accessor (entity).

nme.vldb.xidMemory.totalSize

1000000

Number of records in RW transactions (XIDs) that are stored for faster LTC - total maximum.

Data Collector (LTC) Parameters

The following settings belong to an LTC feature related to the VLDB Persistence (see Persistence Layer, section VLDB Persistence). The LTC stands for Logical Transaction Collector, and it is responsible for deleting obsolete records (for example similar to Java garbage collector but dealing with obsolete transactions' data).

Name Default Value Description

nme.vldb.collector.enabled

true

If true LTC process is enabled.

nme.vldb.collector.batchSize

100

Batch size (#deleted records in one execute() call) for the LTC process.

nme.vldb.collector.batchWait

0

Delay (in ms) between executing individual deletion batches in the LTC process.

nme.vldb.collector.commitSize

10 000

Commit size (#deleted records in deletion commit sets) for the LTC process.

nme.vldb.collector.commitWait

100

Delay (in ms) between processing individual deletion commit sets in the LTC process.

Hint Parameters

The parameters below make it possible to apply hints for some special computations being done during the processing.

The following hint examples are usable for Oracle 11+ database only.
Name Default Value Description

VldbUnorderedInList.match.small

/*+ ORDERED USE_NL(l d x) FULL(l) INDEX(d) INDEX(x) */

Hint for retrieving matched records for SMALL inLists in unordered lists.

VldbUnorderedInList.match.large

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving matched records for LARGE inLists in unordered lists.

VldbUnorderedInList.unmatch

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving unmatched records for inLists in unordered lists.

For unmatch operations, there is no difference between SMALL and LARGE inputs.

VldbOrderedInList.match.small

/*+ ORDERED USE_NL(l d x) FULL(l) INDEX(d) INDEX(x) */

Hint for retrieving matched records for SMALL inLists in ordered lists.

VldbOrderedInList.match.large

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving matched records for LARGE inLists in ordered lists.

VldbOrderedInList.unmatch

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving unmatched records for inLists in ordered lists.

For unmatch operations, there is no difference between SMALL and LARGE inputs.

VldbTableAccessor.find.small

/*+ORDERED USE_NL(x d) INDEX(x) */

Hint for retrieving data via SELECT WHERE for SMALL datasets.

VldbTableAccessor.find.large

/*+ORDERED USE_HASH(x d) FULL(d) FULL(x) SWAP_JOIN_INPUTS(x) */

Hint for retrieving data via SELECT WHERE for LARGE datasets.

VldbTableAccessor.find.main.small

/*+ ORDERED USE_NL(m j) */

Hint for retrieving data via SELECT LEFT JOIN for SMALL datasets.

VldbTableAccessor.find.main.large

/*+ ORDERED USE_HASH(m j) */

Hint for retrieving data via SELECT LEFT JOIN for LARGE datasets.

VldbTableAccessor.count.accuracy.mode

HIGH

Hint for controlling the accuracy of record counting.

If set to HIGH, an exact count of records is provided using a complex query that joins a temporary table with the main table. If set to LOW, an approximate count of records is provided using a simpler and faster query on the main table.

DB Persistence Parameters

The following parameters are usable only when DB Persistence (Persistence Layer, section DB Persistence) is used.

Name Default Value Description

nme.db.inList.batchSize

1 000

Batch size for inList input data.

nme.db.inList.commitSize

infinity

Commit size for inList input data.

nme.db.batchSize

100

Batch size for writing data in the Committing phase.

nme.db.commitSize

infinity

Commit size for writing data in the Committing phase.

nme.db.smallDataThreshold

10 000

Threshold for the number of records deciding if LARGE or SMALL data hints should be used.

nme.db.tinyDataThreshold

10

Threshold for number of records, numbers below are considered tiny and different access method is used - direct access instead of inList.

nme.db.alterTableLevel

addColumns,alterColumnsAlter

Defines what alter table commands will be executed if a table in database does not match the model. This is list of values, permitted values:

  • addColumns. Add missing columns

  • alterColumnsAlter. Alter existing columns that can be altered by DB DDL command, only extending varchar size and on some databases altering to clob is supported, that is it won’t change from small to large numbers (INTEGER → LONG). If database does not support alteration via DDL command, warning is written into log.

Hint Parameters

The parameters below make it possible to apply hints for some special computations being done during the processing.

The following hint examples are usable for Oracle 11+ database only.
Name Default Value Description

DbUnorderedInList.match

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving matched records for inLists in unordered lists.

DbUnorderedInList.unmatch

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving unmatched records for inLists in unordered lists.

DbOrderedInList.match

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving matched records for inLists in ordered lists.

DbOrderedInList.unmatch

/*+ORDERED USE_HASH(l d) SWAP_JOIN_INPUTS(l) FULL(l) FULL(d) */

Hint for retrieving unmatched records for inLists in ordered lists.

DbInList.main.hint

/*+ ORDERED USE_HASH(m j) */

Hint for retrieving data via SELECT LEFT JOIN.

Was this page useful?