Streaming Event Handler

The Streaming Event Handler (EventHandlerStreaming) is a high-performance event handler implementation designed for optimal throughput and simplified configuration. Recommended for all new implementations (available from version 16.3.0 onwards).

The handler combines processors, adapters, and traversing logic into a single component and brings up to 15 times faster processing compared to the legacy implementation. It also features:

Optimized database operations and batch processing (small transactions are grouped together while larger ones are split into chunks).
Enhanced filtering (including operation types and event subtypes).
Modern retry strategy with exponential backoff instead of fixed delays.
Real-time observability in the MDM Web App Admin Center.

Events can be persisted in a database only.

This article describes how the Streaming Event Handler works and the essential concepts for implementation. For configuration instructions, see Add a Streaming Event Handler.

Processing flow

The Streaming Event Handler processes events through several stages.

When a data operation completes, the handler captures change events and applies session-level and event-level filters to determine which events to persist. Filtered events are stored in the eh_db database. Sessions with no events after filtering are skipped to improve processing efficiency.

The handler then collects events into batches based on either time (streaming seconds) or volume (streaming size) triggers—whichever threshold is reached first.

For handlers configured with traversing, the processor groups related events and retrieves any missing data from the main database to build complete record sets. For handlers without traversing, events are passed directly to the publisher.

Finally, batched events are sent to the configured ONE plan for transformation and distribution to external systems.

Processing and batching

The processor controls how events are grouped and when they are sent for publishing.

By default, the handler captures changes to all columns in monitored entities unless filters are configured to narrow the scope.

How processing works

The handler uses an intelligent approach to event processing. Events are split into smaller chunks when they exceed a certain size, which prevents overwhelming downstream systems and ensures reliable processing. The chunk size is controlled by the publisher configuration (Publish Batch Size).

The traversing service ensures loading relevant context for events by:

First looking for related events within the entire session’s temporary storage.
Then fetching any missing data from the main database.
Assembling a complete payload that prioritizes the most recent event data.

Publisher data model (traversing)

The publisher data model defines a hierarchical, tree-like structure that determines what a complete "message" looks like for your downstream systems. You define a root entity and its relationships, and the handler automatically traverses these connections to gather all related data.

When an event occurs on any entity within this model, the handler collects the complete dataset and provides it to the corresponding Integration Input steps in the plan.

For a customer 360 view, you might define root as Customer (party) and include addresses, contacts, and accounts. For an order fulfillment system, you might root at Order and include line items, shipping addresses, and payment methods.

Design the model to match what your downstream consumers need in a single, atomic message.

How traversing works

When processing events, the handler:

Groups related events - Identifies all events in the current batch that belong to the same data tree.
Collects event data - Gathers old and new values from all related events within the temporary event storage (eh_db).
Fetches missing data - If any entities or columns defined in the model are missing from the events, queries the main MDM database (mdc_db) to fetch the current state.
Assembles payload - Combines event data and fetched data into a complete, consistent payload for the ONE plan.

Records loaded via traversal (not directly from events) are marked with the has_event metacolumn set to false. Records that triggered events have has_event set to true. This is a special metadata column (has_event) added to distinguish between the record that triggered the event from records loaded by traversal.

Because the handler operates asynchronously, data traversed from the main database might have been updated by another transaction during event processing. The handler ensures data becomes consistent over time, but the data in events might not exactly match the database state at the moment the original change occurred.

Multi-level traversing

Entities can be nested to traverse multiple relationship levels. While powerful, multi-level traversing impacts performance. For example, a customer service application might traverse Customer > Contact > Contact History (three levels), while a simple notification system might only need Customer > Email Address (two levels).

Balance completeness with performance based on your use case.

Model structure

The root entity can be a master or instance entity:

Master entity as root: Can have both other master and instance entities as related entities.
Instance entity as root: Can have other instance entities as related entities.

You can also define specific columns for traversing and processing (and in case of master entities, the master view), which optimizes performance by reducing how much data is transferred to the publisher.

To trigger publishing, at least one record in each group of related records must contain a change event. The record group is always consistent and complete based on the defined publisher model configuration (that is usually a subset of the MDM logical model).

For records loaded via traversal (where has_event is false), the old_* metadata columns are populated with the same values as new_* columns, and certain meta_* columns are set to null.

Retrying

The Streaming Event Handler uses an exponential backoff mechanism for retrying.

When a batch fails to publish, it is automatically split into individual events.
Each event is retried individually with exponentially increasing delays (for example, 1s, 2s, 4s, 8s, 16s…). The delay is calculated using exponential backoff with the formula 2^(AttemptNumber-1).
If an event fails to publish within the allowed period (Max Retry Timeout), it is skipped.
The handler continues processing other events rather than blocking the entire pipeline. This helps prevent overwhelming the application with publishing failures.

Retrying is done on the publisher level.

Retry configuration balances reliability versus throughput. A mission-critical integration might use Max Retry Timeout set to 300 to retry for five minutes, while a best-effort notification system might use maxRetryTimeout="30" to fail fast and move on.

Consider your downstream system’s SLA and recovery capabilities.

Persistence

The Streaming Event Handler uses database persistence exclusively. Events are temporarily stored in dedicated tables before being processed and published.

By default, the eh_db database is used for this purpose. For details, see MDM Server Application Properties > Data source configuration.

The relevant tables are prefixed using the handler name, for example: MDM_Account_Upsert_event, MDM_Account_Upsert_session.

Monitoring and observability

The Streaming Event Handler provides enhanced visibility into the event processing pipeline. To view information about the event handler, go to MDM Web App Admin Center > Interfaces > Event Handlers screen.

For details about event publishing, see MDM Web App Admin Center > Monitoring > Event publishing.

State management

The handler state can be controlled via REST API.

To get the handler state, send a GET request to the following endpoint:

GET /api/rest/event-handlers/{handler_name}/state

To update the handler state, send a PATCH request to the following endpoint:

PATCH /api/rest/event-handlers/{handler_name}/state

The request should include a Content-Type=application/json header and payload with the new handler state:

{"state": "ACTIVE"}

Available states:

ACTIVE - Handler is running and processing events.
INACTIVE - Handler is temporarily stopped (existing events remain in the queue).
DISABLED - Handler is turned off (events are not captured).

Troubleshooting

Handler fails to start

Problem

The event handler enters FAILED state on server startup.

Some common causes include:

The eh_db database connection is unavailable or misconfigured.
Duplicated handler names.
Invalid ONE expression in the filter configuration.

Solution

Check server logs for specific error messages.
Verify the eh_db data source is configured in MDM runtime configuration and that the database is running and available.
Ensure the handler name is unique across all event handlers.
Validate filter expressions using ONE expression syntax.

Events not being captured

Problem

The event handler is active but no events are saved to the database.

This can occur because:

Filters are too restrictive.
Events are being generated by operations not included in operation types (that is, Event Capturing Scope is too limited).
Entity not included in filtered entities list.

Solution

Temporarily remove all filters to verify events are being generated.
Check the operation type of the data load operation.
Review entity filter configuration.
Enable DEBUG logging to see which events are filtered out.

Too many events being captured

Problem

The event handler is publishing too many events and overwhelming the downstream systems.

This might happen if filters are too permissive. By default, all column changes are captured, which can result in a large number of events produced depending on the specific entity.

Solution

Review and adjust your filtering configuration:

Add entity-specific filters to capture only business-relevant column changes (for example, old.src_name != new.src_name for name changes).
Use the general filter expression to exclude system columns or routine updates.
Test with a subset of data to verify the filter reduces event volume as expected.

Migrating from Async Event Handler

For migration guidance from the legacy Async Event Handler, see Migrating to Streaming Event Handler.

Was this page useful?