Add a Streaming Event Handler

This article describes how to configure the Streaming Event Handler.

For an overview, see Adding a Data Event Handler and Publisher and Streaming Event Handler.

Before you start

Configure the instance layer model as described in Creating an Instance Layer Model using the top-down modeling approach.
Configure a master layer model as described in Creating a Master Data Layer Model using the top-down modeling approach.
Configure a database connection for event persistence (typically named eh_db) in your MDM Server Application Properties > Data source configuration. The relevant properties are ataccama.one.mdm.datasource.eh_db.*.
Create a ONE plan that will process the event data.

For more information, Creating a Logical Model.

Create a Streaming Event Handler

Each handler name must be unique. If multiple event handlers share the same name, the MDM Server fails to start.

Renaming a handler after events have been captured can result in data loss. Existing events in the old tables are not automatically migrated to the new one.

To add a new Streaming Event Handler:

Go to Output Interfaces > Event Handlers (right-click) > New Event Handler Definition.
On the General tab, set the following parameters:
- Class - Select EventHandlerStreaming.
- Name - Provide a unique name (this becomes the database table prefix). See Streaming Event Handler > Persistence.
- Description (optional) - Description of the handler.
- Enable - Activate (true) or deactivate (false) the event handler.
- Streaming - Configure whether streaming is used. See Streaming.
- Event Capturing Scope - Filters events based on their origin (that is, the type of operation that produces the event). Select all that apply.
  - SERVICE - Events from online service calls (for example, service.processDelta, service.REST.*).
  - BATCH - Events from batch operations (for example, batchLoad., multiLoad., reprocess operations).
  - WEB_APPLICATION - Events from MDM Web App operations (for example, mdm.publish).
  - STREAM - Events from streaming interfaces (for example, stream.*).
  - BATCH_SERVICE - Events from SOAP batch invocations.
- Filter - Define a filter to preselect events (for example, only record updates). For details, see Filters.

Streaming

Configure the batching behavior:

Enable - Activate (true) or deactivate (false) streaming.
Seconds - Maximum time (in seconds) to wait before triggering a batch, even if the volume threshold has not been reached. Acts as a latency guarantee.

Default: 60.
Size - Maximum number of records in the configured root entity that are collected before triggering a batch.

Default: 1000.

The handler triggers a batch based on whichever threshold is reached first.

For guidance on setting these values, see Best practices for configuring streaming event handler.

Filters

Filters pre-select which events are processed.

Events that do not match the filter criteria are skipped and not saved to the database. This is critical for ensuring optimal performance and cost management as it reduces the required storage and processing times.

For example, if your organization is a financial services company, you might only want to capture changes to customer PII (personally identifiable information) or account status changes, ignoring routine balance updates that occur thousands of times per day. Similarly, an e-commerce platform might track order status changes but ignore shopping cart updates that are frequently abandoned.

These are further combined with the operation type filters selected in Event Capturing Scope.

If no filters are defined, changes on all columns are captured, even if the publisher is configured for only a subset of entity columns.

For example, if you are interested only in name changes, in addition to defining the name column in the publisher, you need to add a corresponding filter, such as old.src_name != new.src_name. For an overview of common issues with filtering, see Troubleshooting.

Filter types

When defining a filter for events, specify the following:

General Filter Expression - A Boolean ONE expression that filters based on event metadata. Only the meta dot-source is available here.
Example: Capture only INSERT and UPDATE events
```
meta.event_type = 'INSERT' OR meta.event_type = 'UPDATE'
```
Example: Capture changed records with specific values
```
new.country is in {'US', 'CA', 'MX'}
```
Filtered Entities - Specifies which entities to monitor. Each entity can have its own filter expression where meta, new, and old dot-sources are available.
Example: Capture only records where the status changed
```
new.status != old.status
```

Available metadata columns

Filter by metadata columns such as:

entity - Name of the entity.
event_type - Type of event: INSERT, UPDATE, DELETE, or PURGE. When writing filters that match deleted records, note that the Streaming Event Handler distinguishes purge (PURGE) from delete (DELETE).

For semantics and the full MDM-operation-to-event_type mapping, see Event types and semantics.
layer - Layer of the entity (instance, master, or match_proposal).
master_view - Name of the master view (master entities only).
origin - Origin of the data (instance entities only).
operation_type - Type of operation that created the event. See Event types and semantics for possible values.
source_system - Source system of the data (instance entities only).

For a full list of available options, enter meta.. This also works with new. and old. dot-sources.

Configure publishers

For distributing events to external systems, the Streaming Event Handler supports event plan publishers, which execute a ONE plan to process and distribute event data.

The publisher uses a two-layer architecture, consisting of a wrapper for handling retry logic and a delegate for executing the ONE plan with event data (that is, the actual publisher).

You can define multiple publishers for a single event handler.

To configure a publisher:

In the Streaming Event Handler definition, switch to the Publishers tab.
Select Add.
Double-click the row number of the publisher to open the configuration.
Configure the following:
- Enable (Boolean) - Must be selected to use the publisher.
- Name - The name of the publisher. Used to refer to the publisher in the plan.
- Root Entity - The entity for which the publisher is configured.
  
  If using traversing, this defines the root entity for which the related entities are configured and the data retrieved. The root entity should be the top-level parent in the data tree that is sent to the ONE plan.
  
  Use Ctrl+Space to see a list of entities.
  
  A matching proposal entity can also be selected as the root. Such a root is always a leaf: it cannot have related entities, and it cannot be used as a related entity of an instance or master root. Columns are selected as for any other root, but Related Entities does not apply to it. See Matching proposal root.
- All Columns (Boolean) - If selected:
  - Events for all columns in the selected entity are published. See also Filters.
  - Including meta columns and columns in this section returns an error.
- Max Retry Timeout: Maximum time (in seconds) to retry publishing a failed event using exponential backoff. After this timeout, the event is skipped.
- Publish Batch Size - Maximum number of root entities included in a single batch sent to the ONE plan. Large batches are automatically split into chunks of this size.
  
  The recommended range is between 100 and 10,000. Use a lower value for complex entities requiring traversing to prevent high memory usage. Use a higher value for simple, singular entities to maximize throughput.
- Meta Columns - Define the metadata columns that are provided in the publishing plan. Use Ctrl+Space to see a list of the available columns.
  
  Alternatively, select Fill Columns and choose all that apply.
- Entity Columns - Define the entity columns that are provided in the publishing plan.
  - Old Value (Boolean) - If selected, the original values for those columns are sent to the plan.
  - Actual Value (Boolean) - If selected, the current values for those columns are sent to the plan.
- Related Entities - If using traversing, choose the related entities you want to include in the publisher (only child entities of the defined root entity can be chosen). The depth of relations is dependent on your model.

Matching proposal root

A publisher can be rooted at a matching proposal entity. Matching proposals are potential duplicates captured by automated matching that await a manual decision (for an overview, see Matching Proposals).

The entity is identified by its flat name in the format <instance>_proposal_<matching>, where <matching> is the matching name (k by default), for example, party_proposal_k. This is the same name used by the Async Event Handler.

In the generated configuration, this is a <matchProposal> element placed directly under <root>. See Event Handler > StreamingRetryingPublisher.

No traversing is performed from a matching proposal root, so no related instance or master records are fetched. Consumers that need them must join on record_id, related_record_id, master_id, or related_master_id themselves.

Columns are selected as for an instance or master root. With All Columns selected, all the following columns are published:

Entity columns:
- type
- confidence
- master_id
- related_master_id
- origin
- match_rule_name
- is_rejected
- record_id
- related_record_id
- match_quality (obsolete but still persisted)
- eng_active
Meta columns:
- meta_id
- meta_event_type
- meta_entity
- meta_layer
- meta_master_view
- meta_operation_type
- meta_activation_date
- meta_creation_date
- meta_deactivation_date
- meta_deletion_date
- meta_last_update_date
- meta_activation_tid
- meta_creation_tid
- meta_deactivation_tid
- meta_deletion_tid
- meta_last_update_tid

Every entity column is published as an old_/record_ pair, including eng_active. The only exception is id, which appears only as meta_id.

meta_master_view is always null. It is kept for payload parity with the Async Event Handler.

Meta columns that are never set for a proposal event are not published: meta_origin, meta_source_system, meta_modified_by, meta_last_source_update_date, meta_last_source_update_tid, meta_source_timestamp, and meta_nature.

All event types are supported as for an instance or master root. Similarly to instance and master roots, PURGE events carry a narrower payload: only meta_id, meta_entity, meta_layer, meta_master_view, meta_event_type, has_event, and the old_/record_ entity column pairs.

Because a publisher has exactly one root, each matching proposal entity needs its own publisher with its own plan file. A single event handler can host all of them, alongside instance and master publishers, fed from the same batch.

Using the data in ONE plans

Your ONE plan can use these columns to:

Transform event data into the required output format.
Filter or route events based on metadata.
Enrich events with additional data from other sources.
Publish to external systems (Kafka, REST APIs, databases, and so on).

Example: Routing based on event type

You can route events to different outputs based on the meta_event_type column. For instance, use the condition meta_event_type = 'INSERT' to send new records to one destination and updates to another.

Example: Filtering out traversed records

The has_event column indicates whether a record actually triggered an event or was loaded via traversal. Use the condition has_event = true to process only records with actual changes, excluding traversed parent or child records that haven’t changed.

Common configuration patterns

The following examples illustrate some common configuration patterns. Make sure to adapt them to your specific use case before implementing.

Notifying systems of customer profile changes only

If you need to send only customer profile updates without related data:

Configure the root entity as party with master view masters.
Include only essential profile columns:
- Customer name
- Email address
- Active status
Set Publish Batch Size to 1000 for efficient throughput.
Set Max Retry Timeout to 60 seconds for quick failure handling.
Do not include child or parent entities.

This lightweight configuration minimizes payload size and maximizes throughput for high-volume profile updates.

Synchronizing complete customer records with relationships

If you need to send full customer data including addresses and contacts:

Configure the root entity as party with master view masters.
Include customer profile columns: name, date of birth, active status.
Add child entities:
- address (with relationship party_has_address) - include street, city, country columns.
- contact (with relationship party_has_contact) - include contact type and value columns.
Include the instance party entity with source tracking columns: source_id, source_system.
Set Publish Batch Size to 500 to better handle a larger payload.
Set Max Retry Timeout to 120 seconds to allow more time for complex data.

This configuration provides a complete customer 360 view for downstream systems like CRMs or data warehouses.

Was this page useful?