Add a Streaming Event Handler

This article describes how to configure the Streaming Event Handler.

For an overview, see Adding a Data Event Handler and Publisher and Streaming Event Handler.

Before you start

Configure the instance layer model as described in Creating an Instance Layer Model using the top-down modeling approach.
Configure a master layer model as described in Creating a Master Data Layer Model using the top-down modeling approach.
Configure a database connection for event persistence (typically named eh_db) in your MDM Server Application Properties > Data source configuration. The relevant properties are ataccama.one.mdm.datasource.eh_db.*.
Create a ONE plan that will process the event data.

For more information, Creating a Logical Model.

Create a Streaming Event Handler

Each handler name must be unique. If multiple event handlers share the same name, the MDM Server fails to start.

Renaming a handler after events have been captured can result in data loss. Existing events in the old tables are not be automatically migrated to the new one.

To add a new Streaming Event Handler:

Go to Output Interfaces > Event Handlers (right-click) > New Event Handler Definition.
On the General tab, set the following parameters:
- Class - Select EventHandlerStreaming.
- Name - Provide a unique name (this becomes the database table prefix). See Streaming Event Handler > Persistence.
- Description (optional) - Description of the handler.
- Enable - Activate (true) or deactivate (false) the event handler.
- Event Capturing Scope: Filters events based on their origin (that is, the type of operation that produces the event). Select all that apply.
  - SERVICE - Events from online service calls (for example, service.processDelta, service.REST.*).
  - BATCH - Events from batch operations (for example, batchLoad., multiLoad., reprocess operations).
  - WEB_APPLICATION - Events from MDM Web App operations (for example, mdm.publish).
  - STREAM - Events from streaming interfaces (for example, stream.*).
  - BATCH_SERVICE - Events from SOAP batch invocations.
- Filter - Define a filter to preselect events (for example, only record updates). For details, see Filters.

Filters

Filters pre-select which events are processed.

Events that do not match the filter criteria are skipped and not saved to the database. This is critical for ensuring optimal performance and cost management as it reduces the required storage and processing times.

For example, if your organization is a financial services company, you might only want to capture changes to customer PII (personally identifiable information) or account status changes, ignoring routine balance updates that occur thousands of times per day. Similarly, an e-commerce platform might track order status changes but ignore shopping cart updates that are frequently abandoned.

These are further combined with the operation type filters selected in Event Capturing Scope.

If no filters are defined, changes on all columns are captured, even if the publisher is configured for only a subset of entity columns.

For example, if you are interested only in name changes, in addition to defining the name column in the publisher, you need to add a corresponding filter, such as old.src_name != new.src_name. For an overview of common issues with filtering, see Troubleshooting.

Filter types

When defining a filter for events, specify the following:

General Filter Expression - A Boolean ONE expression that filters based on event metadata. Only the meta dot-source is available here.
Example: Capture only INSERT and UPDATE events
```
meta.event_type = 'INSERT' OR meta.event_type = 'UPDATE'
```
Example: Capture changed records with specific values
```
new.country is in {'US', 'CA', 'MX'}
```
Filtered Entities - Specifies which entities to monitor. Each entity can have its own filter expression where meta, new, and old dot-sources are available.
Example: Capture only records where the status changed
```
new.status != old.status
```

Available metadata columns

Filter by metadata columns such as:

entity - Name of the entity.
event_type - Type of event: INSERT, UPDATE, or PURGE. See Event types.
layer - Layer of the entity (instance or master).
master_view - Name of the master view (master entities only).
origin - Origin of the data (instance entities only).
operation_type - Type of operation that created the event: SERVICE, BATCH, WEB_APPLICATION, STREAM, BATCH_SERVICE. See Event Capturing Scope in Create a Streaming Event Handler.
source_system - Source system of the data (instance entities only).

For a full list of available options, enter meta.. This also works with new. and old. dot-sources.

Event types

Event types are mapped to actions as follows:

INSERT Events - Standard record creation or UNDELETE (previously soft-deleted record restored).
UPDATE Events - Record modifications, with subtypes:
- UPDATE - Standard attribute changes.
- DEACTIVATE - Record deactivated (eng_active set to false).
- DELETE - Record soft-deleted (eng_existing set to false).
- REACTIVATE - Previously deactivated record reactivated.
PURGE Events - Hard deletion from the database (physical delete). The DELETE event type from Async Event Handler corresponds to PURGE.

Includes SYSTEM_PURGE and MANUAL_PURGE.

Configure publishers

For distributing events to external systems, the Streaming Event Handler supports event plan publishers, which execute a ONE plan to process and distribute event data.

The publisher uses a two-layer architecture, consisting of a wrapper for handling retry logic and a delegate for executing the ONE plan with event data (that is, the actual publisher).

You can define multiple publishers for a single event handler.

To configure a publisher:

In the Streaming Event Handler definition, switch to the Publishers tab.
Select Add.
Double-click the row number of the publisher to open the configuration.
Configure the following:
- Enable (Boolean) - Must be selected to use the publisher.
- Name - The name of the publisher. Used to refer to the publisher in the plan.
- Root Entity - The entity for which the publisher is configured.
  
  If using traversing, this defines the root entity for which the related entities are configured and the data retrieved. The root entity should be the top-level parent in the data tree that is sent to the ONE plan.
  
  Use Ctrl+Space to see a list of entities.
- All Columns (Boolean) - If selected:
  - Events for all columns in the selected entity are published. See also Filters.
  - Including meta columns and columns in this section returns an error.
- Max Retry Timeout: Maximum time (in seconds) to retry publishing a failed event using exponential backoff. After this timeout, the event is skipped.
- Publish Batch Size - Maximum number of root entities included in a single batch sent to the ONE plan. Large batches are automatically split into chunks of this size.
  
  The recommended range is between 100 and 10,000. Use a lower value for complex entities requiring traversing to prevent high memory usage. Use a higher value for simple, singular entities to maximize throughput.
- Meta Columns - Define the metadata columns that are provided in the publishing plan. Use Ctrl+Space to see a list of the available columns.
  
  Alternatively, select Fill Columns and choose all that apply.
- Entity Columns - Define the entity columns that are provided in the publishing plan.
  - Old Value (Boolean) - If selected, the original values for those columns are sent to the plan.
  - Actual Value (Boolean) - If selected, the current values for those columns are sent to the plan.
- Related Entities - If using traversing, choose the related entities you want to include in the publisher (only child entities of the defined root entity can be chosen). The depth of relations is dependent on your model.

Using the data in ONE plans

Your ONE plan can use these columns to:

Transform event data into the required output format.
Filter or route events based on metadata.
Enrich events with additional data from other sources.
Publish to external systems (Kafka, REST APIs, databases, and so on).

Example: Routing based on event type

You can route events to different outputs based on the meta_event_type column. For instance, use the condition meta_event_type = 'INSERT' to send new records to one destination and updates to another.

Example: Filtering out traversed records

The has_event column indicates whether a record actually triggered an event or was loaded via traversal. Use the condition has_event = true to process only records with actual changes, excluding traversed parent or child records that haven’t changed.

Common configuration patterns

The following examples illustrate some common configuration patterns. Make sure to adapt them to your specific use case before implementing.

Notifying systems of customer profile changes only

If you need to send only customer profile updates without related data:

Configure the root entity as party with master view masters.
Include only essential profile columns:
- Customer name
- Email address
- Active status
Set Publish Batch Size to 1000 for efficient throughput.
Set Max Retry Timeout to 60 seconds for quick failure handling.
Do not include child or parent entities.

This lightweight configuration minimizes payload size and maximizes throughput for high-volume profile updates.

Synchronizing complete customer records with relationships

If you need to send full customer data including addresses and contacts:

Configure the root entity as party with master view masters.
Include customer profile columns: name, date of birth, active status.
Add child entities:
- address (with relationship party_has_address) - include street, city, country columns.
- contact (with relationship party_has_contact) - include contact type and value columns.
Include the instance party entity with source tracking columns: source_id, source_system.
Set Publish Batch Size to 500 to better handle a larger payload.
Set Max Retry Timeout to 120 seconds to allow more time for complex data.

This configuration provides a complete customer 360 view for downstream systems like CRMs or data warehouses.

Was this page useful?