Designing Events and Event Streams: Engineering Principles
Source: Building and Designing Events and Event Streams with Apache Kafka — Confluent, an IBM Company
https://youtu.be/zOWJ5hAcp9Y
⟐ ep 548 Before Choosing a Broker: Preparing the Question Together
Summary
Adam Bellemare joins Kris Jenkins to detail the architectural principles required for modeling events and event streams in distributed systems. Rather than focusing merely on storage mechanics, Bellemare outlines four key design dimensions: internal vs. external data boundaries, facts vs. deltas, normalization vs. denormalization, and discrete vs. continuous event flows.
Aspects
Internal vs. External Data Boundaries (04:46-06:30)
System architectures must isolate internal data models from external shared interfaces [05:20-06:30]. By creating an anti-corruption abstraction layer over private database state, producer services safely evolve internal storage mechanisms without breaking downstream contracts across domain boundaries [06:05-06:29].
Facts vs. Delta Events (10:18-15:40)
Fact events represent a complete snapshot of domain state, easing downstream consumer processing without requiring state reconstruction [10:28-10:55, 15:08-15:24]. Delta events record incremental modifications, reducing producer burden but requiring consumers to aggregate history to compute final state [10:55-11:39, 15:25-15:40].
Denormalization at the Stream Boundary (22:54-28:35)
While relational databases excel at multi-table joins, relational normalization creates difficult stream-join overhead in event architectures [23:00-24:35]. Pre-denormalizing datasets via outbox patterns or stream processors like ksqlDB produces readily consumable domain entities for diverse polyglot clients [28:15-28:30, 32:28-32:45].
Topic Typing and Event Cardinality (33:40-36:30)
Fact-based streams work best with a single event schema per topic for straightforward materialized table creation [34:00-34:15, 35:00-35:30]. Conversely, complex sequence workflows and event-sourced entities benefit from multi-event polymorphic topics to preserve strict partition ordering across state transitions [35:45-36:28].
Discrete vs. Continuous Workflows (40:43-46:45)
Discrete workflows possess deterministic lifecycles with explicit completion boundaries, such as purchase orders [40:43-44:55]. Continuous workflows evaluate unending time-series streams or sensor measurements, constantly reacting to thresholds and updating state without terminal endpoints [45:57-46:45].
End-results and Core Aspect
Core Aspect — Domain-Driven Event Stream Modeling (05:56-09:58)
The core implementation objective is establishing explicit, decoupled event contracts that accurately reflect domain semantics while isolating internal database implementations from public stream consumers [06:05-06:29, 09:38-09:58].
Results
- Decoupled Cross-Team Interfaces (06:05-08:20): Teams publish curated external facts rather than raw internal table mutations, maintaining compatibility across organizational boundaries while evolving private domain logic independently [06:05-06:29, 08:08-08:20].
- Simplified Consumer State Aggregation (27:00-32:45): Publishing denormalized domain objects reduces downstream computational overhead, eliminating the need for complex, brittle multi-stream stateful joins across microservices [27:00-27:50, 32:28-32:45].
- Strict Sequential Lifecycle Ordering (35:45-45:15): Co-locating multiple event types into unified partitioned topics guarantees causal ordering for event-sourced entities and stateful BPMN workflows [35:45-36:28, 44:45-45:15].