Sponsor: Do you build complex software systems? See how NServiceBus makes it easier to design, build, and manage software systems that use message queues to achieve loose coupling. Get started for free.
So, you need consistency between making state changes to your database and publishing events to a message broker for other services to consume. A common pattern to handle this situation is the Outbox Pattern, but another pattern called Listen to Yourself tries to solve the consistency issue. Does it work? Well lets find out.
YouTube
Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.
Problem: Consistency
Here’s the problem. A possible consistency problem exists because you’re interacting and trying to perform two non-atomic operations. You need to save data to your database and, in a different operation, publish an event to a message broker or event stream. Let’s say you’re saving a new Order to your database, and you need to publish an OrderPlaced event so other downstream services can react.
The first operation saving to your database could succeed.
However, publishing the PlaceOrder event to the message broker could fail.
Now, your system is likely in an inconsistent state as other downstream services are unaware that an order was placed. When events become first-class citizens of your system, and you’re driven by events, you need the the guarantee that if you make a state change that the relevant events are guaranteed to be published.
One of the most common solutions to this problem is using the Outbox Pattern. However, the listen to yourself pattern also tries to solve this differently.
Listen to Yourself Pattern
The way the listen to yourself pattern works is by first publishing an event and then listening to (consuming) that event before it makes the state change. It’s reversing the order of operations.
This means we would first publish an OrderPlaced event to our broker.
Then, we would consume that message. In other words, the publisher is also the subscriber/consumer.
When we are consuming and handling the message, we than can add the Order to our database or make whatever state changes are required.
This “solves” the consistency issue because if we never publish the event, we can never consume it to have an Order. If we publish the event but fail to consume it and make our state changes to our database, the assumption (which is fair) is that we can retry and re-process messages. When handling messages from a message broker, you generally have many more ways to handle failures and re-processing.
Issues
I used “solved” in quotes because I do not believe the listen to yourself pattern has enough benefits compared to its trade-offs. Some of these issues are because of semantics.
Events are to indicate something that has already occurred. If you were to place an OrderPlaced event, is that really what happened? No, rather, you intend to place an order. When has the Order truly been placed in the system? It is likely when you have a recorded record of it in your database.
Now, you can argue that publishing the OrderPlaced event is a request. A request to place an order. Which really is a command, not an event.
Using the guide above, you could reframe the event name and call it a PlaceOrderRequested. That’s fair, but there’s also another issue.
Consumers have zero or many consumers.
This means you could have other consumers simultaneously handle the OrderPlaced or PlaceOrderRequested event as “yourself”.
Event-driven architecture lets you remove the temporal coupling between publisher and consumer(s) when using something like a message broker or event stream. But since we could have many consumers if an OrderPlaced event occurs, are other consumers waiting for “yourself” to first process the event? There’s no order of operation between consumers.
Consumers aren’t, nor should they be, dependent on each other. They should operate independently and freely from each other.
You also can’t force there to be a single consumer for an event. That defeats the purposes of events and publish/subscribe.
Forcing an event to act as the command is a hammer-and-nail situation.
Alternatives
The most common approaches for handling consistency is using the outbox pattern, having a fallback, and workflow engine/platforms.
I cover all three of these alternatives and how they work in a post: Don’t Fail Publishing Events!
In most cases, I’d recommend these ahead of the listen to yourself pattern.
Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.