Building Our Own ESB - Publish / Subscribe (Part 2)
Almost Ready to Code
Before we dig into the code, let's talk at a high level about the various parts of our system.
Reliable Delivery
As we discussed previously, the fact that our systems are distributed presents some interesting problems. How do we ensure that the message makes it to each of our publishers if we know that they might be down when we try to deliver the message? Rather than attempting to deliver the message immediately, we will determine which subscribers need a copy of our message, and then place a copy of the message for each of them into a delivery queue1.
Meanwhile, another process will be watching our delivery queue and attempting to deliver each message. If the message cannot be delivered for any reason, it will be moved from our primary queue to a retry queue. The reason for the second queue is that we don't want bad messages clogging up our system or impacting the delivery time of good messages. For this reason, there will be two important differences between messages in our primary queue and our retry queue. The first is that messages in the retry queue will be marked for delayed delivery. If our subscriber is down temporarily, we need to give it a reasonable amount of time to come back up before we retry. The second important difference is that delivery of messages in the retry queue will be throttled. There is a chance that the reason the other system could not respond is that it is overloaded. We don't want to add to the problem by pummelling it with retry messages.
If after a predetermined number of attempts we still have not managed to deliver the message, it will be placed in a failure queue (commonly known as a dead-letter or poison message queue). Messages in the failure queue will not be automatically processed, so we will need a way to manage them manually.
Communication Made Easy
A few years ago, supporting multiple communication protocols would have been a major undertaking. With WCF, it is a breeze. Out of the box, we have support for named pipes, queued communication, basic SOAP webservices, full blown WS-* web services, TCP/IP to name a few. Each of these protocols (bindings in WCF speak) can be further configured and extended, giving you hundreds of options out of the box. Not only does WCF allow us to send messages via all these protocols, it also allows us to host these protocols in our own applications with less than 10 lines of code. As a result, WCF will be our platform of choice for handling the communication portions of or ESB.
Here's a high level diagram of how communication will flow through our system at this point:
Extensibility
We will be using MSMQ, WCF, and SQL Server in our ESB implementation. However, there is no good reason that our ESB should be tightly bound to these technologies. For example, we might want to store subscriptions in a configuration file instead of SQL server or we might just want to use an in memory queue instead of MSMQ. We will keep this in mind while building our ESB and try to provide a model that allows customization without direct modification of any of the core code or libraries.
Tomorrow, we'll get to the code.
[1] For performance reasons, we might want to store a single copy of our message in a database and then reference that message from the queue. One of the advantages of this approach is that it cuts down significantly on the serialization cost when placing messages into the queue. Additionally, it can help you get around the 4 MB size restriction placed on messages inside MSMQ or decrease the memory footprint of your application by allowing you to stream messages instead of keeping the entire message in memory.