LogoLogo
  • README
  • VERSION 9
    • Version Begins
  • Overview
    • Show me the code!
    • Basic Concepts
    • Why Brighter?
  • Brighter Configuration
    • Basic Configuration
    • How Configuring the Command Processor Works
    • How Configuring a Dispatcher for an External Bus Works
    • RabbitMQ Configuration
    • AWS SNS Configuration
    • Kafka Configuration
    • Azure Service Bus Configuration
    • Azure Archive Provider Configuration
  • Darker Configuration
    • Basic Configuration
  • Brighter Request Handlers and Middleware Pipelines
    • Building an Async Pipeline of Request Handlers
    • Basic Configuration
    • How to Implement an Async Request Handler
    • Requests, Commands and an Events
    • Dispatching Requests
    • Dispatching An Async Request
    • Returning results from a Handler
    • Using an External Bus
    • Message Mappers
    • Routing
    • Building a Pipeline of Request Handlers
    • Passing information between Handlers in the Pipeline
    • Failure and Dead Letter Queues
    • Supporting Retry and Circuit Breaker
    • Failure and Fallback
    • Feature Switches
  • Guaranteed At Least Once
    • Outbox Support
    • Inbox Support
    • EFCore Outbox
    • Dapper Outbox
    • Dynamo Outbox
    • MSSQL Inbox
    • MySQL Inbox
    • Postgres Inbox
    • Sqlite Inbox
    • Dynamo Inbox
  • Darker Query Handlers and Middleware Pipelines
    • How to Implement a Query Handler
  • Health Checks and Observability
    • Logging
    • Monitoring
    • Health Checks
    • Telemetry
  • Command, Processors and Dispatchers
    • Command, Processor and Dispatcher Patterns
  • Under the Hood
    • How The Command Processor Works
    • How Service Activator Works
  • Event Driven Architectures
    • Microservices
    • Event Driven Collaboration
    • Event Carried State Transfer
    • Outbox Pattern
  • Task Queues
    • Using a Task Queue
  • FAQ
    • FAQ
  • END OF VERSION
    • Version Ends
  • VERSION 10
    • Version Begins
  • Overview
    • Show me the code!
    • Basic Concepts
    • Why Brighter?
  • Brighter Configuration
    • Basic Configuration
    • How Configuring the Command Processor Works
    • How Configuring a Dispatcher for an External Bus Works
    • RabbitMQ Configuration
    • AWS SNS Configuration
    • Kafka Configuration
    • Azure Service Bus Configuration
    • Azure Archive Provider Configuration
  • Darker Configuration
    • Basic Configuration
  • Brighter Request Handlers and Middleware Pipelines
    • Building an Async Pipeline of Request Handlers
    • Basic Configuration
    • How to Implement an Async Request Handler
    • Requests, Commands and an Events
    • Dispatching Requests
    • Dispatching An Async Request
    • Returning results from a Handler
    • Using an External Bus
    • Message Mappers
    • Routing
    • Building a Pipeline of Request Handlers
    • Passing information between Handlers in the Pipeline
    • Failure and Dead Letter Queues
    • Supporting Retry and Circuit Breaker
    • Failure and Fallback
    • Feature Switches
  • Guaranteed At Least Once
    • Outbox Support
    • Inbox Support
    • EFCore Outbox
    • Dapper Outbox
    • Dynamo Outbox
    • MSSQL Inbox
    • MySQL Inbox
    • Postgres Inbox
    • Sqlite Inbox
    • Dynamo Inbox
  • Darker Query Handlers and Middleware Pipelines
    • How to Implement a Query Handler
  • Health Checks and Observability
    • Logging
    • Monitoring
    • Health Checks
    • Telemetry
  • Command, Processors and Dispatchers
    • Command, Processor and Dispatcher Patterns
  • Under the Hood
    • How The Command Processor Works
    • How Service Activator Works
  • Event Driven Architectures
    • Microservices
    • Event Driven Collaboration
    • Event Carried State Transfer
    • Outbox Pattern
  • Task Queues
    • Using a Task Queue
  • FAQ
    • FAQ
  • END OF VERSION
    • Version Ends
Powered by GitBook
On this page
  • Producer Correctness
  • The Outbox Pattern

Was this helpful?

Edit on GitHub
  1. Event Driven Architectures

Outbox Pattern

PreviousEvent Carried State TransferNextUsing a Task Queue

Last updated 1 year ago

Was this helpful?

Producer Correctness

When a microservice changes the state for which it is the system of record, and then signals to subscribers via an event that it has changed its state, how do we ensure that subscribers receive the event and are therefore consistent with the producer?

The system of record may become inconsistent with downstream consumers because after writing changes to an entity, we may fail to publish the corresponding message. Our connection to the message-oriented middleware (MoM) may fail, or MoM may fail.

The new record is saved to the backing store, but the event is not raised so subscribing systems become inconsistent. We have a lost send.

CorrectnessProblem

Distributed Transactions may seem like an answer, but possess two issues. First, we are probably using a backing store and message-oriented middleware from different vendors or OSS projects that don't support the same distributed transaction protocol. Second, distributed transactions don't scale well.

We might naively try to fix this by sending the message first, then updating the backing store if that succeeds. But this won't necessarily work either, as we might fail to write to the database.

The new record is posted to downstream systems, but the local database call is rejected, and so the upstream system is now inconsistent. A phantom send.

In either solution we might simply decide that the best option is to ensure that we can retry what is hopefully a transient error. This may solve the problem in many instances, and is a good first step. But an endless retry loop has its own dangers, consuming resources and reducing throughput, and if the app crashes we will still only be partially complete. So it cannot guarantee delivery of the message that matches the write.

The Outbox Pattern

In the Outbox pattern, we use the ACID properties of an RDBMS. We write not only to the table that stores the entity that we are inserting, updating, or deleting, but also we write the message we intend to send to an 'outbox table' in the same Db.

We mark the time that the message was written, as part of the transaction, on the record.

Then when we send the message via the Broker, we mark the message as dispatched in the table.

An out-of-band sweeper process can then run, and query for messages that have not been sent within a time window (their written date is over a threshold of milliseconds ago, and they have no dispatched time stamp). It then resends those messages. If it sends them, it marks them as dispatched. As the sweeper process keeps polling for messages that have not yet been sent, we will eventually send all the messages. So we have guaranteed delivery but eventual consistency.

It is possible that the write to the row to update the dispatched status will fail. It is not in a transaction between broker and RDBMS either. If that happens, we may send the message twice.

For this reason, the Outbox pattern offers us guaranteed, at least once delivery. Consumers must be prepared for this. Either they can use an Inbox, which records all the messages they have seen recently and discards duplicates, or they must be idempotent and the result of processing the message twice has no side-affects.

OutboxPattern

See for more on how to ensure Producer-Consumer correctness in Brighter.

Brighter Outbox Support