Dynamo Outbox
Reference · Applies to Brighter V10
DynamoDb Outbox Usage
The DynamoDb Outbox allows integration between DynamoDb and Brighter's outbox support. The configuration is described in Command Processor Configuration Reference.
To support transactional messaging when using DynamoDb requires us to use DynamoDb's support for ACID transactions. You should understand best practices for using transactions with DynamoDb.
For this we will need the Outbox package for DynamoDb:
AWS SDK v3 (legacy support):
Paramore.Brighter.Outbox.DynamoDB
AWS SDK v4 (recommended for new projects):
Paramore.Brighter.Outbox.DynamoDB.V4
Paramore.Brighter.Outbox.DynamoDB (or .V4) will pull in another package:
Paramore.Brighter.DynamoDb (or Paramore.Brighter.DynamoDb.V4)
The examples on this page use the v3 packages' namespaces. The v4 packages append .V4 to theirs: Paramore.Brighter.Outbox.DynamoDB.V4 and Paramore.Brighter.DynamoDb.V4, which also holds DynamoDbTableFactory.
See AWS SQS Migration for migration guidance between v3 and v4.
As described in Command Processor Configuration Reference, we configure Brighter to use an outbox by setting Outbox on the options passed to AddProducers().
As we want to use DynamoDb with the outbox, we also set ConnectionProvider and TransactionProvider — both to DynamoDbUnitOfWork, which is one type playing both parts — so that we can share your transaction scope when persisting messages to the outbox.
using System;
using Amazon.DynamoDBv2;
using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter;
using Paramore.Brighter.DynamoDb;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.Outbox.DynamoDB;
using Paramore.Brighter.Outbox.Hosting;
public void ConfigureServices(IServiceCollection services)
{
// ... dynamoDb is your IAmazonDynamoDB client, producerRegistry your transport
services.AddSingleton<IAmazonDynamoDB>(dynamoDb);
services.AddBrighter()
.AddProducers(configure =>
{
configure.ProducerRegistry = producerRegistry;
configure.Outbox = new DynamoDbOutbox(
dynamoDb, new DynamoDbConfiguration(), TimeProvider.System);
configure.ConnectionProvider = typeof(DynamoDbUnitOfWork);
configure.TransactionProvider = typeof(DynamoDbUnitOfWork);
})
.UseOutboxSweeper()
.AutoFromAssemblies();
}
The handler below is AddGreetingHandlerAsync from the Brighter sample at Brighter/samples/WebAPI/WebAPI_Dynamo/, which also declares the AddGreeting request, the Person entity and the GreetingMade event it uses.
In our handler we take a dependency on Brighter's IAmADynamoDbTransactionProvider interface, which DynamoDbUnitOfWork implements. We explicitly start a transaction within the handler on the Database within that provider.
We call DepositPostAsync within that transaction to write the message to the Outbox. Once the transaction has closed we can call ClearOutboxAsync to immediately clear, or we can rely on the Outbox Sweeper, if we have configured one to clear for us. (There are equivalent synchronous versions of these APIs).
Running more than one instance? Configure a distributed lock so only one Sweeper (and Archiver) runs at a time — see DynamoDB Distributed Lock.
DynamoDb Outbox Options
DynamoDbConfiguration names the table and its five global secondary indexes, and carries the two settings that govern expiry and scan concurrency.
TableName
string
"brighter_outbox"
Names the DynamoDB table that holds the Outbox.
DeliveredIndexName
string
"Delivered"
Names the global secondary index of dispatched messages, indexed by topic.
DeliveredAllTopicsIndexName
string
"DeliveredAllTopics"
Names the global secondary index of dispatched messages covering every topic.
OutstandingIndexName
string
"Outstanding"
Names the global secondary index of undispatched messages, indexed by topic.
OutstandingAllTopicsIndexName
string
"OutstandingAllTopics"
Names the global secondary index of undispatched messages covering every topic.
CausationIndexName
string
"Causation"
Names the global secondary index over the Causation Id, which Replay On Seen queries; leave it at the default, for the reason under The index name.
TimeToLive
TimeSpan?
null
Sets how long a message survives in the table before DynamoDB expires it; messages do not expire when it is null.
ScanConcurrency
int
3
Sets how many segments a parallel scan for outstanding messages uses.
timeout and numberOfShards are constructor parameters, not properties you can set. They surface as the get-only Timeout and NumberOfShards. numberOfShards defaults to 3 and shards the outstanding index so an active topic does not build a hot partition; the Outbox throws ArgumentOutOfRangeException above 20. timeout is the deadline, in milliseconds, for each Outbox operation, and defaults to 500; zero or -1 disables it, and anything below -1 throws ArgumentOutOfRangeException. The deadline covers the whole operation, batch items and query pages included, and an operation that overruns it is cancelled and surfaces as OperationCanceledException. The outBoxTimeout argument the IAmAnOutbox methods take overrides it for one call: -1, the default, uses the configured timeout, and zero disables the deadline. Timeouts you configure on the AmazonDynamoDBClient still apply alongside it. tableName and scanConcurrency are constructor parameters as well, but each has a settable property and is in the table above.
Replay Support: The Causation Index
Requires Brighter 10.8.0 or later. Earlier versions of both DynamoDB Outbox packages have no
DynamoDbConfiguration.CausationIndexNameand noCausationIdattribute.
Replay On Seen resends every Outbox message produced under a given Causation Id. On DynamoDB that means querying on a non-key attribute, which needs a Global Secondary Index — by default one named Causation, with CausationId as its hash key.
This applies to both Paramore.Brighter.Outbox.DynamoDB and .V4. The DynamoDB Inbox needs nothing: it looks a Causation Id up by the table's own primary key.
A new table gets the index for free
MessageItem.CausationId is decorated with [DynamoDBGlobalSecondaryIndexHashKey(indexName: "Causation")], and DynamoDbTableFactory reflects over those attributes when it builds the request. So a table you create through the factory already has the index — you only add a throughput entry for it:
Adding the index to an existing table
A table provisioned before replay shipped does not have the index, and there is no migration runner for DynamoDB. Add it yourself with UpdateTable:
Two operational notes:
The index backfills asynchronously. DynamoDB populates a new GSI in the background, and the index reports
CREATINGuntil it finishes. Until then a replay may find only part of a causation's messages, so wait for the index to becomeACTIVEbefore you rely on it.Restart your hosts afterwards. The Outbox probes for the index once, with
DescribeTable, and caches the answer for the life of the store instance. An Outbox built before the index existed keeps reporting that replay is unsupported until the process restarts.
The index name
DynamoDbConfiguration.CausationIndexName defaults to "Causation". Leave it there.
It has a public setter, as the other index names do, and the same catch applies to all of them: each index name is also declared on MessageItem as an attribute argument, and attribute arguments must be compile-time constants, so the annotation cannot read your configured value. DynamoDbTableFactory therefore always generates an index called Causation, whatever you set here.
Point CausationIndexName at another name and the probe and the replay query both target a GSI the table model never declares. Nothing errors: SupportsCausationTracking() simply reports false, and replay silently finds no messages. If you have a genuine reason to rename it, you must create the index under that name yourself.
If you skip the index
Nothing breaks, and replay never happens:
Deposits are unaffected. The
CausationIdattribute is still written; with no index to populate it is simply an ordinary attribute.Startup warns.
SupportsCausationTracking()reportsfalse, and pipeline validation raises a warning — not an error — for any handler configured withOnceOnlyAction.Replay.Duplicates are skipped quietly.
ReplayCausationreturnsfalserather than throwing, so a duplicate does not fail the consumer pipeline with a DynamoDBValidationException. Nothing is resent.
See When Replay Does Not Fire for how to tell this apart from the other reasons a replay produces nothing.
Last updated
Was this helpful?
