> For the complete documentation index, see [llms.txt](https://brightercommand.gitbook.io/paramore-brighter-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brightercommand.gitbook.io/paramore-brighter-documentation/outbox-and-inbox/brighteroutboxsupport/dynamooutbox.md). # Dynamo Outbox The DynamoDb Outbox allows integration between DynamoDb and Brighter's outbox support. > **Reference** · Applies to **Brighter V10** ## DynamoDb Outbox Usage The DynamoDb Outbox allows integration between DynamoDb and [Brighter's outbox support](/paramore-brighter-documentation/outbox-and-inbox/brighteroutboxsupport.md). The configuration is described in [Command Processor Configuration Reference](/paramore-brighter-documentation/brighter-configuration/brighterbasicconfiguration/commandprocessorconfigurationreference.md#outbox-support). To support transactional messaging when using DynamoDb requires us to use DynamoDb's support for ACID transactions. You should understand best practices for using transactions with DynamoDb. For this we will need the *Outbox* package for DynamoDb: **AWS SDK v3** (legacy support): * **Paramore.Brighter.Outbox.DynamoDB** **AWS SDK v4** (recommended for new projects): * **Paramore.Brighter.Outbox.DynamoDB.V4** **Paramore.Brighter.Outbox.DynamoDb** (or **.V4**) will pull in another package: * **Paramore.Brighter.DynamoDb** (or **Paramore.Brighter.DynamoDb.V4**) The examples on this page use the v3 packages' namespaces. The v4 packages append `.V4` to theirs: `Paramore.Brighter.Outbox.DynamoDB.V4` and `Paramore.Brighter.DynamoDb.V4`, which also holds `DynamoDbTableFactory`. See [AWS SQS Migration](/paramore-brighter-documentation/transports/awssqsconfiguration/awssqsmigratetov10.md#migrating-from-aws-sdk-v3-to-v4) for migration guidance between v3 and v4. As described in [Command Processor Configuration Reference](/paramore-brighter-documentation/brighter-configuration/brighterbasicconfiguration/commandprocessorconfigurationreference.md#outbox-support), we configure Brighter to use an outbox by setting **Outbox** on the options passed to **AddProducers()**. As we want to use DynamoDb with the outbox, we also set **ConnectionProvider** and **TransactionProvider** — both to **DynamoDbUnitOfWork**, which is one type playing both parts — so that we can share your transaction scope when persisting messages to the outbox. ```csharp using System; using Amazon.DynamoDBv2; using Microsoft.Extensions.DependencyInjection; using Paramore.Brighter; using Paramore.Brighter.DynamoDb; using Paramore.Brighter.Extensions.DependencyInjection; using Paramore.Brighter.Outbox.DynamoDB; using Paramore.Brighter.Outbox.Hosting; public void ConfigureServices(IServiceCollection services) { // ... dynamoDb is your IAmazonDynamoDB client, producerRegistry your transport services.AddSingleton(dynamoDb); services.AddBrighter() .AddProducers(configure => { configure.ProducerRegistry = producerRegistry; configure.Outbox = new DynamoDbOutbox( dynamoDb, new DynamoDbConfiguration(), TimeProvider.System); configure.ConnectionProvider = typeof(DynamoDbUnitOfWork); configure.TransactionProvider = typeof(DynamoDbUnitOfWork); }) .UseOutboxSweeper() .AutoFromAssemblies(); } ``` The handler below is `AddGreetingHandlerAsync` from the Brighter sample at `Brighter/samples/WebAPI/WebAPI_Dynamo/`, which also declares the `AddGreeting` request, the `Person` entity and the `GreetingMade` event it uses. In our handler we take a dependency on Brighter's **IAmADynamoDbTransactionProvider** interface, which **DynamoDbUnitOfWork** implements. We explicitly start a transaction within the handler on the Database within that provider. We call **DepositPostAsync** within that transaction to write the message to the Outbox. Once the transaction has closed we can call **ClearOutboxAsync** to immediately clear, or we can rely on the Outbox Sweeper, if we have configured one to clear for us. (There are equivalent synchronous versions of these APIs). > **Running more than one instance?** Configure a [distributed lock](/paramore-brighter-documentation/outbox-and-inbox/distributedlock.md) so only one Sweeper (and Archiver) runs at a time — see [DynamoDB Distributed Lock](/paramore-brighter-documentation/outbox-and-inbox/distributedlock/dynamodbdistributedlock.md). ```csharp using System; using System.Collections.Generic; using System.Threading; using System.Threading.Tasks; using Amazon.DynamoDBv2.DataModel; using Amazon.DynamoDBv2.Model; using Microsoft.Extensions.Logging; using Paramore.Brighter; using Paramore.Brighter.DynamoDb; public class AddGreetingHandlerAsync : RequestHandlerAsync { private readonly IAmADynamoDbTransactionProvider _transactionProvider; private readonly IAmACommandProcessor _postBox; private readonly ILogger _logger; public AddGreetingHandlerAsync(IAmADynamoDbTransactionProvider transactionProvider, IAmACommandProcessor postBox, ILogger logger) { _transactionProvider = transactionProvider; _postBox = postBox; _logger = logger; } public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default) { var posts = new List(); //We use the transaction provider to grab connection and transaction, because Outbox needs //to share them 'behind the scenes' var context = new DynamoDBContext(_transactionProvider.DynamoDb); var transaction = await _transactionProvider.GetTransactionAsync(cancellationToken); try { var person = await context.LoadAsync(addGreeting.Name, cancellationToken); person.Greetings.Add(addGreeting.Greeting); var document = context.ToDocument(person); var attributeValues = document.ToAttributeMap(); //write the added child entity to the Db - just replace the whole entity as we grabbed the original //in production code, an update expression would be faster transaction.TransactItems.Add(new TransactWriteItem{Put = new Put{TableName = "People", Item = attributeValues}}); //Now write the message we want to send to the Db in the same transaction. posts.Add(await _postBox.DepositPostAsync( new GreetingMade(addGreeting.Greeting), _transactionProvider, cancellationToken: cancellationToken)); //commit both new greeting and outgoing message await _transactionProvider.CommitAsync(cancellationToken); } catch (Exception e) { _logger.LogError(e, "Exception thrown handling Add Greeting request"); //it went wrong, rollback the entity change and the downstream message _transactionProvider.Rollback(); return await base.HandleAsync(addGreeting, cancellationToken); } //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. //Alternatively, you can let the Sweeper do this, but at the cost of increased latency await _postBox.ClearOutboxAsync(posts, cancellationToken: cancellationToken); return await base.HandleAsync(addGreeting, cancellationToken); } } ``` ## DynamoDb Outbox Options `DynamoDbConfiguration` names the table and its five global secondary indexes, and carries the two settings that govern expiry and scan concurrency. | Option | Type | Default | Description | | ------------------------------- | ----------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `TableName` | `string` | `"brighter_outbox"` | Names the DynamoDB table that holds the Outbox. | | `DeliveredIndexName` | `string` | `"Delivered"` | Names the global secondary index of dispatched messages, indexed by topic. | | `DeliveredAllTopicsIndexName` | `string` | `"DeliveredAllTopics"` | Names the global secondary index of dispatched messages covering every topic. | | `OutstandingIndexName` | `string` | `"Outstanding"` | Names the global secondary index of undispatched messages, indexed by topic. | | `OutstandingAllTopicsIndexName` | `string` | `"OutstandingAllTopics"` | Names the global secondary index of undispatched messages covering every topic. | | `CausationIndexName` | `string` | `"Causation"` | Names the global secondary index over the Causation Id, which [Replay On Seen](/paramore-brighter-documentation/outbox-and-inbox/replayonseen.md) queries; leave it at the default, for the reason under [The index name](#the-index-name). | | `TimeToLive` | `TimeSpan?` | `null` | Sets how long a message survives in the table before DynamoDB expires it; messages do not expire when it is null. | | `ScanConcurrency` | `int` | `3` | Sets how many segments a parallel scan for outstanding messages uses. | **`timeout` and `numberOfShards` are constructor parameters, not properties you can set.** They surface as the get-only `Timeout` and `NumberOfShards`. `numberOfShards` defaults to 3 and shards the outstanding index so an active topic does not build a hot partition; the Outbox throws `ArgumentOutOfRangeException` above 20. `timeout` is the deadline, in milliseconds, for each Outbox operation, and defaults to 500; zero or -1 disables it, and anything below -1 throws `ArgumentOutOfRangeException`. The deadline covers the whole operation, batch items and query pages included, and an operation that overruns it is cancelled and surfaces as `OperationCanceledException`. The `outBoxTimeout` argument the `IAmAnOutbox` methods take overrides it for one call: -1, the default, uses the configured `timeout`, and zero disables the deadline. Timeouts you configure on the `AmazonDynamoDBClient` still apply alongside it. `tableName` and `scanConcurrency` are constructor parameters as well, but each has a settable property and is in the table above. ## Replay Support: The Causation Index > **Requires Brighter 10.8.0 or later.** Earlier versions of both DynamoDB Outbox packages have no `DynamoDbConfiguration.CausationIndexName` and no `CausationId` attribute. [Replay On Seen](/paramore-brighter-documentation/outbox-and-inbox/replayonseen.md) resends every Outbox message produced under a given Causation Id. On DynamoDB that means querying on a non-key attribute, which needs a Global Secondary Index — by default one named **Causation**, with `CausationId` as its hash key. This applies to both `Paramore.Brighter.Outbox.DynamoDB` and `.V4`. The DynamoDB **Inbox** needs nothing: it looks a Causation Id up by the table's own primary key. ### A new table gets the index for free `MessageItem.CausationId` is decorated with `[DynamoDBGlobalSecondaryIndexHashKey(indexName: "Causation")]`, and `DynamoDbTableFactory` reflects over those attributes when it builds the request. So a table you create through the factory already has the index — you only add a throughput entry for it: ```csharp using System.Collections.Generic; using Amazon.DynamoDBv2; using Amazon.DynamoDBv2.Model; using Paramore.Brighter.DynamoDb; using Paramore.Brighter.Outbox.DynamoDB; var createTableRequest = new DynamoDbTableFactory().GenerateCreateTableRequest( new DynamoDbCreateProvisionedThroughput( new ProvisionedThroughput { ReadCapacityUnits = 10, WriteCapacityUnits = 10 }, new Dictionary { ["Outstanding"] = new() { ReadCapacityUnits = 10, WriteCapacityUnits = 10 }, ["OutstandingAllTopics"] = new() { ReadCapacityUnits = 10, WriteCapacityUnits = 10 }, ["Delivered"] = new() { ReadCapacityUnits = 10, WriteCapacityUnits = 10 }, ["DeliveredAllTopics"] = new() { ReadCapacityUnits = 10, WriteCapacityUnits = 10 }, ["Causation"] = new() { ReadCapacityUnits = 10, WriteCapacityUnits = 10 }, // replay })); var builder = new DynamoDbTableBuilder(client); await builder.Build(createTableRequest); await builder.EnsureTablesReady([createTableRequest.TableName], TableStatus.ACTIVE); ``` ### Adding the index to an existing table A table provisioned before replay shipped does not have the index, and there is no migration runner for DynamoDB. Add it yourself with `UpdateTable`: ```csharp using Amazon.DynamoDBv2; using Amazon.DynamoDBv2.Model; var request = new UpdateTableRequest { TableName = "brighter_outbox", // The index's key attribute has to be declared, even though the table already stores it AttributeDefinitions = [ new AttributeDefinition { AttributeName = "CausationId", AttributeType = ScalarAttributeType.S } ], GlobalSecondaryIndexUpdates = [ new GlobalSecondaryIndexUpdate { Create = new CreateGlobalSecondaryIndexAction { IndexName = "Causation", KeySchema = [ new KeySchemaElement { AttributeName = "CausationId", KeyType = KeyType.HASH } ], // Replay only reads MessageId, which is the base table's hash key and is // therefore always present in a KEYS_ONLY index Projection = new Projection { ProjectionType = ProjectionType.KEYS_ONLY }, // Omit this on a PAY_PER_REQUEST table — it applies to PROVISIONED billing only ProvisionedThroughput = new ProvisionedThroughput { ReadCapacityUnits = 10, WriteCapacityUnits = 10 } } } ] }; await client.UpdateTableAsync(request); ``` Two operational notes: * **The index backfills asynchronously.** DynamoDB populates a new GSI in the background, and the index reports `CREATING` until it finishes. Until then a replay may find only part of a causation's messages, so wait for the index to become `ACTIVE` before you rely on it. * **Restart your hosts afterwards.** The Outbox probes for the index once, with `DescribeTable`, and caches the answer for the life of the store instance. An Outbox built before the index existed keeps reporting that replay is unsupported until the process restarts. ### The index name `DynamoDbConfiguration.CausationIndexName` defaults to `"Causation"`. **Leave it there.** It has a public setter, as the other index names do, and the same catch applies to all of them: each index name is also declared on `MessageItem` as an attribute argument, and attribute arguments must be compile-time constants, so the annotation cannot read your configured value. `DynamoDbTableFactory` therefore always generates an index called `Causation`, whatever you set here. Point `CausationIndexName` at another name and the probe and the replay query both target a GSI the table model never declares. Nothing errors: `SupportsCausationTracking()` simply reports `false`, and replay silently finds no messages. If you have a genuine reason to rename it, you must create the index under that name yourself. ### If you skip the index Nothing breaks, and replay never happens: * **Deposits are unaffected.** The `CausationId` attribute is still written; with no index to populate it is simply an ordinary attribute. * **Startup warns.** `SupportsCausationTracking()` reports `false`, and [pipeline validation](/paramore-brighter-documentation/commands-handlers-and-pipelines/buildingapipeline/pipelinevalidation.md) raises a *warning* — not an error — for any handler configured with `OnceOnlyAction.Replay`. * **Duplicates are skipped quietly.** `ReplayCausation` returns `false` rather than throwing, so a duplicate does not fail the consumer pipeline with a DynamoDB `ValidationException`. Nothing is resent. See [When Replay Does Not Fire](/paramore-brighter-documentation/outbox-and-inbox/replayonseen/replayonseenreference.md#when-replay-does-not-fire) for how to tell this apart from the other reasons a replay produces nothing. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://brightercommand.gitbook.io/paramore-brighter-documentation/outbox-and-inbox/brighteroutboxsupport/dynamooutbox.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.