> For the complete documentation index, see [llms.txt](https://brightercommand.gitbook.io/paramore-brighter-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brightercommand.gitbook.io/paramore-brighter-documentation/outbox-and-inbox/sweepercircuitbreaking.md).

# Sweeper Circuit Breaking

Sweeper Circuit Breaking is a resilience feature that prevents failures to publish to one topic from blocking attempts to publish to other topics, when publishing messages from the Outbox.

> **Reference** · Applies to **Brighter V10**

## Sweeper Circuit Breaking Overview

Sweeper Circuit Breaking is a resilience feature that prevents failures to publish to one topic from blocking attempts to publish to other topics, when publishing messages from the Outbox. When a topic repeatedly fails to publish, the circuit breaker "trips" that topic, temporarily preventing further publish attempts until a cooldown period expires.

This feature is particularly valuable in scenarios where:

* **Transport failures**: A message broker or queue becomes unavailable
* **Topic-specific issues**: A specific topic/queue has configuration problems or capacity issues
* **Cascade prevention**: Failing topics would otherwise block the Outbox Sweeper from processing healthy topics
* **Resource protection**: Repeated failures to unhealthy topics consume resources without benefit

## How Sweeper Circuit Breaking Works

The Sweeper Circuit Breaker operates at the topic level during Outbox clearing operations:

### Normal Operation

1. **Outbox Sweeper runs**: Periodically attempts to clear outstanding messages from the Outbox
2. **Tripped topics left out**: The sweeper asks the Outbox for outstanding messages, passing it the tripped topics to leave out
3. **Publish attempts**: The sweeper attempts to publish messages to their respective topics
4. **Success**: Messages are published and marked as dispatched

### Circuit Breaking Behavior

When a topic fails to publish:

1. **Failure detected**: An exception occurs during message publication to a specific topic
2. **Circuit trips**: The circuit breaker marks that topic as "tripped"
3. **Cooldown begins**: A cooldown counter is set for the tripped topic (default: 10 sweeps)
4. **Subsequent sweeps**: Each sweep decrements the counter for every tripped topic before it reads the Outbox, and skips the tripped topics' messages
5. **Recovery**: The sweep after the counter reaches zero removes the topic from the tripped list, so a topic sits out `CooldownCount` sweeps
6. **Retry**: That same sweep publishes the topic's messages again

### Benefits

* **Prevents blocking**: Healthy topics continue to be processed even when some topics fail
* **Automatic recovery**: Topics automatically recover after the cooldown period
* **Resource efficiency**: Avoids wasting resources on repeated failures to unhealthy topics
* **Observability**: Tripped topics can be monitored and alerted on

## Sweeper Circuit Breaking Configuration

### Enabling Circuit Breaking

To enable Sweeper Circuit Breaking, register an `IAmAnOutboxCircuitBreaker` implementation with your IoC container:

```csharp
using Paramore.Brighter.CircuitBreaker;

public void ConfigureServices(IServiceCollection services)
{
    // Register the circuit breaker
    services.AddSingleton<IAmAnOutboxCircuitBreaker>(
        new InMemoryOutboxCircuitBreaker(new OutboxCircuitBreakerOptions
        {
            CooldownCount = 10  // Number of sweeps before recovery (default: 10)
        })
    );

    services.AddBrighter(options =>
    {
        // Configure Brighter as normal
    })
    .AddProducers(/* producer configuration */)
    .UseOutboxSweeper();  // Enable the Outbox Sweeper
}
```

### Configuration Options

The `OutboxCircuitBreakerOptions` class provides the following configuration:

| Option          | Type  | Default | Description                                                    |
| --------------- | ----- | ------- | -------------------------------------------------------------- |
| `CooldownCount` | `int` | `10`    | Sweeper iterations a tripped topic waits before it is retried. |

### Calculating Cooldown Time

The actual cooldown time depends on how often the Outbox Sweeper runs, which you set with `TimerInterval`, in seconds, on the options you pass to `UseOutboxSweeper`. A topic trips during one sweep, sits out the next `CooldownCount` sweeps, and is retried on the one after:

**Formula**: `Time until retry = (CooldownCount + 1) × TimerInterval`

**Example**:

* `CooldownCount = 10`
* `TimerInterval = 60`, so the Sweeper runs every 60 seconds
* **Time until retry = (10 + 1) × 60s = 11 minutes**

```csharp
using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter.CircuitBreaker;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.Outbox.Hosting;

services.AddBrighter()
    .AddProducers(configure =>
    {
        // ... your producer registry and Outbox
    })
    .UseOutboxSweeper(options =>
    {
        options.TimerInterval = 60;  // Sweep every 60 seconds
    });

// A tripped topic sits out 10 sweeps and is retried on the 11th: (10 + 1) × 60s = 11 minutes
services.AddSingleton<IAmAnOutboxCircuitBreaker>(
    new InMemoryOutboxCircuitBreaker(new OutboxCircuitBreakerOptions
    {
        CooldownCount = 10
    })
);
```

## Sweeper Circuit Breaking Monitoring and Observability

### Checking Tripped Topics

You can query the circuit breaker to see which topics are currently tripped:

```csharp
public class OutboxMonitorService
{
    private readonly IAmAnOutboxCircuitBreaker _circuitBreaker;
    private readonly ILogger<OutboxMonitorService> _logger;

    public OutboxMonitorService(
        IAmAnOutboxCircuitBreaker circuitBreaker,
        ILogger<OutboxMonitorService> logger)
    {
        _circuitBreaker = circuitBreaker;
        _logger = logger;
    }

    public void CheckCircuitBreakerStatus()
    {
        var trippedTopics = _circuitBreaker.TrippedTopics;

        if (trippedTopics.Any())
        {
            _logger.LogWarning(
                "Circuit breaker has {Count} tripped topics: {Topics}",
                trippedTopics.Count(),
                string.Join(", ", trippedTopics.Select(t => t.Value))
            );
        }
        else
        {
            _logger.LogInformation("All topics are healthy");
        }
    }
}
```

### Logging and Alerts

Set up monitoring to track circuit breaker events:

```csharp
public class CircuitBreakerHealthCheck : IHealthCheck
{
    private readonly IAmAnOutboxCircuitBreaker _circuitBreaker;

    public CircuitBreakerHealthCheck(IAmAnOutboxCircuitBreaker circuitBreaker)
    {
        _circuitBreaker = circuitBreaker;
    }

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        var trippedTopics = _circuitBreaker.TrippedTopics.ToList();

        if (!trippedTopics.Any())
        {
            return Task.FromResult(HealthCheckResult.Healthy("No tripped topics"));
        }

        var data = new Dictionary<string, object>
        {
            { "tripped_topics", trippedTopics.Select(t => t.Value).ToArray() },
            { "count", trippedTopics.Count }
        };

        return Task.FromResult(
            HealthCheckResult.Degraded(
                $"{trippedTopics.Count} topic(s) are currently tripped",
                data: data
            )
        );
    }
}

// Register health check
services.AddHealthChecks()
    .AddCheck<CircuitBreakerHealthCheck>("outbox_circuit_breaker");
```

## Sweeper Circuit Breaking Outbox Support

The sweeper does not filter tripped topics itself. It passes `TrippedTopics` to the Outbox when it asks for outstanding messages, and the Outbox leaves those topics out of its query — so circuit breaking works only where the Outbox honours that list. At Brighter 10.7.0:

| Outbox                                   | Leaves tripped topics out                                                                                                                   |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| MS SQL Server, MySQL, PostgreSQL, SQLite | Yes                                                                                                                                         |
| MongoDB                                  | Yes                                                                                                                                         |
| Firestore                                | Yes                                                                                                                                         |
| InMemory                                 | Yes                                                                                                                                         |
| DynamoDB, both the V3 and V4 packages    | **No** — it accepts the list and ignores it ([#4443](https://github.com/BrighterCommand/Brighter/issues/4443))                              |
| Spanner                                  | **No** — its query has no place for the filter, so the filter is dropped ([#4444](https://github.com/BrighterCommand/Brighter/issues/4444)) |

With DynamoDB or Spanner, a registered breaker still records trips, and `TrippedTopics` still reports them, so the monitoring above works. But every sweep reads and sends a tripped topic's messages as though nothing had tripped.

An Outbox that honours the list needs nothing extra: register the breaker and the sweeper beside it as usual. With MongoDB, for example:

```csharp
using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter.CircuitBreaker;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.MongoDb;
using Paramore.Brighter.Outbox.Hosting;
using Paramore.Brighter.Outbox.MongoDb;

var mongoDbConfiguration = new MongoDbConfiguration("mongodb://localhost:27017", "BrighterDatabase")
{
    Outbox = new MongoDbCollectionConfiguration { Name = "Outbox" }
};

services.AddSingleton<IAmAnOutboxCircuitBreaker>(new InMemoryOutboxCircuitBreaker());

services.AddBrighter()
    .AddProducers(configure =>
    {
        // ... your producer registry
        configure.Outbox = new MongoDbOutbox(mongoDbConfiguration);
        configure.ConnectionProvider = typeof(MongoDbConnectionProvider);
        configure.TransactionProvider = typeof(MongoDbUnitOfWork);
    })
    .UseOutboxSweeper();
```

## Bulk Dispatch Support

With `UseBulk` set on its options, the sweeper sends each topic's outstanding messages in batches, through a producer that implements `IAmABulkMessageProducerAsync`. Circuit breaking works as it does for single messages: tripped topics are left out when the sweeper reads the Outbox, and a batch that fails to send trips its topic.

```csharp
using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.Outbox.Hosting;

services.AddBrighter()
    .AddProducers(configure =>
    {
        // ... your producer registry and Outbox
    })
    .UseOutboxSweeper(options =>
    {
        options.UseBulk = true;  // needs a producer that implements IAmABulkMessageProducerAsync
    });
```

## Sweeper Circuit Breaking Best Practices

### 1. Choose Appropriate Cooldown Periods

Balance between quick recovery and avoiding repeated failures:

* **Short cooldown (3-5 sweeps)**: For transient issues, quick recovery desired
* **Medium cooldown (10-15 sweeps)**: General purpose, good balance
* **Long cooldown (20-30 sweeps)**: For persistent issues, reduce retry overhead

### 2. Align Cooldown with Sweep Interval

Consider the time until a tripped topic is retried, `(CooldownCount + 1) × TimerInterval`:

```csharp
using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter.CircuitBreaker;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.Outbox.Hosting;

// Fast sweeping with a short cooldown = quick recovery
services.AddBrighter()
    .AddProducers(configure =>
    {
        // ... your producer registry and Outbox
    })
    .UseOutboxSweeper(options =>
    {
        options.TimerInterval = 30;  // Sweep every 30 seconds
    });

services.AddSingleton<IAmAnOutboxCircuitBreaker>(
    new InMemoryOutboxCircuitBreaker(new OutboxCircuitBreakerOptions
    {
        CooldownCount = 5  // (5 + 1) × 30s = 3 minutes until retry
    })
);
```

### 3. Monitor Tripped Topics

Set up monitoring and alerting:

* **Health checks**: Use ASP.NET Core health checks to expose tripped topics
* **Metrics**: Export circuit breaker metrics to Prometheus, DataDog, etc.
* **Logging**: Log when topics trip and recover
* **Alerts**: Alert when topics remain tripped for extended periods

### 4. Investigate Root Causes

When topics trip repeatedly:

1. **Check broker health**: Ensure message broker is operational
2. **Verify permissions**: Ensure the application has permissions to publish
3. **Check queue/topic existence**: Verify the destination exists
4. **Review capacity**: Check if the queue/topic has reached capacity limits
5. **Inspect network**: Look for network connectivity issues

### 5. Use with Outbox Sweeper

Circuit breaking is designed to work with the Outbox Sweeper:

```csharp
using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.MsSql;
using Paramore.Brighter.Outbox.Hosting;
using Paramore.Brighter.Outbox.MsSql;

// ... outboxConfiguration comes from your database configuration
services.AddBrighter()
    .AddProducers(configure =>
    {
        configure.Outbox = new MsSqlOutbox(outboxConfiguration);
        configure.ConnectionProvider = typeof(MsSqlConnectionProvider);
        configure.TransactionProvider = typeof(MsSqlTransactionProvider);
    })
    .UseOutboxSweeper();  // Required for circuit breaking to function
```

### 6. Consider Immediate vs. Sweeper Clearing

Only the sweeper skips tripped topics. When you clear explicitly with `ClearOutbox` or `ClearOutboxAsync`, Brighter sends every message you name, whether or not its topic is tripped.

A failed send from `ClearOutboxAsync` does trip the topic, so the sweeper then skips it. A failed send from `ClearOutbox` trips it only when the producer reports failures through publish confirmation.

```csharp
// ...
// Explicit clearing - sends every message named, tripped topic or not
await commandProcessor.ClearOutboxAsync(messageIds);

// Sweeper clearing - skips tripped topics
// Happens automatically via UseOutboxSweeper
```

### 7. Test Failure Scenarios

Regularly test circuit breaker behavior:

* Simulate broker outages
* Test individual topic failures
* Verify healthy topics continue processing
* Confirm automatic recovery after cooldown

## Sweeper Circuit Breaking Troubleshooting

### Topics Not Recovering

**Problem**: Topics remain tripped indefinitely

**Solutions**:

1. Verify Outbox Sweeper is running
2. Check cooldown count is not excessively high
3. Ensure the Sweeper's `TimerInterval` is appropriate
4. Confirm circuit breaker is properly registered

### All Topics Tripping

**Problem**: All topics become tripped at once

**Possible Causes**:

* Broker is completely down
* Network connectivity issues
* Authentication/authorization failures
* Shared resource exhaustion

**Solutions**:

1. Check broker health and connectivity
2. Verify credentials and permissions
3. Review broker logs for errors
4. Consider infrastructure capacity

### Messages Stuck in Outbox

**Problem**: Messages accumulate in Outbox without being dispatched

**Check**:

1. Is the Outbox Sweeper enabled?
2. Are topics currently tripped? Check `TrippedTopics`
3. Is the circuit breaker cooldown too long?
4. Are there persistent transport issues?

**Solutions**:

* Enable Outbox Sweeper if not already enabled
* Investigate why topics are tripping
* Reduce cooldown count if appropriate
* Fix underlying transport issues

### Circuit Breaker Not Working

**Problem**: Failed topics continue to be retried

**Verify**:

1. Circuit breaker is registered: `services.AddSingleton<IAmAnOutboxCircuitBreaker>`
2. Using the sweeper: `UseOutboxSweeper()`
3. Exceptions are being thrown during publish (not silently failing)
4. Circuit breaker implementation is correct
5. Your Outbox leaves tripped topics out — DynamoDB and Spanner do not (see [Outbox Support](#sweeper-circuit-breaking-outbox-support))

## Sweeper Circuit Breaking Summary

Sweeper Circuit Breaking provides automatic resilience for Outbox clearing operations by:

* **Preventing cascade failures** when specific topics fail
* **Automatically recovering** after a configurable cooldown period
* **Allowing healthy topics** to continue processing
* **Protecting resources** from repeated failures

Enable circuit breaking by registering `IAmAnOutboxCircuitBreaker` with your IoC container and configuring appropriate cooldown periods for your application's needs.

## Further Reading

* [Using Sweeper Circuit Breaking](/paramore-brighter-documentation/outbox-and-inbox/sweepercircuitbreaking/usingsweepercircuitbreaking.md) - Wiring it up, tuning the cooldown, and custom breakers
* [Outbox Support](/paramore-brighter-documentation/outbox-and-inbox/brighteroutboxsupport.md) - The Outbox and the Sweeper
* [Distributed Lock](/paramore-brighter-documentation/outbox-and-inbox/distributedlock.md) - Keeping a single Sweeper active


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://brightercommand.gitbook.io/paramore-brighter-documentation/outbox-and-inbox/sweepercircuitbreaking.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.