Sweeper Circuit Breaking

Reference · Applies to Brighter V10

Sweeper Circuit Breaking Overview

Sweeper Circuit Breaking is a resilience feature that prevents failures to publish to one topic from blocking attempts to publish to other topics, when publishing messages from the Outbox. When a topic repeatedly fails to publish, the circuit breaker "trips" that topic, temporarily preventing further publish attempts until a cooldown period expires.

This feature is particularly valuable in scenarios where:

• Transport failures: A message broker or queue becomes unavailable

• Topic-specific issues: A specific topic/queue has configuration problems or capacity issues

• Cascade prevention: Failing topics would otherwise block the Outbox Sweeper from processing healthy topics

• Resource protection: Repeated failures to unhealthy topics consume resources without benefit

How Sweeper Circuit Breaking Works

The Sweeper Circuit Breaker operates at the topic level during Outbox clearing operations:

Normal Operation

1. Outbox Sweeper runs: Periodically attempts to clear outstanding messages from the Outbox

2. Tripped topics left out: The sweeper asks the Outbox for outstanding messages, passing it the tripped topics to leave out

3. Publish attempts: The sweeper attempts to publish messages to their respective topics

4. Success: Messages are published and marked as dispatched

Circuit Breaking Behavior

When a topic fails to publish:

1. Failure detected: An exception occurs during message publication to a specific topic

2. Circuit trips: The circuit breaker marks that topic as "tripped"

3. Cooldown begins: A cooldown counter is set for the tripped topic (default: 10 sweeps)

4. Subsequent sweeps: Each sweep decrements the counter for every tripped topic before it reads the Outbox, and skips the tripped topics' messages

5. Recovery: The sweep after the counter reaches zero removes the topic from the tripped list, so a topic sits out CooldownCount sweeps

6. Retry: That same sweep publishes the topic's messages again

Benefits

• Prevents blocking: Healthy topics continue to be processed even when some topics fail

• Automatic recovery: Topics automatically recover after the cooldown period

• Resource efficiency: Avoids wasting resources on repeated failures to unhealthy topics

• Observability: Tripped topics can be monitored and alerted on

Sweeper Circuit Breaking Configuration

Enabling Circuit Breaking

To enable Sweeper Circuit Breaking, register an IAmAnOutboxCircuitBreaker implementation with your IoC container:

using Paramore.Brighter.CircuitBreaker;

public void ConfigureServices(IServiceCollection services)
{
    // Register the circuit breaker
    services.AddSingleton<IAmAnOutboxCircuitBreaker>(
        new InMemoryOutboxCircuitBreaker(new OutboxCircuitBreakerOptions
        {
            CooldownCount = 10  // Number of sweeps before recovery (default: 10)
        })
    );

    services.AddBrighter(options =>
    {
        // Configure Brighter as normal
    })
    .AddProducers(/* producer configuration */)
    .UseOutboxSweeper();  // Enable the Outbox Sweeper
}

Configuration Options

The OutboxCircuitBreakerOptions class provides the following configuration:

Option	Type	Default	Description
CooldownCount	int	10	Sweeper iterations a tripped topic waits before it is retried.

Calculating Cooldown Time

The actual cooldown time depends on how often the Outbox Sweeper runs, which you set with TimerInterval, in seconds, on the options you pass to UseOutboxSweeper. A topic trips during one sweep, sits out the next CooldownCount sweeps, and is retried on the one after:

Formula: Time until retry = (CooldownCount + 1) × TimerInterval

Example:

• CooldownCount = 10

• TimerInterval = 60, so the Sweeper runs every 60 seconds

• Time until retry = (10 + 1) × 60s = 11 minutes

using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter.CircuitBreaker;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.Outbox.Hosting;

services.AddBrighter()
    .AddProducers(configure =>
    {
        // ... your producer registry and Outbox
    })
    .UseOutboxSweeper(options =>
    {
        options.TimerInterval = 60;  // Sweep every 60 seconds
    });

// A tripped topic sits out 10 sweeps and is retried on the 11th: (10 + 1) × 60s = 11 minutes
services.AddSingleton<IAmAnOutboxCircuitBreaker>(
    new InMemoryOutboxCircuitBreaker(new OutboxCircuitBreakerOptions
    {
        CooldownCount = 10
    })
);

Sweeper Circuit Breaking Monitoring and Observability

Checking Tripped Topics

You can query the circuit breaker to see which topics are currently tripped:

public class OutboxMonitorService
{
    private readonly IAmAnOutboxCircuitBreaker _circuitBreaker;
    private readonly ILogger<OutboxMonitorService> _logger;

    public OutboxMonitorService(
        IAmAnOutboxCircuitBreaker circuitBreaker,
        ILogger<OutboxMonitorService> logger)
    {
        _circuitBreaker = circuitBreaker;
        _logger = logger;
    }

    public void CheckCircuitBreakerStatus()
    {
        var trippedTopics = _circuitBreaker.TrippedTopics;

        if (trippedTopics.Any())
        {
            _logger.LogWarning(
                "Circuit breaker has {Count} tripped topics: {Topics}",
                trippedTopics.Count(),
                string.Join(", ", trippedTopics.Select(t => t.Value))
            );
        }
        else
        {
            _logger.LogInformation("All topics are healthy");
        }
    }
}

Logging and Alerts

Set up monitoring to track circuit breaker events:

public class CircuitBreakerHealthCheck : IHealthCheck
{
    private readonly IAmAnOutboxCircuitBreaker _circuitBreaker;

    public CircuitBreakerHealthCheck(IAmAnOutboxCircuitBreaker circuitBreaker)
    {
        _circuitBreaker = circuitBreaker;
    }

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        var trippedTopics = _circuitBreaker.TrippedTopics.ToList();

        if (!trippedTopics.Any())
        {
            return Task.FromResult(HealthCheckResult.Healthy("No tripped topics"));
        }

        var data = new Dictionary<string, object>
        {
            { "tripped_topics", trippedTopics.Select(t => t.Value).ToArray() },
            { "count", trippedTopics.Count }
        };

        return Task.FromResult(
            HealthCheckResult.Degraded(
                $"{trippedTopics.Count} topic(s) are currently tripped",
                data: data
            )
        );
    }
}

// Register health check
services.AddHealthChecks()
    .AddCheck<CircuitBreakerHealthCheck>("outbox_circuit_breaker");

Sweeper Circuit Breaking Outbox Support

The sweeper does not filter tripped topics itself. It passes TrippedTopics to the Outbox when it asks for outstanding messages, and the Outbox leaves those topics out of its query — so circuit breaking works only where the Outbox honours that list. At Brighter 10.7.0:

Outbox	Leaves tripped topics out
MS SQL Server, MySQL, PostgreSQL, SQLite	Yes
MongoDB	Yes
Firestore	Yes
InMemory	Yes
DynamoDB, both the V3 and V4 packages	No — it accepts the list and ignores it (#4443)
Spanner	No — its query has no place for the filter, so the filter is dropped (#4444)

With DynamoDB or Spanner, a registered breaker still records trips, and TrippedTopics still reports them, so the monitoring above works. But every sweep reads and sends a tripped topic's messages as though nothing had tripped.

An Outbox that honours the list needs nothing extra: register the breaker and the sweeper beside it as usual. With MongoDB, for example:

using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter.CircuitBreaker;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.MongoDb;
using Paramore.Brighter.Outbox.Hosting;
using Paramore.Brighter.Outbox.MongoDb;

var mongoDbConfiguration = new MongoDbConfiguration("mongodb://localhost:27017", "BrighterDatabase")
{
    Outbox = new MongoDbCollectionConfiguration { Name = "Outbox" }
};

services.AddSingleton<IAmAnOutboxCircuitBreaker>(new InMemoryOutboxCircuitBreaker());

services.AddSingleton<IAmAMongoDbConfiguration>(mongoDbConfiguration);

services.AddBrighter()
    .AddProducers(configure =>
    {
        // ... your producer registry
        configure.Outbox = new MongoDbOutbox(mongoDbConfiguration);
        configure.ConnectionProvider = typeof(MongoDbConnectionProvider);
        configure.TransactionProvider = typeof(MongoDbUnitOfWork);
    })
    .UseOutboxSweeper();

Bulk Dispatch Support

With UseBulk set on its options, the sweeper sends each topic's outstanding messages in batches, through a producer that implements IAmABulkMessageProducerAsync. Circuit breaking works as it does for single messages: tripped topics are left out when the sweeper reads the Outbox, and a batch that fails to send trips its topic.

using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.Outbox.Hosting;

services.AddBrighter()
    .AddProducers(configure =>
    {
        // ... your producer registry and Outbox
    })
    .UseOutboxSweeper(options =>
    {
        options.UseBulk = true;  // needs a producer that implements IAmABulkMessageProducerAsync
    });

Sweeper Circuit Breaking Best Practices

1. Choose Appropriate Cooldown Periods

Balance between quick recovery and avoiding repeated failures:

• Short cooldown (3-5 sweeps): For transient issues, quick recovery desired

• Medium cooldown (10-15 sweeps): General purpose, good balance

• Long cooldown (20-30 sweeps): For persistent issues, reduce retry overhead

2. Align Cooldown with Sweep Interval

Consider the time until a tripped topic is retried, (CooldownCount + 1) × TimerInterval:

using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter.CircuitBreaker;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.Outbox.Hosting;

// Fast sweeping with a short cooldown = quick recovery
services.AddBrighter()
    .AddProducers(configure =>
    {
        // ... your producer registry and Outbox
    })
    .UseOutboxSweeper(options =>
    {
        options.TimerInterval = 30;  // Sweep every 30 seconds
    });

services.AddSingleton<IAmAnOutboxCircuitBreaker>(
    new InMemoryOutboxCircuitBreaker(new OutboxCircuitBreakerOptions
    {
        CooldownCount = 5  // (5 + 1) × 30s = 3 minutes until retry
    })
);

3. Monitor Tripped Topics

Set up monitoring and alerting:

• Health checks: Use ASP.NET Core health checks to expose tripped topics

• Metrics: Export circuit breaker metrics to Prometheus, DataDog, etc.

• Logging: Log when topics trip and recover

• Alerts: Alert when topics remain tripped for extended periods

4. Investigate Root Causes

When topics trip repeatedly:

1. Check broker health: Ensure message broker is operational

2. Verify permissions: Ensure the application has permissions to publish

3. Check queue/topic existence: Verify the destination exists

4. Review capacity: Check if the queue/topic has reached capacity limits

5. Inspect network: Look for network connectivity issues

5. Use with Outbox Sweeper

Circuit breaking is designed to work with the Outbox Sweeper:

using Microsoft.Extensions.DependencyInjection;
using Paramore.Brighter;
using Paramore.Brighter.Extensions.DependencyInjection;
using Paramore.Brighter.MsSql;
using Paramore.Brighter.Outbox.Hosting;
using Paramore.Brighter.Outbox.MsSql;

// ... outboxConfiguration comes from your database configuration
services.AddSingleton<IAmARelationalDatabaseConfiguration>(outboxConfiguration);

services.AddBrighter()
    .AddProducers(configure =>
    {
        configure.Outbox = new MsSqlOutbox(outboxConfiguration);
        configure.ConnectionProvider = typeof(MsSqlConnectionProvider);
        configure.TransactionProvider = typeof(MsSqlTransactionProvider);
    })
    .UseOutboxSweeper();  // Required for circuit breaking to function

6. Consider Immediate vs. Sweeper Clearing

Only the sweeper skips tripped topics. When you clear explicitly with ClearOutbox or ClearOutboxAsync, Brighter sends every message you name, whether or not its topic is tripped.

A failed send from ClearOutboxAsync does trip the topic, so the sweeper then skips it. A failed send from ClearOutbox trips it only when the producer reports failures through publish confirmation.

// ...
// Explicit clearing - sends every message named, tripped topic or not
await commandProcessor.ClearOutboxAsync(messageIds);

// Sweeper clearing - skips tripped topics
// Happens automatically via UseOutboxSweeper

7. Test Failure Scenarios

Regularly test circuit breaker behavior:

• Simulate broker outages

• Test individual topic failures

• Verify healthy topics continue processing

• Confirm automatic recovery after cooldown

Sweeper Circuit Breaking Troubleshooting

Topics Not Recovering

Problem: Topics remain tripped indefinitely

Solutions:

1. Verify Outbox Sweeper is running

2. Check cooldown count is not excessively high

3. Ensure the Sweeper's TimerInterval is appropriate

4. Confirm circuit breaker is properly registered

All Topics Tripping

Problem: All topics become tripped at once

Possible Causes:

• Broker is completely down

• Network connectivity issues

• Authentication/authorization failures

• Shared resource exhaustion

Solutions:

1. Check broker health and connectivity

2. Verify credentials and permissions

3. Review broker logs for errors

4. Consider infrastructure capacity

Messages Stuck in Outbox

Problem: Messages accumulate in Outbox without being dispatched

Check:

1. Is the Outbox Sweeper enabled?

2. Are topics currently tripped? Check TrippedTopics

3. Is the circuit breaker cooldown too long?

4. Are there persistent transport issues?

Solutions:

• Enable Outbox Sweeper if not already enabled

• Investigate why topics are tripping

• Reduce cooldown count if appropriate

• Fix underlying transport issues

Circuit Breaker Not Working

Problem: Failed topics continue to be retried

Verify:

1. Circuit breaker is registered: services.AddSingleton<IAmAnOutboxCircuitBreaker>

2. Using the sweeper: UseOutboxSweeper()

3. Exceptions are being thrown during publish (not silently failing)

4. Circuit breaker implementation is correct

5. Your Outbox leaves tripped topics out — DynamoDB and Spanner do not (see Outbox Support)

Sweeper Circuit Breaking Summary

Sweeper Circuit Breaking provides automatic resilience for Outbox clearing operations by:

• Preventing cascade failures when specific topics fail

• Automatically recovering after a configurable cooldown period

• Allowing healthy topics to continue processing

• Protecting resources from repeated failures

Enable circuit breaking by registering IAmAnOutboxCircuitBreaker with your IoC container and configuring appropriate cooldown periods for your application's needs.

Further Reading

• Using Sweeper Circuit Breaking - Wiring it up, tuning the cooldown, and custom breakers

• Outbox Support - The Outbox and the Sweeper

• Distributed Lock - Keeping a single Sweeper active