Edit

Telemetry

Brighter provides comprehensive OpenTelemetry integration for distributed tracing across message boundaries, enabling end-to-end observability in distributed systems.

OpenTelemetry Semantic Conventions

V10 introduces support for OpenTelemetry Semantic Conventions, replacing the custom conventions used in V9.

  • OTel Semantic Conventions for Messaging: Standard span names and attributes for messaging operations

  • W3C TraceContext: Standard context propagation across service boundaries

  • CloudEvents Integration: Trace propagation via CloudEvents traceparent and tracestate headers

  • Configurable Instrumentation: Fine-grained control over what attributes are recorded

  • Comprehensive Coverage: Tracing for Command Processor, Dispatcher, Outbox, Inbox, and Transform pipelines

Configuring OpenTelemetry

The OpenTelemetry SDK can be configured to listen to Activities emitted by Brighter. For more information, see OpenTelemetry Tracing in .NET.

Activity Source

Brighter emits traces using the following Activity Source:

  • Source Name: paramore.brighter

  • Version: Includes the Brighter version number

Basic Configuration

The following code configures OpenTelemetry to:

  • Enable tracing

  • Set the service name

  • Listen to Brighter and Microsoft sources

  • Export traces to Jaeger

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

const string serviceName = "MyService";
var jaegerEndpoint = new Uri("http://localhost:14268/api/traces");

using var tracerProvider =
    Sdk.CreateTracerProviderBuilder()
        .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(serviceName))
        .AddSource("paramore.brighter", "Microsoft.*")
        .AddJaegerExporter(o =>
        {
            o.Endpoint = jaegerEndpoint;
        })
        .Build();

Configuration with Different Backends

Jaeger

.AddJaegerExporter(o =>
{
    o.AgentHost = "localhost";
    o.AgentPort = 6831;
})

Zipkin

.AddZipkinExporter(o =>
{
    o.Endpoint = new Uri("http://localhost:9411/api/v2/spans");
})

OTLP (OpenTelemetry Protocol)

.AddOtlpExporter(o =>
{
    o.Endpoint = new Uri("http://localhost:4317");
    o.Protocol = OtlpExportProtocol.Grpc;
})

Azure Monitor / Application Insights

.AddAzureMonitorTraceExporter(o =>
{
    o.ConnectionString = "InstrumentationKey=...";
})

Configurable Instrumentation

V10 provides fine-grained control over which attributes are recorded to optimize performance and reduce costs.

Instrumentation Options

Configure instrumentation using BrighterInstrumentation:

using Paramore.Brighter.Observability;

var instrumentation = BrighterInstrumentation.InstrumentationOptions;

// Control Command Processor attributes
instrumentation.CommandProcessorInstrumentationOptions = new InstrumentationOptions
{
    RecordRequestInformation = true,      // Request ID, type, operation
    RecordRequestBody = false,            // Request body as JSON (expensive)
    RecordRequestContext = true           // Custom span context attributes
};

// Control Message attributes (for Producers and Consumers)
instrumentation.MessagingInstrumentationOptions = new InstrumentationOptions
{
    RecordMessageInformation = true,      // Message ID, channel, partition
    RecordMessageBody = false,            // Message payload (expensive)
    RecordMessageHeaders = true,          // Message headers
    RecordServerInformation = true        // Broker address
};

Best Practice: Only enable RecordRequestBody and RecordMessageBody in development or debugging scenarios, as they can significantly increase trace size and cost.

Command Processor Spans

When Brighter operates as a Command Processor, it creates spans for each operation:

Span Names and Operations

Operation
Span Name
Kind
Description

send

<request type> send

Internal

Command routed to single handler

publish

<request type> publish

Internal

Event routed to multiple handlers

deposit

<request type> deposit

Internal

Request transformed and stored in Outbox

clear

clear

Internal

Messages dispatched from Outbox to broker

create

<channel> create

Producer

Single message sent to broker

publish (messaging)

<channel> publish

Producer

Batch of messages sent to broker

Example: Send Operation

// Creates span: "MyNamespace.ProcessOrderCommand send"
await commandProcessor.SendAsync(new ProcessOrderCommand { OrderId = 123 });

Command Processor Attributes

Attribute
Type
Description
Example

paramore.brighter.requestid

string

Request ID

"1234-5678-9012-3456"

paramore.brighter.requestids

string

Batch: comma-separated IDs

"1234..., 2345..."

paramore.brighter.requesttype

string

Full type name

"MyNamespace.MyCommand"

paramore.brighter.request_body

string

Request as JSON

{"orderId": 123}

paramore.brighter.operation

string

Operation performed

"send"

paramore.brighter.spancontext.*

varies

Custom context attributes

spancontext.userid: "1234"

Adding Custom Span Attributes

You can add custom attributes via the Request Context:

var context = new RequestContext();
context.Bag["paramore.brighter.spancontext.userid"] = userId;
context.Bag["paramore.brighter.spancontext.tenantid"] = tenantId;

await commandProcessor.SendAsync(command, context);

Any context bag entries starting with paramore.brighter.spancontext. will be added as span attributes.

Handler Pipeline Events

Brighter records an event for each handler entered in the pipeline:

Attribute
Type
Description
Example

paramore.brighter.handlername

string

Full handler type name

"MyNamespace.MyHandler"

paramore.brighter.handlertype

string

Sync or async

"async"

paramore.brighter.is_sink

bool

Final handler in chain

true

Dispatcher (Consumer) Spans

When Brighter operates as a Dispatcher (message consumer), it creates spans for each message received:

Span Names

Transport Type
Span Name
Kind
Description

Pull-based (Kafka)

<channel> receive

Consumer

Message pulled from broker

Push-based (RabbitMQ)

<channel> process

Consumer

Message pushed by broker

Example Flow

Dispatcher Span: "task.commands receive" (Consumer)
  └─> Message Translation (sibling)
  └─> Command Processor Span: "ProcessTaskCommand send" (Internal)
      └─> Handler Events

Message Attributes

Attribute
Type
Description
Example

messaging.system

string

Broker type

"rabbitmq", "kafka"

messaging.destination

string

Channel name

"task.commands"

messaging.operation

string

Operation type

"receive", "process"

messaging.message_id

string

Message ID

"msg-1234"

messaging.destination.partition.id

string

Partition ID (Kafka)

"0"

messaging.message.body.size

int

Payload size in bytes

1024

server.address

string

Broker address

"localhost:5672"

Outbox Tracing

Outbox operations create child spans for database operations:

Deposit Operation

deposit span (Internal)
  └─> Transform pipeline spans
  └─> Outbox add span (Database)

Clear Operation

create/clear span (Internal)
  └─> Outbox get span (Database)
  └─> Produce message span (Producer)
  └─> Outbox mark dispatched span (Database)

Database Span Attributes

Outbox and Inbox database operations follow OTel Database Semantic Conventions:

Attribute
Description
Example

db.system

Database type

"mysql", "postgresql"

db.name

Database name

"myapp"

db.operation

Operation type

"outbox_add", "outbox_get"

Inbox Tracing

Inbox operations create child spans for deduplication checks:

Dispatcher receive span (Consumer)
  └─> Message translation
      └─> Inbox check span (Database)
  └─> Command Processor send span (Internal)

Inbox Operations

Operation
Span Name
Description

Check

inbox_check

Check if message already processed

Add

inbox_add

Record message as processed

Transform Pipeline Tracing

Transform operations (Claim Check, Compression, Encryption) create child spans for external calls:

Claim Check (S3 Example)

deposit span (Internal)
  └─> ClaimCheck transform span
      └─> S3 put object span (HTTP Client)
          Attributes: s3.bucket, s3.key, http.request.method

Retrieve Claim

Message translation span
  └─> RetrieveClaim transform span
      └─> S3 get object span (HTTP Client)
          Attributes: s3.bucket, s3.key, http.request.method

External call spans follow their respective OTel conventions:

W3C TraceContext Propagation

Brighter automatically propagates trace context across service boundaries using W3C TraceContext headers.

How It Works

  1. Producer: Brighter injects traceparent and tracestate into message headers

  2. Consumer: Brighter extracts traceparent and tracestate to continue the trace

Message Headers

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: congo=t61rcWkgMzE

Integration with ASP.NET

Brighter participates in existing traces. When called from an ASP.NET controller, the Command Processor span becomes a child of the ASP.NET request span:

ASP.NET Request: "POST /orders"
  └─> Command Processor: "ProcessOrderCommand send"
      └─> Handler: OrderHandler
      └─> Publish: "OrderCreatedEvent publish"
          └─> Outbox add

CloudEvents Integration

When using CloudEvents, Brighter propagates trace context via the CloudEvents Distributed Tracing Extension.

CloudEvents Attributes

CloudEvents adds alternative attribute names following CloudEvents Semantic Conventions:

Messaging Convention
CloudEvents Convention
Value

messaging.message_id

cloudevents.event_id

Message ID

messaging.destination

cloudevents.event_source

Event source

N/A

cloudevents.event_type

Event type

Enabling CloudEvents Conventions

var instrumentation = BrighterInstrumentation.InstrumentationOptions;

instrumentation.MessagingInstrumentationOptions.UseCloudEventsConventionsAttributes = true;
instrumentation.MessagingInstrumentationOptions.UseMessagingSemanticConventionsAttributes = true;

You can enable both conventions simultaneously, and both sets of attributes will be recorded.

Complete Configuration Example

Producer Service

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using Paramore.Brighter;
using Paramore.Brighter.Observability;

var builder = WebApplication.CreateBuilder(args);

// Configure OpenTelemetry
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .SetResourceBuilder(ResourceBuilder.CreateDefault()
                .AddService("OrderService"))
            .AddSource("paramore.brighter")
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddOtlpExporter(o =>
            {
                o.Endpoint = new Uri("http://localhost:4317");
            });
    });

// Configure Brighter instrumentation
var instrumentation = BrighterInstrumentation.InstrumentationOptions;
instrumentation.CommandProcessorInstrumentationOptions.RecordRequestInformation = true;
instrumentation.MessagingInstrumentationOptions.RecordMessageInformation = true;
instrumentation.MessagingInstrumentationOptions.RecordServerInformation = true;

// Configure Brighter
builder.Services.AddBrighter(options =>
{
    options.HandlerLifetime = ServiceLifetime.Scoped;
})
.AddProducers(configure =>
{
    // Producer configuration
})
.AutoFromAssemblies();

var app = builder.Build();
app.Run();

Consumer Service (Dispatcher)

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using Paramore.Brighter;
using Paramore.Brighter.Observability;

var builder = Host.CreateDefaultBuilder(args);

builder.ConfigureServices(services =>
{
    // Configure OpenTelemetry
    services.AddOpenTelemetry()
        .WithTracing(tracing =>
        {
            tracing
                .SetResourceBuilder(ResourceBuilder.CreateDefault()
                    .AddService("TaskProcessor"))
                .AddSource("paramore.brighter")
                .AddOtlpExporter(o =>
                {
                    o.Endpoint = new Uri("http://localhost:4317");
                });
        });

    // Configure Brighter instrumentation
    var instrumentation = BrighterInstrumentation.InstrumentationOptions;
    instrumentation.MessagingInstrumentationOptions.RecordMessageInformation = true;
    instrumentation.MessagingInstrumentationOptions.RecordMessageBody = false; // Expensive

    // Configure Brighter Consumer
    services.AddConsumers(options =>
    {
        options.Subscriptions = subscriptions;
    })
    .AutoFromAssemblies();
});

var host = builder.Build();
await host.RunAsync();

Distributed Tracing Example

A complete distributed trace across services:

ASP.NET Request (OrderService): "POST /api/orders"
  └─> Command Processor: "CreateOrderCommand send"
      └─> Handler: CreateOrderCommandHandler
      └─> Deposit: "CreateOrderCommand deposit"
          └─> Outbox add (MySQL)

  ─── Outbox Sweeper ───

  └─> Clear: "clear"
      └─> Outbox get (MySQL)
      └─> Produce: "orders.created publish"
      └─> Outbox mark dispatched (MySQL)

  ─── Message Broker (RabbitMQ) ───

Dispatcher (TaskService): "orders.created process"
  └─> Inbox check (PostgreSQL)
  └─> Command Processor: "OrderCreatedEvent send"
      └─> Handler: SendEmailHandler
      └─> Handler: UpdateInventoryHandler
  └─> Inbox add (PostgreSQL)

Best Practices

  1. Start with Minimal Instrumentation: Enable RecordRequestInformation and RecordMessageInformation, disable expensive options like RecordRequestBody

  2. Use Sampling: Configure sampling in production to reduce costs:

    .SetSampler(new TraceIdRatioBasedSampler(0.1)) // Sample 10% of traces

  3. Add Custom Attributes Judiciously: Only add context attributes that are essential for debugging and analysis

  4. Monitor Trace Costs: Large payloads and high cardinality attributes can significantly increase observability costs

  5. Use Structured Logging: Combine tracing with structured logging for comprehensive observability

  6. Enable CloudEvents for Cross-Organization Tracing: If exchanging messages with external systems, use CloudEvents for standard trace propagation

  7. Configure Appropriate Exporters: Use OTLP for flexibility, or native exporters for specific backends

  8. Test Trace Propagation: Verify that traces flow correctly across service boundaries in development

Migration from V9

Changed Span Names

V9 Span Name
V10 Span Name

Custom handler names

<request type> <operation>

Outbox.Add

Follows database conventions

Transport-specific names

<channel> create/publish/receive/process

Changed Attributes

V9 used custom attribute names. V10 uses OTel standard conventions:

V9 Attribute
V10 Attribute

Custom attributes

paramore.brighter.* and OTel standard attributes

No standard messaging attributes

messaging.* attributes following OTel conventions

Action Required

  1. Update Dashboards: Update queries and visualizations to use V10 span names and attributes

  2. Update Alerts: Update alert rules based on new span structure

  3. Review Instrumentation Options: Configure which attributes to record based on your needs

  4. Test Trace Propagation: Verify distributed traces work correctly with V10

Troubleshooting

Traces Not Appearing

Problem: No traces appear in your observability backend.

Solutions:

  • Verify Activity Source is registered: .AddSource("paramore.brighter")

  • Check exporter configuration and endpoint

  • Ensure services can reach the exporter endpoint

  • Check firewall rules

Incomplete Traces

Problem: Traces are missing child spans or appear disconnected.

Solutions:

  • Verify traceparent header is being propagated

  • Check that all services have OpenTelemetry configured

  • Ensure consistent trace propagation format (W3C TraceContext)

  • Review CloudEvents configuration if using CloudEvents

High Trace Costs

Problem: Observability costs are too high.

Solutions:

  • Disable RecordRequestBody and RecordMessageBody

  • Reduce sampling rate: .SetSampler(new TraceIdRatioBasedSampler(0.1))

  • Disable unnecessary attribute collection

  • Use tail-based sampling to only keep interesting traces

Missing Attributes

Problem: Expected attributes are not appearing on spans.

Solutions:

  • Check Activity.IsAllDataRequested is true (controlled by sampling)

  • Verify instrumentation options are configured correctly

  • Ensure custom context attributes start with paramore.brighter.spancontext.

Additional Resources

PreviousHealth ChecksNextScheduler

Last updated

Was this helpful?