> For the complete documentation index, see [llms.txt](https://brightercommand.gitbook.io/paramore-brighter-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brightercommand.gitbook.io/paramore-brighter-documentation/health-checks-and-observability/telemetry.md). # Telemetry Brighter provides comprehensive OpenTelemetry integration for distributed tracing across message boundaries, enabling end-to-end observability in distributed systems. > **Reference** · Applies to **Brighter V10** Brighter provides comprehensive OpenTelemetry integration for distributed tracing across message boundaries, enabling end-to-end observability in distributed systems. ## OpenTelemetry Semantic Conventions **V10 introduces support for** [**OpenTelemetry Semantic Conventions**](https://opentelemetry.io/docs/concepts/semantic-conventions/), replacing the custom conventions used in V9. * [**OTel Semantic Conventions for Messaging**](https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/): Standard span names and attributes for messaging operations * [**W3C TraceContext**](https://w3c.github.io/trace-context/): Standard context propagation across service boundaries * [**CloudEvents Integration**](https://opentelemetry.io/docs/specs/semconv/cloudevents/cloudevents-spans/): Trace propagation via CloudEvents `traceparent` and `tracestate` headers * **Configurable Instrumentation**: Fine-grained control over what attributes are recorded * **Comprehensive Coverage**: Tracing for Command Processor, Dispatcher, Outbox, Inbox, and Transform pipelines *** ## Enabling Brighter's Spans Brighter records spans only when a tracer, an `IAmABrighterTracer`, is registered in your container, and `AddBrighter()` does not register one. `AddBrighterInstrumentation()`, from the `Paramore.Brighter.Extensions.Diagnostics` package, registers it and adds its `ActivitySource`, `Paramore.Brighter`, to your tracer provider: ```csharp using Microsoft.Extensions.DependencyInjection; using OpenTelemetry.Trace; using Paramore.Brighter.Extensions.Diagnostics; var services = new ServiceCollection(); services.AddOpenTelemetry() .WithTracing(tracing => tracing .AddBrighterInstrumentation() .AddOtlpExporter()); ``` `AddOpenTelemetry()` comes from `OpenTelemetry.Extensions.Hosting`, and `AddOtlpExporter()` from `OpenTelemetry.Exporter.OpenTelemetryProtocol`. Without a registered tracer, `AddSource("paramore.brighter")` listens to a source nothing writes to, and no Brighter span appears. [Configuring OpenTelemetry](/paramore-brighter-documentation/health-checks-and-observability/telemetry/configuringopentelemetry.md) shows the setup for a producer and a consumer. **From Brighter 10.8.0 the Command Processor records spans with or without an external bus.** At 10.7.0 it was given the tracer only when you configured producers with `AddProducers`; with none, `Send`, `Publish` and their async forms recorded no span, even with the tracer registered ([#4510](https://github.com/BrighterCommand/Brighter/issues/4510)). *** ## Configurable Instrumentation V10 provides fine-grained control over which attributes are recorded to optimize performance and reduce costs. ### Instrumentation Options `InstrumentationOptions` is a flags enum, so you combine the attributes you want with `|`. Set it on the Command Processor through `AddBrighter`, and on producers through `AddProducers`. Left unset, it is `None`: the spans are still created, with no Brighter attributes on them: | Flag | Records | | --------------------- | -------------------------------------------------------------------------------------------------------------------- | | `RequestInformation` | Request ID, type and operation; on messages, the CloudEvents ID, type, source and subject | | `RequestBody` | The request body as JSON; on messages, the message body (expensive) | | `RequestContext` | Nothing at 10.8.0: no span reads this flag | | `Messaging` | Messaging attributes: destination, partition, message ID and type, body size, headers | | `DatabaseInformation` | Database attributes for Outbox and Inbox operations | | `ClamCheck` | Claim check operations (the member is spelled `ClamCheck`) | | `Brighter` | Brighter's handler instrumentation | | `CircuitBreaker` | On the Outbox circuit breaker's spans, `circuit_breaker.topic` and `circuit_breaker.cooldown_count`; added at 10.8.0 | | `All` | Every flag above | | `None` | Nothing | ```csharp using Microsoft.Extensions.DependencyInjection; using Paramore.Brighter.Extensions.DependencyInjection; using Paramore.Brighter.Observability; var services = new ServiceCollection(); services.AddBrighter(options => { // Command Processor spans: request ID, type and operation options.InstrumentationOptions = InstrumentationOptions.RequestInformation; }) .AddProducers(configure => { // Producer spans: add the messaging attributes, but not the request body configure.InstrumentationOptions = InstrumentationOptions.RequestInformation | InstrumentationOptions.Messaging; }); ``` **Best Practice**: Only enable `RequestBody` in development or debugging scenarios, as it can significantly increase trace size and cost. *** ## Command Processor Spans When Brighter operates as a Command Processor, it creates spans for each operation. A span is named for the request type's name, without its namespace: ### Span Names and Operations | Operation | Span Name | Kind | Description | | --------------------- | ----------------------------------------- | -------- | -------------------------------------------------------------------------------- | | `send` | ` send` | Internal | Command routed to single handler | | `create` | ` create` | Internal | An event published to its handlers; the parent of one `publish` span per handler | | `publish` | ` publish` | Internal | Event dispatched to one of its handlers | | `deposit` | ` deposit` | Internal | Request transformed and stored in Outbox | | `scheduler` | ` scheduler` | Internal | Request handed to the scheduler, with a delay or a time | | `create` (clear) | `paramore.brighter.clear_messages create` | Producer | Messages cleared from the Outbox; the parent of a `clear` span | | `publish` (messaging) | ` publish` | Producer | Message sent to the broker | ### Example: Send Operation ```csharp using Paramore.Brighter; // Creates span: "ProcessOrderCommand send" await commandProcessor.SendAsync(new ProcessOrderCommand { OrderId = 123 }); ``` ### Command Processor Attributes | Attribute | Type | Description | Example | | -------------------------------- | ------ | ------------------- | ---------------------------------------- | | `paramore.brighter.request.id` | string | Request ID | `"01a0e7cf-3c53-75f1-8566-3fd80badd0a1"` | | `paramore.brighter.request.type` | string | Request type's name | `"ProcessOrderCommand"` | | `paramore.brighter.request.body` | string | Request as JSON | `{"orderId":123, ...}` | | `paramore.brighter.operation` | string | Operation performed | `"send"` | | `messaging.operation.type` | string | Operation performed | `"send"` | ### Adding Custom Span Attributes At 10.8.0 Brighter copies nothing from the request context onto a span. To record an attribute of your own, set it on `Activity.Current` in your handler: while the handler runs, that is the Command Processor's span for the request. ```csharp using System.Diagnostics; using System.Threading; using System.Threading.Tasks; using Paramore.Brighter; public class ProcessOrderHandler : RequestHandlerAsync { public override async Task HandleAsync( ProcessOrderCommand command, CancellationToken cancellationToken = default) { // Recorded on the "ProcessOrderCommand send" span Activity.Current?.SetTag("app.order_id", command.OrderId); return await base.HandleAsync(command, cancellationToken); } } ``` ### Handler Pipeline Events Brighter records an event on the span for each handler entered in the pipeline, named for the handler: | Attribute | Type | Description | Example | | -------------------------------- | ------ | ---------------------- | ----------------------- | | `paramore.brighter.handler.name` | string | Handler type's name | `"ProcessOrderHandler"` | | `paramore.brighter.handler.type` | string | Sync or async | `"async"` | | `paramore.brighter.is_sink` | bool | Final handler in chain | `true` | *** ## Dispatcher (Consumer) Spans When Brighter operates as a Dispatcher (message consumer), each message pump creates spans named for the routing key it reads, whichever transport it reads from: ### Span Names | Span Name | Kind | Description | | ----------------------- | -------- | ------------------------------------------------------------------------------ | | ` begin` | Consumer | The message pump starting | | ` receive` | Consumer | One read of the channel, a root span, including reads that find no message | | ` process` | Consumer | One message handled; its parent is the producer's span, carried in the message | ### Example Flow Measured with the in-memory transport, a published event and its handler: ``` order.placed publish (Producer, the sending service) └─> order.placed process (Consumer) └─> OrderPlaced create (Internal) └─> OrderPlaced publish (Internal) └─> Handler events: OrderPlacedHandler ``` ### Message Attributes | Attribute | Type | Description | Example | | ------------------------------------ | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | | `messaging.system` | string | The transport the pump reads, such as `postgres`, or `internal_bus` for the in-memory bus. At 10.7.0 it was always `internal_bus` on pump spans | `"postgres"` | | `messaging.destination.name` | string | Routing key | `"order.placed"` | | `messaging.operation.type` | string | Operation type | `"receive"`, `"process"` | | `messaging.message.id` | string | Message ID | `"01a0e7cf-3c70-7089-9e82-425719101515"` | | `messaging.message.type` | string | Message type | `"MT_EVENT"` | | `messaging.destination.partition.id` | string | Partition key | `"customer-12345"` | | `messaging.message.body.size` | int | Payload size in bytes | `66` | | `paramore.brighter.handled_count` | int | Times the message has been handled | `0` | *** ## Outbox Tracing Outbox operations create child spans for database operations, each named ` `: ### Deposit Operation ``` OrderPlaced deposit (Internal) └─> Mapper and transform events: JsonMessageMapper`1, CloudEventsTransformer └─> add.message outbox requests (Client) ``` ### Clear Operation ``` paramore.brighter.clear_messages create (Producer) └─> retrieve.message outbox requests (Client) └─> paramore.brighter.clear_messages clear (Producer) └─> order.placed publish (Producer) └─> count.outstanding_messages outbox requests (Client) order.placed settle (Producer), a trace of its own └─> mark_as_dispatched.outstanding_messages outbox requests (Client) ``` ### Database Span Attributes Outbox and Inbox database operations follow [OTel Database Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/database/database-spans/): | Attribute | Description | Example | | ------------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------- | | `db.system` | Database type; `brighter` for the in-memory stores | `"postgresql"` | | `db.name` | The configuration's database name, not the server's | `"Brighter"` | | `db.table` | Table | `"outbox"` | | `db.operation` | Operation | `"add.message"`, `"retrieve.message"`, `"mark_as_dispatched.outstanding_messages"` | | `db.operation.name` | SQL verb, on relational stores | `"INSERT"` | | `db.query.text` | SQL, on relational stores | `INSERT INTO {0} (...) VALUES (...)` | The PostgreSQL examples above are from `PostgreSqlOutbox`; `db.name` is the `databaseName` of its `RelationalDatabaseConfiguration`, which defaults to `Brighter`. *** ## Inbox Tracing Inbox operations create child spans for deduplication checks, under the span of the request the Inbox guards: ``` order.placed process (Consumer) └─> OrderPlaced create (Internal) └─> OrderPlaced publish (Internal) └─> message.exists inbox requests (Client) └─> add.message inbox requests (Client) ``` ### Inbox Operations | Operation | Span Name | Description | | --------- | ---------------------------------------- | ---------------------------------- | | Check | `message.exists
` | Check if message already processed | | Add | `add.message
` | Record message as processed | *** ## Transform Pipeline Tracing The Claim Check transform creates a span for each call to its luggage store, named ` `, with `claim_check.*` attributes: `claim_check.operation`, and, with `ClamCheck` set, `claim_check.provider`, `claim_check.bucket_name`, `claim_check.id` and `claim_check.content_lenght` (the attribute is spelled that way). ### Claim Check ``` BigEvent deposit (Internal) └─> store.message (Client) ``` ### Retrieve Claim ``` big.event process (Consumer) └─> retrieve.message (Client) └─> delete.message (Client) ``` The `delete.message` span appears because `[RetrieveClaim]` defaults to `retain: false`. The trees were measured with the in-memory luggage store, whose provider and bucket are both `in-memory`. *** ## W3C TraceContext Propagation Brighter automatically propagates trace context across service boundaries using [W3C TraceContext](https://w3c.github.io/trace-context/). ### How It Works 1. **Producer**: Brighter writes the producer span's `traceparent` and `tracestate` into the message 2. **Consumer**: Brighter reads them, and the `process` span becomes a child of the producer's span, in the same trace ### Message Headers Each transport names the headers its own way. Measured on RabbitMQ (`Paramore.Brighter.MessagingGateway.RMQ.Async`) and Kafka: ``` RabbitMQ: cloudEvents_traceparent: 00-3519372db2bb5402dc1be3b11c7fc4e5-8675f8104bc40dbe-01 cloudevents_tracestate: congo=t61rcWkgMzE Kafka: ce_traceparent: 00-c37cfd2c01adac953978b1abfb2c8d4c-3ae8fe73da7b454f-01 ce_tracestate: congo=t61rcWkgMzE ``` RabbitMQ also writes the trace state under `cloudevents_:tracestate`, an older spelling kept for consumers that still read it. Brighter injects through OpenTelemetry's default propagator, which the OpenTelemetry SDK sets up; without the SDK, nothing is written. ### Integration with ASP.NET Brighter participates in existing traces. When called from an ASP.NET controller, the Command Processor span becomes a child of the ASP.NET request span: ``` ASP.NET Request: "POST /orders" └─> Command Processor: "ProcessOrderCommand send" └─> Handler events: OrderHandler ``` *** ## CloudEvents Integration When using [CloudEvents](/paramore-brighter-documentation/using-an-external-bus/cloudeventssupport.md), Brighter propagates trace context via the [CloudEvents Distributed Tracing Extension](https://github.com/cloudevents/spec/blob/main/cloudevents/extensions/distributed-tracing.md). ### CloudEvents Attributes CloudEvents adds alternative attribute names following [CloudEvents Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/cloudevents/cloudevents-spans/): | Messaging Convention | CloudEvents Convention | Value | | ---------------------- | -------------------------------- | ------------- | | `messaging.message.id` | `cloudevents.event_id` | Message ID | | N/A | `cloudevents.event_source` | Event source | | N/A | `cloudevents.event_type` | Event type | | N/A | `cloudevents.event_subject` | Event subject | | N/A | `cloudevents.event_spec_version` | `"1.0"` | ### Enabling CloudEvents Conventions There is no separate switch for the CloudEvents attributes. Brighter records them on message spans whenever `InstrumentationOptions.RequestInformation` is set, and records the messaging attributes when `InstrumentationOptions.Messaging` is set: ```csharp using Microsoft.Extensions.DependencyInjection; using Paramore.Brighter.Extensions.DependencyInjection; using Paramore.Brighter.Observability; var services = new ServiceCollection(); services.AddBrighter() .AddProducers(configure => { configure.InstrumentationOptions = InstrumentationOptions.RequestInformation | InstrumentationOptions.Messaging; }); ``` Set both flags and both sets of attributes will be recorded. *** ## Outbox Sweeper Metrics The [Outbox Sweeper](/paramore-brighter-documentation/outbox-and-inbox/brighteroutboxsupport.md#implicit-clear) runs in the background, so when it stalls nothing fails visibly. Messages simply stay in the Outbox. To make a stall visible, the sweeper records five instruments about its own health on the `Paramore.Brighter` meter. They tell you whether sweeps are running, whether they start on time, how long they take, how long the sweep in progress has been running, and how many topics the outbox circuit breaker has tripped. Everything in this section applies from Brighter 10.8.1. ### Enabling the Metrics `AddBrighterInstrumentation()` on the `MeterProviderBuilder` registers the sweeper's meter, alongside Brighter's messaging and database meters, and listens to the `Paramore.Brighter` meter. It comes from the `Paramore.Brighter.Extensions.Diagnostics` package: ```csharp using Microsoft.Extensions.DependencyInjection; using OpenTelemetry.Metrics; using Paramore.Brighter.Extensions.Diagnostics; builder.Services.AddOpenTelemetry() .WithMetrics(metrics => metrics .AddBrighterInstrumentation() .AddOtlpExporter()); ``` If you do not call `AddBrighterInstrumentation()`, the sweeper uses a no-op meter and records nothing. ### Instruments | Instrument | Type | Unit | Description | | ----------------------------------------------------- | ------------------ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `paramore.brighter.outbox_sweeper.tick.lag` | Histogram | `s` | How long after its due time a sweep started | | `paramore.brighter.outbox_sweeper.sweep.duration` | Histogram | `s` | How long a sweep took | | `paramore.brighter.outbox_sweeper.sweeps` | Counter | `{sweep}` | Number of sweeps, by how each ended | | `paramore.brighter.outbox_sweeper.sweep.running_time` | Gauge (observable) | `s` | How long the sweep in progress has been running, including any wait for the [distributed lock](/paramore-brighter-documentation/outbox-and-inbox/distributedlock.md), or 0 when no sweep is in progress | | `paramore.brighter.outbox_sweeper.tripped_topics` | Gauge (observable) | `{topic}` | Number of topics whose outbox circuit breaker is tripped | `sweep.duration` and `sweeps` carry a `paramore.brighter.outbox_sweeper.outcome` attribute: | Outcome | Meaning | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `completed` | The sweep ran to the end | | `failed` | The sweep threw an exception. The sweeper logs it and sweeps again on the next tick | | `lock_unavailable` | The sweep did not run, because another sweeper holds the [distributed lock](/paramore-brighter-documentation/outbox-and-inbox/distributedlock.md) | The two gauges carry no attributes. They read the sweeper's state each time your exporter collects, rather than being recorded when a sweep ends, so they report while a sweep is still running. Each reports 0 if it cannot read what it needs: Brighter logs a warning, and the sweeper carries on sweeping. `tripped_topics` reads the `IAmAnOutboxCircuitBreaker` registered in your service provider, and reports 0 when none is registered. Register the circuit breaker as a singleton shared with the outbox producer mediator, or the gauge may not match the topics the sweeper skips. It counts only while this instance is the active sweeper — the one that obtained the distributed lock — so it reads 0 on an instance that does not hold the lock, and while a sweep is waiting for the lock. `InMemoryOutboxCircuitBreaker` is per instance, so the count is one instance's view. These metrics describe the sweeper, not the messages it sends. They do not count how many messages a sweep found or dispatched. ### How the Sweeper Keeps Time The sweeper runs on its own dedicated thread, not on a thread-pool timer. So a busy or starved thread pool cannot stop sweeps from starting. Each sweep is due one `TimerInterval` after the previous one was due: * If a sweep overruns its interval, the next sweep starts late, and `tick.lag` shows by how much. * If a sweep starts a whole interval or more late, the schedule restarts from that sweep. The sweeper never runs sweeps back to back to catch up. The sweep itself also stays on that thread: when a send completes, as Kafka's does, on the broker client's own thread, the sweep resumes on the sweeper's thread rather than waiting for a pool thread. A producer or Outbox that awaits with `ConfigureAwait(false)` internally, as database clients and the AWS SDK do, still resumes on the pool. With those, a starved pool shows up first as a longer `sweep.duration`, and then as `tick.lag` on the sweeps that follow. The schedule is measured in elapsed time, not by the wall clock, so a clock that steps backwards does not delay a sweep. Because sweeps run one after another on that one thread, **a hung sweep records nothing on the histograms or the counter**. No later sweep starts while it is stuck, so there is no further `tick.lag`, and no `sweep.duration` or `sweeps`, until it ends. `lock_unavailable` shows up only on *other* instances that share the distributed lock, and with a single instance or the in-memory lock, not even that. So don't use `lock_unavailable` or `tick.lag` to detect a stall. The `sweep.running_time` gauge does see it: it reads the sweep in progress each time it is collected, so a hung sweep shows as a value that keeps growing. Alert on that, as in [Suggested Alerts](#suggested-alerts). The same thread also means that code in the sweep's path that blocks on a task, such as a custom distributed lock, producer or Outbox that calls `.GetAwaiter().GetResult()` on an async method, can deadlock the sweeper. Brighter's own implementations don't do this. ### Publish Confirmation Queue Depth Kafka and RabbitMQ (`RMQ.Sync`) producers raise publish confirmations on a dedicated thread per producer, in batches of up to 32. Brighter's own handler for a confirmation marks the message dispatched in the Outbox. If confirmations arrive faster than those handlers finish, they queue. Messages then stay unmarked past `MinimumMessageAge`, and the sweeper sends them again. | Instrument | Type | Unit | Attributes | | ---------------------------------------------------- | -------------------------- | ---------------- | ------------------------------------------------ | | `paramore.brighter.publish_confirmation.queue.depth` | UpDownCounter (observable) | `{confirmation}` | `messaging.system`, `messaging.destination.name` | It counts the confirmations waiting or running for each producer. `AddBrighterInstrumentation()` registers it. A depth that keeps growing means the Outbox cannot keep up with the broker's confirmations: ```promql min_over_time(sum by (messaging_destination_name) (paramore_brighter_publish_confirmation_queue_depth)[5m:]) > 100 ``` ### Suggested Alerts The examples below use PromQL, with the names a Prometheus exporter produces: dots become underscores, and the unit and `_total` are added as suffixes. Adjust them for your backend. **The sweeper has stalled.** The sweep in progress has run for more than three `TimerInterval`s. With a 5-second `TimerInterval`, this alerts once a sweep has run for 15 seconds: ```promql max(paramore_brighter_outbox_sweeper_sweep_running_time_seconds) > 15 ``` This catches a hung sweep directly, on the instance where it is hung. Size the threshold above the longest healthy sweep, which you can read from the `sweep.duration` histogram. **No sweep is completing.** No sweep has completed for several intervals. This also fires when no instance is sweeping at all, which `sweep.running_time` cannot show. With a 5-second `TimerInterval`, this alerts after about 30 seconds: ```promql sum(increase(paramore_brighter_outbox_sweeper_sweeps_total{paramore_brighter_outbox_sweeper_outcome="completed"}[30s])) == 0 ``` Sum across instances, and count only `completed`. When several instances share a distributed lock, only one of them completes sweeps, and the rest record `lock_unavailable`. An alert on each instance would fire for every instance that is not holding the lock. An alert on sweeps of any outcome would never fire while the lock holder is hung, because the other instances keep recording `lock_unavailable`. A stalled sweeper may also stop reporting altogether, so pair the alert with `absent()` on the same series if your backend drops series that go stale. **Sweeps start late.** The p99 tick lag is above the sweep interval: ```promql histogram_quantile(0.99, sum by (le) (rate(paramore_brighter_outbox_sweeper_tick_lag_seconds_bucket[5m]))) > 5 ``` **Sweeps are failing.** ```promql sum(increase(paramore_brighter_outbox_sweeper_sweeps_total{paramore_brighter_outbox_sweeper_outcome="failed"}[5m])) > 0 ``` To find the reason for a failure, check the sweeper's error log. **Topics are tripped.** The active sweeper is skipping topics whose outbox circuit breaker has tripped, so messages for them are not swept until the cooldown ends. Because the gauge reads 0 while the sweeper waits for the lock, a single sample can miss a tripped topic; take the maximum over a window of three `TimerInterval`s: ```promql max(max_over_time(paramore_brighter_outbox_sweeper_tripped_topics[15s])) > 0 ``` ### Histogram Buckets The OpenTelemetry .NET SDK's default histogram buckets were designed for millisecond values: 0, 5, 10, 25, 50 and so on. Both sweeper histograms record seconds, and most values fall below 5, so with the default buckets nearly every value lands in the first one or two. Add a view that sets buckets that fit your `TimerInterval`: ```csharp using Microsoft.Extensions.DependencyInjection; using OpenTelemetry.Metrics; using Paramore.Brighter.Extensions.Diagnostics; var sweeperBuckets = new ExplicitBucketHistogramConfiguration { Boundaries = [0, 0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60] }; builder.Services.AddOpenTelemetry() .WithMetrics(metrics => metrics .AddBrighterInstrumentation() .AddView("paramore.brighter.outbox_sweeper.tick.lag", sweeperBuckets) .AddView("paramore.brighter.outbox_sweeper.sweep.duration", sweeperBuckets) .AddOtlpExporter()); ``` ### Telling a Slow Broker From a Starved Process | `sweep.duration` | `tick.lag` | Likely cause | | ---------------- | ---------- | -------------------------------------------------------------------------------------------------------------------- | | Low | Low | Healthy | | High | Low | The broker or Outbox store is slow, but each sweep still finishes within its interval | | High | High | Sweeps overrun their interval. Check whether the broker or Outbox is slow, or the thread pool is starved (see below) | | Low | High | The process is not getting CPU, for example because of container CPU throttling | To see whether the thread pool is starved, enable the .NET runtime's built-in `System.Runtime` meter (.NET 9 and later) and look at the thread pool's queue length and thread count: ```csharp using Microsoft.Extensions.DependencyInjection; using OpenTelemetry.Metrics; using Paramore.Brighter.Extensions.Diagnostics; builder.Services.AddOpenTelemetry() .WithMetrics(metrics => metrics .AddBrighterInstrumentation() .AddMeter("System.Runtime") .AddOtlpExporter()); ``` A `dotnet.thread_pool.queue.length` that keeps growing, together with a `dotnet.thread_pool.thread.count` that climbs only slowly, means work is queuing faster than the pool can add threads. On Kubernetes, compare this with CPU throttling (`container_cpu_cfs_throttled_periods_total`). Brighter does not re-export these runtime metrics. See [.NET runtime metrics](https://learn.microsoft.com/dotnet/core/diagnostics/built-in-metrics-runtime). *** ## Telemetry Best Practices 1. **Start with Minimal Instrumentation**: Enable `RequestInformation` and `Messaging`, and leave out expensive flags like `RequestBody` 2. **Use Sampling**: Configure sampling in production to reduce costs: ```csharp using Microsoft.Extensions.DependencyInjection; using OpenTelemetry.Trace; using Paramore.Brighter.Extensions.Diagnostics; var services = new ServiceCollection(); services.AddOpenTelemetry() .WithTracing(tracing => tracing .AddBrighterInstrumentation() .SetSampler(new TraceIdRatioBasedSampler(0.1))); // Sample 10% of traces ``` 3. **Add Custom Attributes Judiciously**: Only add attributes that are essential for debugging and analysis 4. **Monitor Trace Costs**: Large payloads and high cardinality attributes can significantly increase observability costs 5. **Use Structured Logging**: Combine tracing with structured logging for comprehensive observability 6. **Enable CloudEvents for Cross-Organization Tracing**: If exchanging messages with external systems, use CloudEvents for standard trace propagation 7. **Configure Appropriate Exporters**: Use OTLP for flexibility, or native exporters for specific backends 8. **Test Trace Propagation**: Verify that traces flow correctly across service boundaries in development *** ## Migration from V9 ### Changed Span Names | V9 Span Name | V10 Span Name | | ------------------------ | ---------------------------------------------------------------------------------- | | Custom handler names | ` ` | | `Outbox.Add` | `
`, such as `add.message outbox requests` | | Transport-specific names | ` publish`, and ` begin/receive/process` on the consumer | ### Changed Attributes V9 used custom attribute names. V10 uses OTel standard conventions: | V9 Attribute | V10 Attribute | | -------------------------------- | --------------------------------------------------- | | Custom attributes | `paramore.brighter.*` and OTel standard attributes | | No standard messaging attributes | `messaging.*` attributes following OTel conventions | ### Action Required 1. **Update Dashboards**: Update queries and visualizations to use V10 span names and attributes 2. **Update Alerts**: Update alert rules based on new span structure 3. **Review Instrumentation Options**: Configure which attributes to record based on your needs 4. **Test Trace Propagation**: Verify distributed traces work correctly with V10 *** ## Telemetry Troubleshooting ### Traces Not Appearing **Problem**: No traces appear in your observability backend. **Solutions**: * Verify Brighter's tracer is registered: `.AddBrighterInstrumentation()`, as in [Enabling Brighter's Spans](#enabling-brighters-spans) * Check exporter configuration and endpoint * Ensure services can reach the exporter endpoint * Check firewall rules ### Incomplete Traces **Problem**: Traces are missing child spans or appear disconnected. **Solutions**: * Verify `traceparent` header is being propagated * Check that all services have OpenTelemetry configured * Ensure consistent trace propagation format (W3C TraceContext) * Review CloudEvents configuration if using CloudEvents ### High Trace Costs **Problem**: Observability costs are too high. **Solutions**: * Leave `RequestBody` out of `InstrumentationOptions` * Reduce sampling rate: `.SetSampler(new TraceIdRatioBasedSampler(0.1))` * Disable unnecessary attribute collection * Use tail-based sampling to only keep interesting traces: `Paramore.Brighter.Extensions.Diagnostics` adds `SetTailSampler()` to the tracer provider builder ### Missing Attributes **Problem**: Expected attributes are not appearing on spans. **Solutions**: * Check `Activity.IsAllDataRequested` is true (controlled by sampling) * Verify instrumentation options are configured correctly * Set attributes of your own on `Activity.Current` in the handler, as in [Adding Custom Span Attributes](#adding-custom-span-attributes): Brighter copies nothing from the request context *** ## Further Reading * [Configuring OpenTelemetry](/paramore-brighter-documentation/health-checks-and-observability/telemetry/configuringopentelemetry.md) - Wiring up the SDK, exporters and a worked producer/consumer example * [OpenTelemetry Semantic Conventions for Messaging](https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/) * [W3C TraceContext Specification](https://w3c.github.io/trace-context/) * [CloudEvents Distributed Tracing Extension](https://github.com/cloudevents/spec/blob/main/cloudevents/extensions/distributed-tracing.md) * [OpenTelemetry .NET Documentation](https://opentelemetry.io/docs/instrumentation/net/) * [Brighter CloudEvents Support](/paramore-brighter-documentation/using-an-external-bus/cloudeventssupport.md) * [Brighter Request Context](/paramore-brighter-documentation/commands-handlers-and-pipelines/buildingapipeline/usingthecontextbag.md) * [Outbox Support](/paramore-brighter-documentation/outbox-and-inbox/brighteroutboxsupport.md) - The Outbox Sweeper, whose health metrics are described above * [.NET Runtime Metrics](https://learn.microsoft.com/dotnet/core/diagnostics/built-in-metrics-runtime) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://brightercommand.gitbook.io/paramore-brighter-documentation/health-checks-and-observability/telemetry.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.