How to add manual instrumentation in Rust

Manual instrumentation gives you fine-grained control when you need to capture specific business operations, attach custom attributes, or ensure failures surface with the right context in SigNoz.

Manual instrumentation steps are identical across deployments—only your OTLP endpoint and ingestion key differ.

Prerequisites

Complete the Rust OpenTelemetry instrumentation guide to set up the tracer provider and exporter.
Tested with Rust 1.75+ and OpenTelemetry Rust SDK v0.31.0.

Step 1. Acquire a tracer

Get a tracer instance to create spans. Reuse tracers instead of creating new ones for each request:

use opentelemetry::global;
use opentelemetry::trace::Tracer;
 
fn get_tracer() -> impl Tracer {
    global::tracer("my-service")
}

Tips:

The tracer name typically matches your service or module name.
Store the tracer in a lazy static or pass it through your application context for reuse.

Step 2. Create manual spans

Wrap important operations in spans using in_span:

use opentelemetry::trace::{Tracer, TraceContextExt};
use opentelemetry::{global, KeyValue};
 
pub fn process_order(order_id: &str) {
    let tracer = global::tracer("order-service");
 
    tracer.in_span("process-order", |cx| {
        let span = cx.span();
        span.set_attribute(KeyValue::new("order.id", order_id.to_string()));
        span.set_attribute(KeyValue::new("order.status", "processing"));
 
        // Your order processing logic here
        validate_inventory(order_id);
        charge_payment(order_id);
    });
}

The in_span closure automatically ends the span when the block completes, even if an error occurs.

Step 3. Create nested spans

For operations with sub-steps, create child spans that automatically link to their parent:

use opentelemetry::trace::{Tracer, TraceContextExt};
use opentelemetry::{global, KeyValue};
 
pub fn checkout(cart_id: &str) {
    let tracer = global::tracer("checkout-service");
 
    tracer.in_span("checkout", |cx| {
        let span = cx.span();
        span.set_attribute(KeyValue::new("cart.id", cart_id.to_string()));
 
        // Child span for validation
        tracer.in_span("validate-cart", |cx| {
            let span = cx.span();
            span.set_attribute(KeyValue::new("cart.item_count", 5i64));
            // Validation logic
        });
 
        // Child span for payment
        tracer.in_span("process-payment", |cx| {
            let span = cx.span();
            span.set_attribute(KeyValue::new("payment.method", "credit_card"));
            // Payment logic
        });
    });
}

Child spans created within the parent's in_span closure automatically inherit the trace context.

Step 4. Propagate context

Rust holds the active span in a Context stored per thread. in_span makes its span current for the duration of the closure, so anything you call inside nests correctly. Two things break that chain: a spawned task, and another service. See Context Propagation for the concepts behind this step.

Inside your process

Outside an in_span closure, build the context yourself and attach it. attach consumes the context and returns a ContextGuard that restores the previous context when it drops:

use opentelemetry::trace::{Span, TraceContextExt, Tracer};
use opentelemetry::{global, Context};
 
let tracer = global::tracer("order-service");
 
let span = tracer.start("process-order");
let cx = Context::current_with_span(span);
let _guard = cx.clone().attach();
 
// Child of process-order, because the guard is still alive
tracer.in_span("validate-cart", |_cx| validate_cart(cart_id));
 
// Or parent a span explicitly, with no guard involved
let child = tracer.start_with_context("charge-card", &cx);
 
drop(_guard);
cx.span().end(); // see the note below

Dropping the ContextGuard only restores the previous context. It does not end the span. A span you hand to Context::current_with_span lives as long as the Context that holds it. It ends when that value drops, which can happen after you shut the provider down. A span that ends after shutdown never reaches SigNoz.

End it yourself with cx.span().end() before shutdown. Calling .end() on a BoxedSpan also needs the Span trait in scope, so import opentelemetry::trace::Span. The in_span closure form has neither problem: it ends its span when the closure returns.

A spawned task can run on a different thread, so the context does not follow it. Capture the context before you spawn, then wrap the future with FutureExt::with_context:

use opentelemetry::trace::FutureExt;
use opentelemetry::Context;
 
let cx = Context::current();
 
tokio::spawn(
    async move {
        // Spans started here attach to the caller's span
        settle_order().await;
    }
    .with_context(cx),
);

Do not call cx.attach() inside a spawned task that awaits. attach returns a ContextGuard that is deliberately !Send. Holding it across an await makes the future !Send, and tokio::spawn then rejects it at compile time with "future cannot be sent between threads safely". with_context attaches the context around each poll instead, so no guard is held across the await.

Across a service boundary

HashMap<String, String> implements both Injector and Extractor, so it works as a carrier out of the box:

Set the span kind on both ends. SigNoz reads Client and Server to build the service map and APM metrics.

use opentelemetry::trace::{SpanKind, TraceContextExt, Tracer};
use opentelemetry::{global, Context};
use std::collections::HashMap;
 
// Outgoing: inject writes traceparent into the carrier
let mut carrier = HashMap::<String, String>::new();
global::get_text_map_propagator(|propagator| {
    propagator.inject_context(&Context::current(), &mut carrier);
});
// carrier is now {"traceparent": "00-<trace-id>-<span-id>-01"}
 
// Incoming: extract returns a context to parent the entry span on
let parent_cx = global::get_text_map_propagator(|propagator| propagator.extract(&carrier));
let span = tracer
    .span_builder("handle-charge")
    .with_kind(SpanKind::Server)
    .start_with_context(&tracer, &parent_cx);
let _guard = Context::current_with_span(span).attach();

The opentelemetry-http crate provides HeaderInjector and HeaderExtractor. Use them with reqwest and other http crate clients to pass a HeaderMap straight through, with no copy into a map.

inject_context and extract do nothing unless you registered a propagator with global::set_text_map_propagator. Set opentelemetry_sdk::propagation::TraceContextPropagator::new() during startup, alongside your tracer provider.

Step 5. Add attributes

Attributes are key-value pairs that provide context for filtering and analysis in SigNoz:

use opentelemetry::trace::TraceContextExt;
use opentelemetry::KeyValue;
 
pub fn handle_payment(cx: &opentelemetry::Context, amount: f64, currency: &str) {
    let span = cx.span();
 
    span.set_attribute(KeyValue::new("payment.amount", amount));
    span.set_attribute(KeyValue::new("payment.currency", currency.to_string()));
    span.set_attribute(KeyValue::new("payment.method", "credit_card"));
    span.set_attribute(KeyValue::new("payment.success", true));
}

Supported attribute value types:

String / &str
i64, f64
bool
Arrays of the above types

Step 6. Add events

Events mark notable moments within a span, like retries, cache hits, or state transitions:

use opentelemetry::trace::TraceContextExt;
use opentelemetry::KeyValue;
 
pub fn fetch_from_cache(cx: &opentelemetry::Context, key: &str) -> Option<String> {
    let span = cx.span();
 
    // Check cache
    if let Some(value) = cache_lookup(key) {
        span.add_event("cache.hit", vec![
            KeyValue::new("cache.key", key.to_string()),
        ]);
        return Some(value);
    }
 
    span.add_event("cache.miss", vec![
        KeyValue::new("cache.key", key.to_string()),
    ]);
 
    // Fetch from database and populate cache
    let value = fetch_from_db(key)?;
    span.add_event("cache.populated", vec![
        KeyValue::new("cache.key", key.to_string()),
        KeyValue::new("cache.ttl_seconds", 3600i64),
    ]);
 
    Some(value)
}

Step 7. Record errors

Flag failures on the span so they surface in SigNoz error views and alerts:

use opentelemetry::trace::{Tracer, TraceContextExt, Status};
use opentelemetry::{global, KeyValue};
 
pub fn risky_operation() -> Result<(), Box<dyn std::error::Error>> {
    let tracer = global::tracer("risky-service");
 
    tracer.in_span("risky-operation", |cx| {
        let span = cx.span();
 
        match do_something_risky() {
            Ok(result) => {
                span.set_status(Status::Ok);
                Ok(result)
            }
            Err(e) => {
                // Record the error on the span
                span.set_status(Status::error(e.to_string()));
                span.record_error(&*e);
                span.set_attribute(KeyValue::new("error.type", "RiskyOperationError"));
                Err(e)
            }
        }
    })
}

For recording exceptions with full details:

use opentelemetry::trace::TraceContextExt;
 
pub fn record_exception(cx: &opentelemetry::Context, error: &dyn std::error::Error) {
    let span = cx.span();
    span.record_error(error);
    span.set_status(opentelemetry::trace::Status::error(error.to_string()));
}

Step 8. Span links (optional)

Use span links to connect causally related traces, such as batch processing where one span triggers multiple independent operations:

use opentelemetry::trace::{Tracer, TraceContextExt, Link, SpanContext};
use opentelemetry::global;
 
pub fn process_batch(batch_id: &str, source_contexts: Vec<SpanContext>) {
    let tracer = global::tracer("batch-processor");
 
    // Create links to all source spans
    let links: Vec<Link> = source_contexts
        .into_iter()
        .map(|ctx| Link::new(ctx, vec![], 0))
        .collect();
 
    let span = tracer
        .span_builder("process-batch")
        .with_links(links)
        .start(&tracer);
 
    // Process the batch with the span active
    // ...
}

Validate

Trigger the code paths that create manual spans.
In SigNoz Traces, filter by service.name or your span name.
Open a trace and verify:
- Custom span names appear in the trace waterfall
- Attributes show in the span details panel
- Events appear in the timeline
- Errors are flagged with the correct status
Check Services → Errors to confirm failures surface with recorded exceptions.

Troubleshooting

Still not seeing data in SigNoz? Work through Debug missing traces, logs, and metrics, which covers SDK diagnostics, Collector connectivity, and the common ingestion errors for all three signals.

Why don't I see my custom spans in SigNoz?

Make sure that the tracer provider is initialized before creating spans.
Verify global::set_tracer_provider() is called at application startup.
Check that tracer_provider.shutdown() is called before the application exits to flush buffered spans.

Why are child spans not linked to their parent?

Ensure child spans are created within the parent's in_span closure.
If creating spans manually, pass the context explicitly using with_parent().

Why does the downstream service start its own trace?

Make sure that the caller runs inject_context and that the outgoing request carries a traceparent header.
Make sure that the receiver passes the extracted context to start_with_context, not start.
Make sure that you registered a propagator with global::set_text_map_propagator during startup. Without it, inject and extract are no-ops.
Make sure that both services register the same propagator format. See Context Propagation.

Why don't attributes appear on spans?

Attribute values must be supported types: String, i64, f64, bool, or arrays of these.
Set attributes before the span ends. Mutations after span.end() are ignored.
The in_span closure ends the span automatically when it returns.

Why don't errors show in SigNoz error views?

Call span.set_status(Status::error(...)) to mark the span as failed.
Use span.record_error(&error) to attach the exception details.
Ensure the span is exported (check shutdown is called properly).

Next steps

Set up alerts for error rates in your Rust application
Read Context Propagation for the wire format and the failure modes shared across languages