How to add manual instrumentation in Python

Manual instrumentation gives you fine-grained control when automatic instrumentation alone cannot express important business operations. Use it to capture steps that matter for debugging, to attach business-specific attributes, or to guarantee that failures surface with the right context in SigNoz.

Prerequisites

Finish the core Python setup in the OpenTelemetry Python instrumentation guide so exporters, tracer provider, and propagators are configured.

Install the API and SDK packages:

pip install opentelemetry-api opentelemetry-sdk

Tested with Python 3.11 and OpenTelemetry Python SDK v1.27.0.

Step 1. Create manual spans

Initialize a tracer and wrap important work inside custom spans:

from opentelemetry import trace
 
tracer = trace.get_tracer("order-service")
 
def process_order(order_id: str):
    with tracer.start_as_current_span("process-order") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.status", "processing")
 
        # Business logic here
        # Child spans created in called functions will be linked automatically
        validate_order(order_id)
 
    # Span ends automatically when exiting the 'with' block

For nested operations, create child spans that link to the parent:

def process_order(order_id: str):
    with tracer.start_as_current_span("process-order") as parent:
        parent.set_attribute("order.id", order_id)
 
        # Nested span tracks a sub-operation
        with tracer.start_as_current_span("validate-inventory") as child:
            child.set_attribute("warehouse.id", "WH-001")
            check_inventory(order_id)

Tips:

Reuse tracer instances instead of creating a new one for each request.
Start spans with descriptive names that match business steps (checkout, fetch-user, etc.).
Use with blocks or decorators to ensure spans always end.

Using decorators

For functions where the span should cover the entire execution:

@tracer.start_as_current_span("do_work")
def do_work():
    print("doing some work...")
    # Span is created on function entry and ends on exit

Step 2. Propagate context

Python stores the active span in a contextvars context variable, so start_as_current_span links child spans for you inside a single call chain. Two things break that chain: a new thread, and another service. See Context Propagation for the concepts behind this step.

Inside your process

start_as_current_span sets the span as active. start_span creates the span without activating it, so anything nested under it becomes a sibling instead of a child:

# validate_order's spans become children of process-order
with tracer.start_as_current_span("process-order"):
    validate_order(order_id)
 
# validate_order's spans do NOT become children of process-order
span = tracer.start_span("process-order")
validate_order(order_id)
span.end()

An asyncio task copies the current context when you create it, so spans inside a task nest correctly. A threading.Thread starts with an empty context, so you have to carry it across by hand:

from concurrent.futures import ThreadPoolExecutor
from opentelemetry import context as otel_context
 
def run_with_context(ctx, fn, *args):
    token = otel_context.attach(ctx)
    try:
        return fn(*args)
    finally:
        otel_context.detach(token)
 
with tracer.start_as_current_span("process-batch"):
    ctx = otel_context.get_current()
    with ThreadPoolExecutor() as pool:
        pool.submit(run_with_context, ctx, handle_item, item)

Across a service boundary

Instrumentation libraries for Flask, Django, requests, and Celery inject and extract for you. Write the calls yourself for any hop they do not cover.

Set the span kind on both ends. SigNoz reads Client and Server to build the service map and APM metrics.

from opentelemetry import trace
from opentelemetry.propagate import inject, extract
 
# Outgoing: inject writes traceparent into the carrier
def charge_card(payload):
    with tracer.start_as_current_span("charge-card", kind=trace.SpanKind.CLIENT):
        headers = {}
        inject(headers)
        return requests.post("https://payments.internal/charge", json=payload, headers=headers)
 
# Incoming: extract returns a context to start the entry span in
def handle_charge(request):
    ctx = extract(request.headers)
    with tracer.start_as_current_span(
        "handle-charge", context=ctx, kind=trace.SpanKind.SERVER
    ) as span:
        # handle-charge is now a child of charge-card in the caller's trace
        capture_payment(span)

Two details in Python fail without an error message.

The first is argument order. inject and extract take the carrier first and the context as an optional keyword: inject(carrier, context=None) and extract(carrier, context=None). Go, Java, and JavaScript put the context first, so a snippet ported from another language fails here.

The second is header case. The default getter looks up the literal lowercase key traceparent with a plain carrier.get(key). Flask and Django hand you a case-insensitive mapping, so extract(request.headers) works. A plain dict does not. A carrier that holds Traceparent returns an empty context and raises nothing.

When you assemble the carrier from raw headers, lowercase the keys first:

carrier = {k.lower(): v for k, v in raw_headers.items()}
ctx = extract(carrier)

For a queue, put the carrier on the message and read it back in the consumer:

# Publisher
carrier = {}
inject(carrier)
queue.publish(body=payload, headers=carrier)
 
# Consumer
ctx = extract(message.headers)
with tracer.start_as_current_span("process-message", context=ctx):
    handle(message)

Step 3. Add attributes and events

Attributes show up as key-value pairs in SigNoz so you can filter and aggregate spans. Events capture notable moments inside a span.

from opentelemetry import trace
 
def handle_payment(amount: float, currency: str = "USD"):
    span = trace.get_current_span()
 
    span.set_attribute("payment.amount", amount)
    span.set_attribute("payment.currency", currency)
    span.set_attribute("payment.method", "credit_card")
 
    # Events mark notable moments within the span
    span.add_event("payment.validated")
 
    # Process payment...
 
    span.add_event("payment.processed", {
        "status": "success",
        "transaction.id": "txn_123456"
    })

For semantic attributes, use the conventions package:

pip install opentelemetry-semantic-conventions

from opentelemetry import trace
from opentelemetry.semconv.attributes.http_attributes import HTTP_REQUEST_METHOD
from opentelemetry.semconv.attributes.url_attributes import URL_FULL
 
span = trace.get_current_span()
span.set_attribute(HTTP_REQUEST_METHOD, "GET")
span.set_attribute(URL_FULL, "https://api.example.com/users")

Keep attribute keys consistent (use semantic conventions when possible).
Use events to mark retries, cache hits/misses, queue waits, and similar milestones.

Step 4. Record errors

Flag failures on the span so they are easy to query in SigNoz.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
 
def risky_operation():
    span = trace.get_current_span()
 
    try:
        result = do_something_risky()
        span.set_status(Status(StatusCode.OK))
        return result
    except Exception as ex:
        span.record_exception(ex)
        span.set_status(Status(StatusCode.ERROR, str(ex)))
        raise

record_exception attaches stack trace and message details.
Setting status to StatusCode.ERROR surfaces the span in SigNoz error views and alerts.
Re-raise the exception so calling code can respond appropriately.

Step 5. Add span links (optional)

Links connect spans that are causally related but not in a parent-child relationship:

from opentelemetry import trace
 
tracer = trace.get_tracer(__name__)
 
# First operation
with tracer.start_as_current_span("span-1"):
    ctx = trace.get_current_span().get_span_context()
    link_from_span_1 = trace.Link(ctx)
 
# Second operation linked to the first
with tracer.start_as_current_span("span-2", links=[link_from_span_1]):
    # span-2 is causally associated with span-1, but not a child
    pass

Validate

Trigger the code paths that emit manual spans.
In SigNoz Traces, filter by service.name or your span name.
Open a trace and verify attributes, events, and error status.
Filter traces with has_error = true in the Trace Explorer to confirm failures show up with recorded exceptions.

Troubleshooting

Still not seeing data in SigNoz? Work through Debug missing traces, logs, and metrics, which covers SDK diagnostics, Collector connectivity, and the common ingestion errors for all three signals.

Why don't I see my custom spans in SigNoz?

Make sure that the tracer provider is initialized before your application code runs.
Check sampler configuration. Using ratio-based sampling in dev may drop most manual spans.
Verify traffic hits the functions where you inserted spans.

Why are child spans missing even though I create them?

Ensure you use start_as_current_span which automatically sets the parent context.
If using start_span directly, you need to manually manage context propagation.
Check that the parent span hasn't ended before child spans are created.

Why don't attributes or events appear on the span?

Attribute values must be strings, booleans, numbers, or lists of these types.
Call set_attribute or add_event before the span ends. Post-end mutations are ignored.
When using with blocks, add attributes inside the block before it exits.

Why does the downstream service start its own trace?

Make sure that the caller runs inject and that the outgoing request carries a traceparent header.
Make sure that the receiver passes the context that extract returned into start_as_current_span(..., context=ctx).
Check the argument order. inject(carrier) takes the carrier first, not the context.
Check the header case. The default getter is case-sensitive, so a carrier holding Traceparent rather than traceparent extracts nothing and reports no error.
Make sure that both services register the same propagator format. See Context Propagation.

Spans not connected across async operations?

For async code, use trace.get_current_span() within the async function.
Consider using contextvars or explicitly passing span context for complex async flows.

Next steps

Send logs from Python using auto-instrumentation
Return to the Python instrumentation guide for framework-specific setup
Read Context Propagation for the wire format and the failure modes shared across languages