Daytona Sandbox Monitoring & Tracing with OpenTelemetry

SDK tracing covers the other half: how long Daytona itself takes to create, start, stop, and delete sandboxes, and which of those calls fail. It runs in your application process, not in the sandbox, so the firewall rules above do not apply.

Set otel_enabled on the client and point the standard OpenTelemetry variables at SigNoz:

export DAYTONA_API_KEY="<your-daytona-api-key>"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<region>.signoz.cloud:443"
export OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=<your-ingestion-key>"

import asyncio
 
from daytona import AsyncDaytona, DaytonaConfig
 
 
async def main():
    # The API key is read from DAYTONA_API_KEY, exported above.
    async with AsyncDaytona(DaytonaConfig(otel_enabled=True)) as daytona:
        sandbox = await daytona.create()
        await sandbox.process.code_run('print("hello")')
        await daytona.delete(sandbox)
    # Traces flush when the context manager exits
 
 
asyncio.run(main())

Set DAYTONA_OTEL_ENABLED=true instead of the constructor argument if you would rather keep it in the environment. Daytona also ships TypeScript, Ruby, Go, and Java SDKs with the same flag; see OpenTelemetry Collection.

The Python SDK bundles only the OTLP HTTP exporter, so http/protobuf is the only protocol. Do not set OTEL_EXPORTER_OTLP_PROTOCOL=grpc.

Spans flush when you close the client. The async client exposes close() and works as a context manager. The synchronous Daytona client has neither, so its spans flush only at interpreter shutdown.

One dashboard setting turns on everything Daytona instruments itself: per-sandbox resource metrics, toolbox API spans, and organization quota metrics. You write no code for any of it.

1. Open the Daytona Dashboard and go to the OpenTelemetry section. Only organization owners see it.
2. Set OTLP Endpoint to https://ingest.<region>.signoz.cloud/. Leave the port off, and keep the trailing slash. Daytona appends the signal path itself.
3. Add a header with key signoz-ingestion-key and your ingestion key as the value.
4. Save. Daytona takes up to five minutes to apply the change.

Sandboxes must also be able to reach *.daytona.io, as described in Step 1. Without it the per-sandbox signals below never leave the sandbox.

Per-sandbox metrics

The daemon reports every sandbox under service.name = sandbox-<sandbox-id>, with the sandbox id repeated in service.instance.id and the daemon build in service.version. Ten gauges arrive:

Each carries daytona_organization_id, daytona_region_id, and daytona_snapshot as resource attributes. Note the underscores: Daytona's own telemetry uses snake_case, while attributes you set in your own code follow whatever convention you choose.

Toolbox API spans

The daemon traces the API your SDK calls into, as server spans named for the route: POST /process/execute when you run a command, POST /files/bulk-upload when you upload a file. They carry http.route, url.path, and client.address, and give you the sandbox-side duration of work your SDK spans only see from the outside.

Organization quota metrics

Pushed every 60 seconds, covering consumption against your plan limits across every sandbox. The Daytona dashboard template charts all of these already:

Each carries organization.id as a resource attribute, plus region.id and sandbox.class as data point attributes. sandbox.class separates container from windows sandboxes, which Daytona meters against the same quota, so a headline usage number without it hides the split.

• SDK spans carry no daytona.* attributes. They carry no sandbox id, organization id, or region. The sandbox id appears only inside http_url, for example https://proxy.app-eu.daytona.io/toolbox/<sandbox-id>/process/code-run, so grouping SDK spans by sandbox means parsing that URL. Per-sandbox attribution comes from the telemetry you emit inside the sandbox.
• OTEL_SERVICE_NAME does not apply to SDK tracing. The SDK sets service.name to daytona-python-sdk and service.version to the SDK version, no matter what you set. Every application using the SDK reports under that one name, so you cannot split SDK traffic per app through the service name.
• Span names include the client class. The async client emits AsyncDaytona.create and AsyncSandbox.stop. The synchronous client drops the Async prefix from every span name. A filter written for one client returns nothing for the other.
• daytona.snapshot is a registry digest. It resolves to a value such as cr.app.daytona.io/sbox/daytona-<sha>:daytona, not the friendly daytonaio/sandbox:0.8.0 name you passed at creation.

The async client emits internal spans for AsyncDaytona.create, get, list, list.fetch_page, start, stop, and delete; AsyncSandbox.start, stop, delete, refresh_data, wait_for_sandbox_start, and wait_for_sandbox_stop; AsyncProcess.code_run; and AsyncFileSystem.upload_file and upload_files. Outbound HTTP calls appear as client spans named for the method alone: GET, POST, DELETE.

Sandboxes expose DAYTONA_SANDBOX_ID, DAYTONA_ORGANIZATION_ID, DAYTONA_REGION_ID, and DAYTONA_SANDBOX_SNAPSHOT to every process. Use them for resource attributes, as the sandbox script above does.

Connection reset when exporting from inside a sandbox

Symptom: the exporter retries and gives up, with Connection reset by peer or OpenSSL SSL_connect: Connection reset by peer.

Likely cause: sandbox egress is deny-by-default and the SigNoz ingestion host is not allowed. Package registries work, which makes the block look selective.

Fix: create the sandbox with domain_allow_list including ingest.<region>.signoz.cloud, as in Step 1. Tier 1 and Tier 2 accounts cannot set this, so upgrade the plan or export through a host that is already reachable.

Verify: from inside the sandbox, curl -s -o /dev/null -w '%{http_code}' https://ingest.<region>.signoz.cloud returns 404. That means TLS completed and egress is open. 000 means the connection is still blocked.

No SDK spans arrive

Symptom: sandbox operations succeed, but no daytona-python-sdk service appears in SigNoz.

Likely cause: tracing is off, or the process exited before spans flushed.

Fix: confirm otel_enabled=True or DAYTONA_OTEL_ENABLED=true, and close the client. Use async with AsyncDaytona(...) or call await daytona.close().

Verify: service.name = 'daytona-python-sdk' returns spans in the Traces explorer.

Sandbox CPU, memory, and filesystem metrics never arrive

Symptom: organization gauges arrive and your own spans arrive, but nothing under service.name = 'sandbox-<sandbox-id>'. Nothing reports an error.

Likely cause: the sandbox cannot reach otel-collector.app.daytona.io. A domain_allow_list that omits *.daytona.io blocks the daemon's exporter while leaving your own exporter working, so the failure is invisible from inside your code.

Fix: add *.daytona.io to domain_allow_list and create a new sandbox. Existing sandboxes keep the rules they were created with.

Verify: from inside the sandbox, curl -s -o /dev/null -w '%{http_code}' https://otel-collector.app.daytona.io returns 404. 000 means the daemon is still firewalled.

Organization metrics are missing

Symptom: no daytona.sandbox.used_cpu after saving the configuration.

Likely cause: the push interval has not elapsed, or the header is wrong.

Fix: Daytona takes up to five minutes to apply an endpoint change, then pushes on a 60-second interval. Wait out both with at least one sandbox running, and confirm the header key is exactly signoz-ingestion-key.

Verify: search for daytona.sandbox in the Metrics explorer.

• Sandbox stdout and stderr are not exported. Daytona documents application logs as part of sandbox telemetry, but the only log records that arrive are the daemon's own startup diagnostics, four per sandbox. Anything your process prints stays in the sandbox, so route it through the OpenTelemetry logs SDK as Step 2 does.
• otelEndpointOverride on sandbox creation has no effect. The API accepts the field and returns 200, then reports the sandbox with otelEndpointOverride: null.
• The documented otel-config API call does not work with an API key. PUT /api/organizations/<org>/otel-config returns 403 Invalid authentication context for a dtn_ key regardless of its permissions. Use the Dashboard.
• A long-lived WebSocket span distorts SDK latency. The SDK opens a connection to wss://app.daytona.io/api/socket.io/ that lives as long as the client. It appears as a root span lasting the whole session and dominates p99 for the daytona-python-sdk service. Exclude it when measuring operation latency.
• SDK operations are separate traces. Each call roots its own trace rather than nesting under a session, so there is no single trace covering create through delete. Correlate with your own parent span if you need that view.