Vercel Sandbox Observability & Monitoring with OpenTelemetry

Overview

Vercel Sandbox runs untrusted or agent-generated code in an isolated Linux microVM. Your application creates a sandbox, runs commands in it, reads the output, and stops it. A sandbox is ephemeral, so the command that failed is gone before you can open a terminal.

Vercel Drains do not close that gap. The log drain sources are static, lambda, edge, build, external, firewall, and redirect, and trace drains carry deployment traffic. Nothing that happens in a sandbox reaches SigNoz until you report it.

Prerequisites

• An instance of SigNoz (either Cloud or Self-Hosted)
• Node.js 22 or later
• A Vercel project linked to your working directory. Run vercel link, then vercel env pull to write credentials to .env.local. See the Vercel Sandbox quickstart.

How it works

Vercel Sandbox emits no telemetry of its own. Two paths produce it, and they answer different questions.

Start with the first path. It covers every sandbox your application creates, including the ones whose code you never wrote. The second path is collapsed below. Both use OpenTelemetry and export to the same SigNoz endpoint.

The Vercel dashboard reports Active CPU, provisioned memory, data transfer, and running sandboxes under its own Observability tab. Those numbers are pull-only, and none of them name the command that ran or how it exited.

Monitor Vercel Sandbox from Your Application

This path shows what each sandbox did: which commands ran, how long each took, what they printed, and where they failed.

Step 1: Install the packages

npm install @vercel/sandbox @opentelemetry/api @opentelemetry/api-logs \
  @opentelemetry/sdk-node @opentelemetry/sdk-logs \
  @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-logs-otlp-http
 
npm install --save-dev tsx typescript @types/node

Step 2: Configure the OpenTelemetry SDK

Append the SigNoz settings to .env.local, the file that vercel env pull created.

OTEL_SERVICE_NAME=<your-service-name>
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.<region>.signoz.cloud:443
OTEL_EXPORTER_OTLP_HEADERS=signoz-ingestion-key=<your-ingestion-key>

Verify these values:

• <your-service-name>: What your sandbox telemetry appears under in SigNoz, and what you filter on in Validate.
• <region>: Your SigNoz Cloud region.
• <your-ingestion-key>: Your SigNoz ingestion key.

Create instrumentation.ts. The exporters read the endpoint and the headers from the environment, so they take no arguments. Each one appends its own path, /v1/traces and /v1/logs, to the endpoint.

import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http'
import { BatchLogRecordProcessor } from '@opentelemetry/sdk-logs'
 
export const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
  logRecordProcessors: [
    new BatchLogRecordProcessor({ exporter: new OTLPLogExporter() }),
  ],
})
 
sdk.start()

BatchLogRecordProcessor takes an options object in @opentelemetry/sdk-logs 0.222.0 and later. Passing the exporter as a positional argument fails at shutdown with TypeError: Cannot read properties of undefined (reading 'shutdown').

Step 3: Add the traced sandbox helpers

Create sandbox-otel.ts. It contains two functions:

• withSandbox opens one span for the sandbox session. It records the configuration when the sandbox starts, and the billed usage when the sandbox stops.
• runTracedCommand opens a child span for one command. It streams the output of the command to SigNoz as logs, and records the exit code.

import { Sandbox } from '@vercel/sandbox'
import { SpanStatusCode, trace } from '@opentelemetry/api'
import { logs, SeverityNumber } from '@opentelemetry/api-logs'
 
const tracer = trace.getTracer('vercel-sandbox')
const logger = logs.getLogger('vercel-sandbox')
 
type CreateOptions = Parameters<typeof Sandbox.create>[0]
 
export async function withSandbox<T>(
  options: CreateOptions,
  fn: (sandbox: Sandbox) => Promise<T>
): Promise<T> {
  return tracer.startActiveSpan('sandbox session', async (span) => {
    let sandbox: Sandbox | undefined
    try {
      sandbox = await Sandbox.create(options)
      span.setAttributes({
        'vercel.sandbox.name': sandbox.name,
        'vercel.sandbox.session_id': sandbox.currentSession().sessionId,
        'vercel.sandbox.region': sandbox.region,
        'vercel.sandbox.vcpus': sandbox.vcpus,
        'vercel.sandbox.memory_mb': sandbox.memory,
        'vercel.sandbox.persistent': sandbox.persistent,
      })
      return await fn(sandbox)
    } catch (error) {
      span.recordException(error as Error)
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(error) })
      throw error
    } finally {
      try {
        if (sandbox) {
          const session = await sandbox.stop()
          span.setAttributes({
            'vercel.sandbox.active_cpu_ms': session.activeCpuDurationMs,
            'vercel.sandbox.egress_bytes': session.networkTransfer?.egress,
            'vercel.sandbox.ingress_bytes': session.networkTransfer?.ingress,
            'vercel.sandbox.snapshot_id': session.snapshot?.id,
          })
        }
      } catch (stopError) {
        span.recordException(stopError as Error)
      } finally {
        span.end()
      }
    }
  })
}
 
export async function runTracedCommand(
  sandbox: Sandbox,
  cmd: string,
  args: string[] = []
) {
  return tracer.startActiveSpan(`sandbox exec ${cmd}`, async (span) => {
    span.setAttributes({
      'vercel.sandbox.name': sandbox.name,
      'vercel.sandbox.command': [cmd, ...args].join(' '),
    })
 
    try {
      const command = await sandbox.runCommand({ cmd, args, detached: true })
      span.setAttribute('vercel.sandbox.command_id', command.cmdId)
 
      for await (const line of command.logs()) {
        logger.emit({
          severityNumber:
            line.stream === 'stderr' ? SeverityNumber.WARN : SeverityNumber.INFO,
          severityText: line.stream === 'stderr' ? 'WARN' : 'INFO',
          body: line.data.trimEnd(),
          attributes: {
            'vercel.sandbox.name': sandbox.name,
            'vercel.sandbox.command_id': command.cmdId,
            'vercel.sandbox.stream': line.stream,
          },
        })
      }
 
      const result = await command.wait()
      span.setAttributes({
        'vercel.sandbox.exit_code': result.exitCode,
        'vercel.sandbox.command_duration_ms': result.durationMs,
      })
      if (result.exitCode !== 0) {
        span.setStatus({ code: SpanStatusCode.ERROR })
        span.setAttribute('error.type', `exit_code_${result.exitCode}`)
      }
      return result
    } catch (error) {
      span.recordException(error as Error)
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(error) })
      throw error
    } finally {
      span.end()
    }
  })
}

Two details make the correlation work:

• runTracedCommand starts the command with detached: true. The command runs while command.logs() streams its output line by line. command.wait() then returns the exit code.
• logger.emit() reads the active OpenTelemetry context, so each log record carries the trace ID and the span ID of the command span. SigNoz links the log to the span without extra configuration.

span.end() sits in its own nested finally. OpenTelemetry hands a span to the exporter when the span ends, so a span that never ends is never exported, not even by sdk.shutdown(). Without the nested block, a sandbox.stop() that rejects would skip span.end() and lose the whole session span, including any exception recorded on it. Catching the stop error also keeps it from replacing the error your workload threw.

Step 4: Run your orchestrator

Create index.ts. Import instrumentation.js first so the SDK starts before the Vercel SDK loads.

import { sdk } from './instrumentation.js'
import { runTracedCommand, withSandbox } from './sandbox-otel.js'
 
async function main() {
  await withSandbox({ persistent: false, timeout: 120_000 }, async (sandbox) => {
    await runTracedCommand(sandbox, 'node', ['--version'])
    await runTracedCommand(sandbox, 'python3', ['-c', 'print(sum(range(100)))'])
    await runTracedCommand(sandbox, 'ls', ['/does-not-exist'])
  })
}
 
main()
  .catch((error) => {
    console.error(error)
    process.exitCode = 1
  })
  .finally(async () => {
    await sdk.shutdown()
  })

Run the script with the credentials from .env.local:

npx tsx --env-file=.env.local index.ts

Load the environment with --env-file and not with a dotenv call in index.ts. JavaScript evaluates imports before the body of the module, so the exporters read the environment before a dotenv call can set it.

await sdk.shutdown() flushes the last batch. Without it, a short script exits before the exporter sends the final spans and logs.

Validate

Open Traces in SigNoz and filter on service.name = '<your-service-name>'. One run produces one trace:

• A root span named sandbox session, carrying the sandbox name, region, vCPUs, and the billed CPU and transfer.
• A child span for each command, named sandbox exec node, sandbox exec python3, and sandbox exec ls.
• The sandbox exec ls span with an error status and vercel.sandbox.exit_code = 2.

Open the sandbox session span. Its attributes carry the sandbox configuration and, because the sandbox has stopped, the billed usage.

Trace the Code Inside a Sandbox

The steps above trace the sandbox from outside. To see spans from the generated code itself, run an OpenTelemetry SDK inside the sandbox. Pass the trace context in through an environment variable.

OpenTelemetry defines TRACEPARENT as the carrier for context between processes. See Environment Variables as Context Propagation Carriers. The JavaScript SDK does not read the variable for you, so the workload extracts it.

Step 1: Write the workload script

child.cjs

const { context, propagation, trace } = require('@opentelemetry/api')
const { NodeSDK } = require('@opentelemetry/sdk-node')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http')
 
const sdk = new NodeSDK({ traceExporter: new OTLPTraceExporter() })
sdk.start()
 
async function main() {
  const parent = propagation.extract(context.active(), {
    traceparent: process.env.TRACEPARENT,
  })
 
  const tracer = trace.getTracer('workload')
  context.with(parent, () => {
    const span = tracer.startSpan('generated code')
    span.setAttribute('workload.rows', 42)
    span.end()
  })
}
 
main()
  .catch((error) => {
    console.error(error)
    process.exitCode = 1
  })
  .finally(async () => {
    await sdk.shutdown()
  })

The workload takes the same three environment variables as the orchestrator. The orchestrator passes them in through runCommand.

The flush rule from Step 4 applies here too. A .cjs file cannot use top-level await, so the shutdown goes in the finally of a main() chain. Calling sdk.shutdown() on its own leaves the promise unhandled, which turns an export failure into an unhandled rejection instead of a logged error.

Use NodeSDK here and not a bare NodeTracerProvider. Only NodeSDK runs the resource detectors that read OTEL_SERVICE_NAME. A bare provider reports every span under service.name = unknown_service:node.

Step 2: Run it from the orchestrator

Install the dependencies while egress is open, restrict egress to SigNoz, then run the workload.

in-sandbox.ts

import { sdk } from './instrumentation.js'
import { context, propagation } from '@opentelemetry/api'
import { readFileSync } from 'node:fs'
import { runTracedCommand, withSandbox } from './sandbox-otel.js'
 
const INGEST_HOST = new URL(process.env.OTEL_EXPORTER_OTLP_ENDPOINT!).hostname
 
async function main() {
  await withSandbox({ persistent: false, timeout: 300_000 }, async (sandbox) => {
    await sandbox.writeFiles([
      { path: 'child.cjs', content: readFileSync('child.cjs') },
    ])
 
    await runTracedCommand(sandbox, 'npm', [
      'install',
      '--no-package-lock',
      '@opentelemetry/api',
      '@opentelemetry/sdk-node',
      '@opentelemetry/exporter-trace-otlp-http',
    ])
 
    await sandbox.update({ networkPolicy: { allow: [INGEST_HOST] } })
 
    const carrier: Record<string, string> = {}
    propagation.inject(context.active(), carrier)
 
    await sandbox.runCommand({
      cmd: 'node',
      args: ['child.cjs'],
      env: {
        TRACEPARENT: carrier.traceparent,
        OTEL_SERVICE_NAME: '<your-service-name>-workload',
        OTEL_EXPORTER_OTLP_ENDPOINT: process.env.OTEL_EXPORTER_OTLP_ENDPOINT!,
        OTEL_EXPORTER_OTLP_HEADERS: process.env.OTEL_EXPORTER_OTLP_HEADERS!,
      },
    })
  })
}
 
main()
  .catch((error) => {
    console.error(error)
    process.exitCode = 1
  })
  .finally(async () => {
    await sdk.shutdown()
  })

The generated code span now appears in the same trace as sandbox session, under the service <your-service-name>-workload.

The egress firewall blocks the exporter

A sandbox network policy that does not list your ingestion host drops the telemetry. Add ingest.<region>.signoz.cloud to the allow list of every policy that runs instrumented code.

span.setAttributes({
  'gen_ai.operation.name': 'execute_tool',
  'gen_ai.tool.name': 'vercel_sandbox',
  'gen_ai.tool.call.id': toolCall.id,
})

Next Steps

• Set a trace-based alert on sandbox exec spans with a non-zero vercel.sandbox.exit_code, so a failing agent command pages you.
• Build a dashboard over vercel.sandbox.active_cpu_ms and vercel.sandbox.egress_bytes to attribute sandbox cost per sandbox name.
• Explore the traces to find which command is slowest across sandboxes.
• Send the telemetry of your Vercel deployments with the SigNoz integration for Vercel.
• Instrument the model calls that generate the code with Vercel AI SDK Observability, or browse all LLM observability integrations.

Get Help

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack. If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.