Monitor AWS Lambda MicroVMs with OpenTelemetry Collector

Overview

AWS Lambda MicroVMs are Firecracker-based compute environments with VM-level isolation, snapshot start and resume, and full OS capabilities. They suit long-running, stateful sandboxes such as AI agent code execution, interactive development environments, CI jobs, and vulnerability scanners.

This guide runs an OpenTelemetry Collector inside the MicroVM image. The Collector scrapes host metrics, tails application logs, receives OTLP from your application, and exports everything to SigNoz.

Despite the name, a MicroVM behaves like a long-lived VM, closer to Fargate or EC2 than to a short-lived function. Do not use the per-invocation OpenTelemetry Lambda layers here. Run a Collector inside the image and instrument your application the way you would a normal containerized service. For classic function monitoring, see Monitor AWS Lambda.

Most steps are identical. Update the endpoint and remove the ingestion key header as shown in Cloud to Self-Hosted.

Prerequisites

The AWS CLI with the lambda-microvms command set. Run aws lambda-microvms help to confirm it lists create-microvm-image and run-microvm.
An S3 bucket in the same region for the code artifact.
A build role that Lambda assumes during the image build. It needs s3:GetObject on the artifact and logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents.
An execution role for the running MicroVM. Without it, application stdout and stderr never reach CloudWatch. It needs logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents.
Both roles need a trust policy that lets Lambda assume them, with Principal: {"Service": "lambda.amazonaws.com"} and Action: ["sts:AssumeRole", "sts:TagSession"]. Your own caller needs lambda:CreateMicrovmImage, lambda:RunMicrovm, and iam:PassRole for both roles.
An instance of SigNoz (either Cloud or Self-Hosted)

How monitoring works

Signal	Source inside the MicroVM	Path to SigNoz
Metrics (host CPU, memory, disk, network)	`hostmetrics` receiver in the embedded Collector	OTLP/HTTP to SigNoz
Traces	Application instrumented with an OpenTelemetry SDK	OTLP to the local Collector, then to SigNoz
Logs	`filelog` receiver, or OTLP from the SDK	OTLP/HTTP to SigNoz
Lifecycle and audit	CloudTrail data events	CloudWatch or S3 to SigNoz

How sizing works

MicroVMs use a baseline and peak model. You set the baseline with the memory value when you create the image, and vCPU scales with it at 2 GB per vCPU. During activity the MicroVM scales vertically up to four times the baseline.

Baseline	Peak	Max disk
0.5 GB, 0.25 vCPU	2 GB, 1 vCPU	8 GB
1 GB, 0.5 vCPU	4 GB, 2 vCPU	8 GB
2 GB, 1 vCPU (default)	8 GB, 4 vCPU	8 GB
4 GB, 2 vCPU	16 GB, 8 vCPU	16 GB
8 GB, 4 vCPU	32 GB, 16 vCPU	32 GB

A MicroVM runs for at most 8 hours. Set the limit with --maximum-duration-in-seconds, which accepts 1 to 28,800 seconds.

Send telemetry to SigNoz

Lambda builds the MicroVM image from a ZIP you upload to S3, then snapshots the running result. The Collector is baked into that snapshot, so it is already running the moment a MicroVM starts. Your application exports to it on localhost instead of reaching SigNoz directly.

Step 1: Package the application and Dockerfile

create-microvm-image accepts architecture=ARM_64 and no other value. Every binary you install in the image must be an ARM64 build, including the OpenTelemetry Collector.

The code artifact is a ZIP that contains a Dockerfile at the archive root plus your application files. Lambda pulls the ZIP from S3, runs your Dockerfile on top of the managed base image, starts your application, and snapshots the result.

Two different base images are involved. The MicroVM base image is the operating system environment, passed as --base-image-arn. The container base image is what your Dockerfile uses in its FROM instruction.

FROM public.ecr.aws/lambda/microvms:al2023-minimal
 
# al2023-minimal ships without tar, gzip, or procps. Install what you need
# before extracting anything.
RUN dnf install -y tar gzip python3 python3-pip && dnf clean all
 
# Release assets are version-named, and MicroVMs are ARM64 only. Pin a version
# and use the arm64 tarball.
ARG OTELCOL_VERSION=0.159.0
ADD https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_VERSION}/otelcol-contrib_${OTELCOL_VERSION}_linux_arm64.tar.gz /tmp/otelcol.tar.gz
RUN mkdir -p /usr/local/bin \
 && tar -xzf /tmp/otelcol.tar.gz -C /usr/local/bin otelcol-contrib \
 && rm /tmp/otelcol.tar.gz
 
COPY app.py                     /app/app.py
COPY otel-collector-config.yaml /etc/otelcol/config.yaml
COPY entrypoint.sh              /entrypoint.sh
RUN chmod +x /entrypoint.sh && mkdir -p /var/log/app
 
CMD ["/entrypoint.sh"]

The image build has outbound internet access, so pulling the Collector with ADD works. Start the Collector in the background and your application in the foreground, which keeps the MicroVM alive:

#!/usr/bin/env bash
set -euo pipefail
 
# Lambda forwards only stdout and stderr to CloudWatch, so do not redirect the
# Collector to a file. Writing it into /var/log/app would also make the filelog
# receiver tail the Collector's own output.
/usr/local/bin/otelcol-contrib --config /etc/otelcol/config.yaml 2>&1 &
 
exec python3 -u /app/app.py

Step 2: Configure the OpenTelemetry Collector

receivers:
  # On Collector v0.151.0 and newer, use "host_metrics" to avoid a deprecation warning.
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          # system.cpu.utilization is an optional metric and off by default.
          # Enable it explicitly or it never reaches SigNoz.
          system.cpu.utilization:
            enabled: true
      memory: {}
      load: {}
      filesystem: {}
      network: {}
 
  # On Collector v0.149.0 and newer, use "file_log" to avoid a deprecation warning.
  # This only collects anything if your application writes to this path. An
  # application that logs to stdout alone produces no records here.
  #
  # Keep the Collector's own log out of this glob. A Collector that tails its own
  # output re-exports every line it writes.
  filelog:
    include: [/var/log/app/*.log]
    exclude: [/var/log/app/otelcol*.log]
    start_at: beginning
 
  otlp:
    protocols:
      http:
        endpoint: localhost:4318
 
processors:
  batch: {}
  # On Collector v0.153.0 and newer, use "resource_detection" to avoid a deprecation warning.
  resourcedetection:
    detectors: [env, system]
  resource:
    attributes:
      # Lambda injects this into every MicroVM. It is the same value the
      # CloudWatch Agent uses for its ImageName dimension.
      - key: aws.lambda.microvm.image_name
        value: ${env:AWS_LAMBDA_MICROVM_IMAGE_NAME}
        action: upsert
      - key: aws.lambda.microvm.image_version
        value: ${env:AWS_LAMBDA_MICROVM_IMAGE_VERSION}
        action: upsert
 
exporters:
  # On Collector v0.144.0 and newer, use "otlp_http" to avoid a deprecation warning.
  otlphttp:
    endpoint: ${env:SIGNOZ_ENDPOINT}
    headers:
      signoz-ingestion-key: ${env:SIGNOZ_INGESTION_KEY}
 
service:
  pipelines:
    metrics:
      receivers: [hostmetrics, otlp]
      processors: [resourcedetection, resource, batch]
      exporters: [otlphttp]
    traces:
      receivers: [otlp]
      processors: [resourcedetection, resource, batch]
      exporters: [otlphttp]
    logs:
      receivers: [filelog, otlp]
      processors: [resourcedetection, resource, batch]
      exporters: [otlphttp]

The Collector starts during the image build, so resourcedetection resolves host.name and host.id once, inside the build MicroVM, and the snapshot freezes those values. Every MicroVM restored from that image reports the same host. The image name and version are constant per image too, so nothing in this config distinguishes one running sandbox from another.

To tell them apart, take the microvmId that Lambda passes to the /run lifecycle hook and set it as a resource attribute from your application. See Running and using MicroVMs.

Lambda injects AWS_LAMBDA_MICROVM_IMAGE_NAME, AWS_LAMBDA_MICROVM_IMAGE_ARN, AWS_LAMBDA_MICROVM_IMAGE_VERSION, and AWS_REGION into every MicroVM. Using the image name as a resource attribute lets you filter and group all three signals by image in SigNoz.

Step 3: Instrument your application

Because the MicroVM is long-lived, use standard service-style instrumentation rather than a Lambda layer. Point the SDK at the embedded Collector with these variables:

OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
OTEL_SERVICE_NAME="my-microvm-app"
OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production"

There is no environment variable flag on run-microvm, so pass these to --environment-variables when you build the image in Step 4. Setting them anywhere else has no effect.

See the SigNoz instrumentation guides for your language.

Write your application logs to /var/log/app/ for the filelog receiver from Step 2 to pick them up, or remove that receiver and export logs over OTLP from the SDK.

Your directory now holds everything the build needs. Zip it with the Dockerfile at the archive root and upload it:

zip -r app.zip Dockerfile entrypoint.sh otel-collector-config.yaml app.py
aws s3 cp app.zip s3://<your-bucket>/app.zip

Verify these values:

<your-bucket>: The S3 bucket you created for the code artifact.

Step 4: Build the MicroVM image

Export your SigNoz destination so the build command can read it:

export SIGNOZ_ENDPOINT="https://ingest.<region>.signoz.cloud:443"
export SIGNOZ_INGESTION_KEY="<your-ingestion-key>"

Then create the image:

aws lambda-microvms create-microvm-image \
  --name my-monitored-app \
  --code-artifact uri=s3://<your-bucket>/app.zip \
  --base-image-arn arn:aws:lambda:<aws-region>:aws:microvm-image:al2023-1 \
  --build-role-arn arn:aws:iam::<account-id>:role/MicrovmBuildRole \
  --cpu-configurations '[{"architecture":"ARM_64"}]' \
  --resources '[{"minimumMemoryInMiB":2048}]' \
  --environment-variables "SIGNOZ_ENDPOINT=$SIGNOZ_ENDPOINT,SIGNOZ_INGESTION_KEY=$SIGNOZ_INGESTION_KEY,OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318,OTEL_SERVICE_NAME=my-microvm-app" \
  --logging '{"cloudWatch":{"logGroup":"/aws/lambda-microvms/my-monitored-app"}}'

Verify these values:

<region>: Your SigNoz Cloud region.
<your-ingestion-key>: Your SigNoz ingestion key.
<your-bucket>: The S3 bucket holding your code artifact.
<aws-region>: The AWS region you are deploying to, for example us-east-1. Discover available base images with aws lambda-microvms list-managed-microvm-images.
<account-id>: Your AWS account ID.

The build is asynchronous. Poll until the image reports CREATED:

aws lambda-microvms get-microvm-image \
  --image-identifier arn:aws:lambda:<aws-region>:<account-id>:microvm-image:my-monitored-app

There is no environment variable flag on run-microvm. Variables are image-level and shared by every MicroVM run from that image. create-microvm-image, update-microvm-image, and get-microvm-image-version all return them in plaintext, so anyone with read access to the image version can read your ingestion key.

To keep a secret out of the image, pass a Secrets Manager path through --run-hook-payload. Fetch the value in your /run lifecycle hook.

update-microvm-image does not inherit environment variables from the original create call. Pass --environment-variables again on every update, or the new version loses them.

Step 5: Run the MicroVM

aws lambda-microvms run-microvm \
  --image-identifier arn:aws:lambda:<aws-region>:<account-id>:microvm-image:my-monitored-app \
  --execution-role-arn arn:aws:iam::<account-id>:role/MicrovmExecutionRole \
  --ingress-network-connectors "arn:aws:lambda:<aws-region>:aws:network-connector:aws-network-connector:ALL_INGRESS" \
  --idle-policy '{"maxIdleDurationSeconds":900,"suspendedDurationSeconds":600,"autoResumeEnabled":true}' \
  --maximum-duration-in-seconds 1800

Lambda measures idleness by traffic arriving at the MicroVM endpoint. The Collector's outbound export does not count, so a MicroVM that nobody calls suspends after maxIdleDurationSeconds and stops producing host metrics. Drop --idle-policy for a workload that does not serve inbound requests, or raise the duration to match how long you need continuous metrics.

MicroVMs have public internet access on the egress path by default, so the Collector reaches SigNoz without extra configuration. Attach a customer-managed egress connector when you want outbound traffic to route through your VPC instead:

--egress-network-connectors "<your-vpc-connector-arn>"

Verify these values:

<your-vpc-connector-arn>: The ARN of a Lambda Network Connector in the ACTIVE state.

Create that connector with aws lambda-core create-network-connector and wait for it to reach ACTIVE before you reference it. See Networking for Lambda MicroVMs.

run-microvm returns a microvmId and a dedicated HTTPS endpoint for that MicroVM. Every request to the endpoint needs a token in the X-aws-proxy-auth header, and there is no unauthenticated access:

aws lambda-microvms create-microvm-auth-token \
  --microvm-identifier <microvm-id> \
  --expiration-in-minutes 30 \
  --allowed-ports '[{"port":8080}]'

Verify these values:

<microvm-id>: The microvmId returned by run-microvm.

Validate

Give the Collector a minute, then open SigNoz:

Traces: your OTEL_SERVICE_NAME appears as a service with spans.
Metrics Explorer: system.memory.usage and system.cpu.utilization are present and filterable by aws.lambda.microvm.image_name.
Logs Explorer: your application log lines appear with the same resource attribute.

SigNoz Traces Explorer filtered on service.name my-microvm-app, listing handle-request and run-user-code spans from a Lambda MicroVM — *Traces Explorer filtered on the service name you set in Step 3*

SigNoz trace detail view showing a MicroVM span with aws.lambda.microvm.image_name and aws.lambda.microvm.image_version resource attributes — *Span details for a MicroVM trace, showing the MicroVM resource attributes*

To confirm the Collector itself started, check the runtime log group /aws/lambda-microvms/<image-name> for the line Everything is ready. Begin running and processing data.

Troubleshooting

Image build fails with `tar: command not found`

Symptom: The build stops with exit code 127 and /bin/sh: line 1: tar: command not found.

Likely cause: al2023-minimal ships without tar and gzip.
Fix: Add RUN dnf install -y tar gzip before any step that extracts an archive. The same applies to pgrep and other procps tools.
Verify: The build reaches CREATED.

Image build fails for another reason

Symptom: The image reports CREATE_FAILED, or latestFailedImageVersion is set.

Likely cause: A Dockerfile instruction failed.
Fix: Read the build output in the CloudWatch log group you passed to --logging, which defaults to /aws/lambda-microvms/<image-name>.
Verify: The failing instruction appears with its exit code.

A successful build does not mean your application is healthy. Lambda snapshots the image even when the process inside it crashed on startup, so an image can report CREATED while the Collector is failing. Always confirm signals in SigNoz.

The endpoint returns HTTP 403

Symptom: Requests to the MicroVM endpoint return 403 Forbidden.

Likely cause: The token is missing, expired, or invalid, or the target port is not in the token's allowedPorts.
Fix: Mint a new token. Requests route to port 8080 unless you send an X-aws-proxy-port header, and whichever port you target must appear in allowedPorts.
Verify: The endpoint returns your application's response.

The endpoint returns HTTP 502

Symptom: Requests to the MicroVM endpoint return 502 Bad Gateway with an empty body.

Likely cause: Your application is not listening on the target port, it crashed handling the request, or auto-resume did not finish within the retry limit.
Fix: Check the runtime log group for a stack trace. Confirm your application listens on port 8080, or send X-aws-proxy-port for a different one.
Verify: The endpoint returns your application's response.

No `system.cpu.utilization` in SigNoz

Symptom: system.memory.usage appears but system.cpu.utilization returns no data.

Likely cause: system.cpu.utilization is an optional hostmetrics metric, disabled by default.
Fix: Enable it explicitly under the cpu scraper as shown in Step 2.
Verify: The metric appears in Metrics Explorer.

No logs in SigNoz

Symptom: Traces and metrics arrive but the Logs Explorer is empty.

Likely cause: The filelog receiver watches /var/log/app/*.log, and your application writes to stdout only.
Fix: Write logs to a file under /var/log/app/, or drop filelog and export logs over OTLP from the SDK.
Verify: Records appear in Logs Explorer.

The Collector logs `Configuration references unset environment variable`

Symptom: The Collector cannot resolve ${env:SIGNOZ_ENDPOINT} or ${env:SIGNOZ_INGESTION_KEY}.

Likely cause: An update-microvm-image call omitted --environment-variables, which drops them from the new version.
Fix: Pass --environment-variables on every update.
Verify: The Collector starts and exports without warnings.

The API returns HTTP 502 intermittently

Symptom: run-microvm, get-microvm, or list-microvm-images fails with Bad Gateway.

Likely cause: A transient service error. AWS does not document 502 for these APIs, but it occurs in practice.
Fix: Retry with exponential backoff. Scripts that poll these APIs need retry handling. AWS documents ThrottlingException and InternalServerException as retryable, and ResourceNotFoundException when the image is not yet CREATED.
Verify: The call succeeds on a later attempt.

Limitations

The CloudWatch Agent cannot forward to SigNoz. AWS's recommended metrics agent for MicroVMs runs on Telegraf and the OpenTelemetry Collector. Its Collector build has no otlp exporter. Its only OTLP exporter, otlphttp, rejects non-AWS endpoints with invalid AWS endpoint. The agent exits on either config, which also stops the CloudWatch metrics you already had. Run a separate OpenTelemetry Collector as described above. To get CloudWatch data into SigNoz, pull the /aws/lambda-microvms/<image-name> log group and the LambdaMicroVms/Application namespace through the AWS monitoring integration.
Host metrics reflect guest-visible resources. hostmetrics reads what the guest kernel reports. Those totals differ from the memory and vCPU baseline you configured, so read them as relative signals rather than capacity numbers.
Metrics stop while a MicroVM is suspended. The embedded Collector survives suspend and resume without restarting. No host metrics are produced during the suspended window, so expect gaps in dashboards.

Optional: CloudTrail lifecycle and audit events

MicroVM lifecycle actions are CloudTrail data events, and CloudTrail does not log them by default. Enable them with an advanced event selector on the AWS::Lambda::MicrovmImage resource type.

put-event-selectors overwrites every selector already on the trail. Advanced event selectors are opt-in, so a call that names only data events stops the trail logging management events. Read the current selectors with aws cloudtrail get-event-selectors --trail-name <your-trail> and merge them, or use a trail dedicated to MicroVM auditing.

The selectors below keep management events and add MicroVM data events, so they are safe to apply to a trail that was logging management events by default:

aws cloudtrail put-event-selectors \
  --trail-name <your-trail> \
  --advanced-event-selectors '[
    {
      "Name": "Keep management events",
      "FieldSelectors": [
        { "Field": "eventCategory", "Equals": ["Management"] }
      ]
    },
    {
      "Name": "Log Lambda MicroVM data events",
      "FieldSelectors": [
        { "Field": "eventCategory", "Equals": ["Data"] },
        { "Field": "resources.type", "Equals": ["AWS::Lambda::MicrovmImage"] }
      ]
    }
  ]'

Verify these values:

<your-trail>: The name of an existing CloudTrail trail in the same region.

The second selector captures RunMicrovm, TerminateMicrovm, SuspendMicrovm, ResumeMicrovm, CreateMicrovmAuthToken, and CreateMicrovmShellAuthToken. Delivery to S3 takes a few minutes. Forward the stream into SigNoz using the existing AWS logs ingestion patterns to build suspend and resume timelines or security audit dashboards.

Image and MicroVM management calls such as CreateMicrovmImage and ListMicrovms are management events. CloudTrail logs them by default, and the first selector above preserves that. See Monitoring for Lambda MicroVMs for the full event list.

Next steps

Set up alerts on host metrics and error rates.
Build dashboards grouped by aws.lambda.microvm.image_name.
Correlate traces and logs to move between signals during an investigation.
Monitor AWS Lambda functions for classic, per-invocation Lambda.

Get Help

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack. If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.

Monitor AWS Lambda MicroVMs with OpenTelemetry Collector

Overview

Prerequisites

How monitoring works

How sizing works

Send telemetry to SigNoz

Step 1: Package the application and Dockerfile

Step 2: Configure the OpenTelemetry Collector

Step 3: Instrument your application

Step 4: Build the MicroVM image

Step 5: Run the MicroVM

Validate

Troubleshooting

Image build fails with tar: command not found

Image build fails for another reason

The endpoint returns HTTP 403

The endpoint returns HTTP 502

No system.cpu.utilization in SigNoz

No logs in SigNoz

The Collector logs Configuration references unset environment variable

The API returns HTTP 502 intermittently

Limitations

Optional: CloudTrail lifecycle and audit events

Next steps

Get Help

Image build fails with `tar: command not found`

No `system.cpu.utilization` in SigNoz

The Collector logs `Configuration references unset environment variable`