ECS EC2 - Configure

The install guide provides a working config: one collector task on every container instance collects host metrics, container metrics, container and host logs, and data from instrumented applications. This page explains what each component in that config does and how to customize it through Foundry's casting.yaml.

Prerequisites

• ECS EC2 Collection Agent installed with Foundry and Terraform (see the install guide)
• SigNoz Cloud account or self-hosted SigNoz instance
• The AWS CLI with access to your ECS cluster, and shell access to a container instance
• The casting.yaml and foundryctl you installed with

Apply Changes

For each change:

1. Add the section's snippet to casting.yaml. Foundry already generates the default receivers and processors, so add only what you change.

2. Run this from the directory that contains casting.yaml to apply the change:

   Copy

   foundryctl cast -f casting.yaml

   Foundry updates the collector's config, and the collector restarts with it on every instance.

3. Run this on the machine where you use the AWS CLI to check that the collector tasks restart:

   Copy

   aws ecs describe-services --region <aws-region> --cluster <ecs-cluster-arn> \
     --services signoz-collector-agent \
     --query 'services[0].deployments[].[status,rolloutState,runningCount]' --output table

   Verify these values:

   • <aws-region>: The foundry.signoz.io/ecs-region value in casting.yaml.
   • <ecs-cluster-arn>: The foundry.signoz.io/ecs-cluster-arn value in casting.yaml.

   Then read the collector's logs on a container instance, as View the Collector's Logs shows, to confirm there are no configuration errors.

4. In SigNoz, confirm data in Infrastructure Monitoring for host and container metrics, Logs for container and host logs, and Traces for application traces.

Send Data from Applications to the Collector

Host metrics, container metrics, and container and host logs flow as soon as the collector is running.

Where and how to send data from your application

To send traces, metrics, and logs from an application, instrument it with OpenTelemetry (language guides), then point it at the collector on its instance. The address depends on the task's network mode:

For the host network mode, add this to the application's container definition:

{
  "environment": [
    { "name": "OTEL_EXPORTER_OTLP_PROTOCOL", "value": "http/protobuf" },
    { "name": "OTEL_EXPORTER_OTLP_ENDPOINT", "value": "http://localhost:4318" },
    { "name": "OTEL_SERVICE_NAME", "value": "<service-name>" }
  ]
}

Verify these values:

• <service-name>: The name your application shows under in SigNoz. It becomes service.name.

For the bridge network mode, add this instead. 172.17.0.1 is the instance's address on Docker's default bridge network, unless the instance's Docker uses a different bridge address:

{
  "environment": [
    { "name": "OTEL_EXPORTER_OTLP_PROTOCOL", "value": "http/protobuf" },
    { "name": "OTEL_EXPORTER_OTLP_ENDPOINT", "value": "http://172.17.0.1:4318" },
    { "name": "OTEL_SERVICE_NAME", "value": "<service-name>" }
  ]
}

Verify these values:

• <service-name>: The name your application shows under in SigNoz. It becomes service.name.

Use port 4317 with OTEL_EXPORTER_OTLP_PROTOCOL=grpc for OTLP over gRPC.

In the awsvpc network mode, the task has its own network address, so it reads the instance's private IP from the EC2 instance metadata service in the container's entryPoint:

{
  "entryPoint": [
    "sh",
    "-c",
    "TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token -H \"X-aws-ec2-metadata-token-ttl-seconds: 21600\"); export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_EXPORTER_OTLP_ENDPOINT=\"http://$(curl -s -H \"X-aws-ec2-metadata-token: $TOKEN\" http://169.254.169.254/latest/meta-data/local-ipv4):4318\"; <application-start-command>"
  ],
  "command": [],
  "environment": [
    { "name": "OTEL_SERVICE_NAME", "value": "<service-name>" }
  ]
}

Verify these values:

• <application-start-command>: The command that starts your application. The image needs sh and curl.
• <service-name>: The name your application shows under in SigNoz. It becomes service.name.

This works unless the instance's ECS agent blocks the metadata service for awsvpc tasks, which it does when ECS_AWSVPC_BLOCK_IMDS=true is set, as Amazon EC2 instances additional configuration describes.

In the instance's security group, allow inbound TCP 4317 and 4318 only from the task's security group. Keep the collector task's other ports closed: 13133 for the collector's health check and 2772 for the AppConfig agent.

Redeploy the application and send it some traffic. Within a minute it shows under Services in SigNoz, and its traces carry the instance's host.name.

Export to SigNoz

The OTLP/HTTP exporter sends the telemetry the collector gathers to SigNoz.

Foundry generates this by default: the otlphttp/signoz exporter sends every pipeline to SIGNOZ_INGESTION_ENDPOINT.

Collect Host Metrics

Use the hostmetrics receiver to scrape CPU, memory, disk, filesystem, load, and network metrics from each container instance.

Foundry collects host metrics by default, every 60s. In casting.yaml, add this to collect them every 30s and enable system.cpu.utilization:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              hostmetrics:
                collection_interval: 30s
                scrapers:
                  cpu:
                    metrics:
                      system.cpu.utilization:
                        enabled: true

scrapers is a map, so the scrapers and metrics you add join Foundry's. Then apply the change.

Collect Container Metrics

Use the docker_stats receiver to collect CPU, memory, network, and block I/O metrics of every container on each container instance.

Foundry collects container metrics by default, every 60s. In casting.yaml, add this to collect them every 30s and enable container.uptime:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              docker_stats:
                collection_interval: 30s
                metrics:
                  container.uptime:
                    enabled: true

metrics is a map, so the metrics you add join Foundry's. Then apply the change.

Collect Container Logs

Use the filelog receiver to tail the logs of containers that use the json-file log driver on each container instance.

Foundry collects container logs by default. To attribute them to their ECS task, set the json-file log driver with the labels option in each application container's definition:

{
  "logConfiguration": {
    "logDriver": "json-file",
    "options": {
      "max-size": "10m",
      "max-file": "3",
      "labels": "com.amazonaws.ecs.task-definition-family,com.amazonaws.ecs.container-name,com.amazonaws.ecs.task-arn,com.amazonaws.ecs.cluster"
    }
  }
}

Redeploy the application. Each log record then carries service.name, set to the task family, plus aws.ecs.container.name, aws.ecs.task.arn, and aws.ecs.cluster.name.

Disable Logs Collection

Stop collecting container logs when you only need host logs and the logs your applications send over OTLP. Foundry collects container logs by default. In casting.yaml, add this to remove the filelog receiver from the logs pipeline:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            service:
              pipelines:
                logs:
                  receivers: [otlp/http, otlp/grpc, filelog/host]

Lists replace, so the logs pipeline is restated with Foundry's other receivers. Host logs and logs that applications send over OTLP keep flowing. Then apply the change.

Collect Host Logs

Use the filelog/host receiver to tail the container instance's system logs and the ECS agent logs.

Foundry collects /var/log/messages, /var/log/secure, and /var/log/ecs/*.log by default, under service.name ecs-host. On Amazon Linux 2023, which logs to the systemd journal, /var/log/messages and /var/log/secure do not exist. In casting.yaml, add this to also collect /var/log/cloud-init.log:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              filelog/host:
                include:
                  - /hostfs/var/log/messages
                  - /hostfs/var/log/secure
                  - /hostfs/var/log/ecs/*.log
                  - /hostfs/var/log/cloud-init.log

Lists replace, so include is restated with Foundry's default paths, each under /hostfs. Then apply the change.

Resource Detection

The resourcedetection processor adds host and cloud resource attributes, so you can filter and group by instance in SigNoz.

Foundry runs the env and ec2 detectors by default, and the External casting runs env and system. The env detector adds the collector's OTEL_RESOURCE_ATTRIBUTES to every signal. In casting.yaml, add this to also add the cluster ARN:

spec:
  collector:
    spec:
      env:
        OTEL_RESOURCE_ATTRIBUTES: deployment.environment=<DEPLOYMENT_ENVIRONMENT>,aws.ecs.cluster.arn=<ecs-cluster-arn>

Verify these values:

• <DEPLOYMENT_ENVIRONMENT>: The deployment environment you set at install, for example production.
• <ecs-cluster-arn>: The cluster ARN in the foundry.signoz.io/ecs-cluster-arn annotation.

Then apply the change.

Tune Batch and Memory Limiter

Adjust the batch processor for throughput and the memory_limiter processor for stability based on your workload. See Collector Configuration Best Practices for detailed guidance.

The batch processor batches spans, metrics, and logs before export. That reduces connections and improves compression. It supports both size-based and time-based batching.

Foundry's defaults are send_batch_size: 1000, send_batch_max_size: 2048, timeout: 10s, and a fixed limit_mib: 4000. In casting.yaml, add this to change the batch sizes and size the memory limiter as a share of the instance's memory:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            processors:
              batch:
                send_batch_size: 15000
                send_batch_max_size: 20000
              memory_limiter:
                limit_mib: null
                spike_limit_mib: null
                limit_percentage: 80
                spike_limit_percentage: 25

Keys you leave out keep Foundry's defaults. Foundry sets only a soft memory reservation on the collector container, so the percentages are a share of the instance's memory. To bound the collector, set a hard memory limit with the patch in Resource Requests and Limits, and the percentages then apply to that limit.

Then apply the change.

Resource Requests and Limits

Foundry reserves cpu: 256 and memoryReservation: 512 for the collector container by default. In casting.yaml, add a patch that raises the reservations and sets a hard memory limit:

spec:
  patches:
    - target: collectionagent/collector/agent/collector.tf.json
      operations:
        - op: replace
          path: /locals/containers_collector_agent/1/cpu
          value: 512
        - op: replace
          path: /locals/containers_collector_agent/1/memoryReservation
          value: 1024
        - op: add
          path: /locals/containers_collector_agent/1/memory
          value: 2048

memory must be greater than memoryReservation, as the task definition parameters describe. Then apply the change.

Add Custom Processors to Pipelines

A processor transforms or enriches telemetry once a pipeline lists it under processors.

Foundry's pipelines are traces, metrics and logs. In casting.yaml, add this to define a processor and add it to the traces pipeline:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            processors:
              attributes/team:
                actions:
                  - key: team
                    value: platform
                    action: insert
            service:
              pipelines:
                traces:
                  processors: [memory_limiter, resourcedetection, attributes/team, batch]

Lists replace, so processors is restated with Foundry's defaults (memory_limiter, resourcedetection, batch), with yours before batch. Then apply the change.

Exposing Ports for Custom Receivers

A receiver that listens on a port needs that port reachable from your applications. The following example adds the statsd receiver on UDP port 8125.

In casting.yaml, add the statsd receiver, list it in the metrics pipeline, and map the port on every instance:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              statsd:
                endpoint: 0.0.0.0:8125
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc, docker_stats, hostmetrics, statsd]
  patches:
    - target: collectionagent/collector/agent/collector.tf.json
      operations:
        - op: add
          path: /locals/containers_collector_agent/1/portMappings/-
          value: {containerPort: 8125, hostPort: 8125, protocol: udp}

Lists replace, so the metrics pipeline is restated; applications reach port 8125 at the address for their network mode. Then apply the change.

Disable Metrics Collection

Turn off host and container metrics when you only need the telemetry your applications send. Foundry collects both by default. In casting.yaml, add this to remove the docker_stats and hostmetrics receivers:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              docker_stats: null
              hostmetrics: null
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc]

Lists replace, so the metrics pipeline is restated with Foundry's two OTLP receivers, and OTLP metrics from your applications keep flowing. Then apply the change.

View the Collector's Logs

Read the collector's output after a change, or when data stops arriving in SigNoz, to find configuration errors. The output stays on each container instance.

Run this on a container instance to read the collector's output from the last 5 minutes:

docker logs --since 5m $(docker ps -aq --filter label=com.amazonaws.ecs.container-name=signoz-collector-agent | head -n 1)

curl -sS -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json' -d '{}' http://localhost:4318/v1/traces

Next Steps

• Monitoring ECS EC2 using OpenTelemetry Collection Agents
• Telemetry Data Requirements for Infrastructure Monitoring
• Collector Configuration Best Practices