Collecting Data from ECS using Sidecar

Overview

This documentation will show you how to collect data from your ECS infrastructure using sidecar. The sidecar method involves deploying an additional container (the "sidecar") that runs in each application container of your ECS cluster. A sidecar container, is an auxiliary container that offers supporting features like logging and monitoring to the main application container. The sidecar container will collect metrics and forward any received OTLP data to SigNoz.

Select the type of SigNoz instance you are running: SigNoz Cloud or Self-Hosted.

SigNoz CloudSelf-Host

Below are the steps to collect your metrics and logs from ECS infrastructure:

• Prerequisites
• Create OpenTelemetry Collector Config file
• Create Sidecar Collector Container
• Deploy Task Definition
• Verify Data in SigNoz
• Send Traces Data from your application
• Send Logs Data from your application

Prerequisites

• An ECS cluster running with at least one task definition
• ECS cluster can be either of the launch types: Fargate, EC2 or External
• SigNoz Cloud account

Setting up Sidecar Container

Step 1: Create SigNoz OtelCollector Config

• Navigate to AWS Parameter Store and create a new parameter named /ecs/signoz/otelcol-sidecar.yaml.

• Download the otelcol-sidecar YAML configuration file:

  wget https://github.com/SigNoz/benchmark/raw/main/ecs/otelcol-sidecar.yaml
  

• Update {region} and SIGNOZ_INGESTION_KEY values in your YAML configuration file and copy the updated content of the otelcol-sidecar.yaml file and paste it in the value field of the /ecs/signoz/otelcol-sidecar.yaml parameter that you created.

You will be able to get {region} and SIGNOZ_INGESTION_KEY values in your SigNoz Cloud account under Settings --> Ingestion Settings.

✅ Info

After successful set up, feel free to remove logging exporter if it gets too noisy. To do so, simply remove the logging exporter from the exporters list in the following pipelines: traces, metrics, metrics/aws, and logs from the otelcol-sidecar.yaml file.

Step 2: Create Sidecar Collector Container

This step involves integrating the SigNoz collector into your ECS task definitions as a sidecar container. The sidecar collector container will run alongside your application container(s) within the same ECS task and will collect ECS container metrics and send them to SigNoz Cloud. It also acts as a gateway to send any telemetry data from your application container to SigNoz Cloud.

Update task definition of your application

In your ECS task definition, include a new container definition specifically for the sidecar container. This container will operate alongside your main application container(s) within the same task definition. The JSON configuration for that will look like this:

{
    ...
    "containerDefinitions": [
        ...,
        {
            "name": "signoz-collector",
            "image": "signoz/signoz-otel-collector:0.88.13",
            "user": "root",
            "command": [
                "--config=env:SIGNOZ_CONFIG_CONTENT"
            ],
            "secrets": [
                {
                "name": "SIGNOZ_CONFIG_CONTENT",
                "valueFrom": "/ecs/signoz/otelcol-sidecar.yaml"
                }
            ],
            "memory": 1024,
            "cpu": 512,
            "essential": true,
            "portMappings": [
                {
                    "protocol": "tcp",
                    "containerPort": 4317
                },
                {
                    "protocol": "tcp",
                    "containerPort": 4318
                },
                {
                    "containerPort": 8006,
                    "protocol": "tcp"
                }
            ],
            "healthCheck": {
                "command": [
                    "CMD-SHELL",
                    "wget -qO- http://localhost:13133/ || exit 1"
                ],
                "interval": 5,
                "timeout": 6,
                "retries": 5,
                "startPeriod": 1
            },
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                "awslogs-group": "/ecs/signoz-otel-EC2-sidcar",
                "awslogs-region": "<aws-region>",
                "awslogs-stream-prefix": "ecs",
                "awslogs-create-group": "True"
                }
            }
        }
    ]
...
}

The sidecar container will run the SigNoz collector and includes essential components like the name ("signoz-collector"), image (e.g., "signoz/signoz-otel-collector:0.88.7"), and command for execution, alongside secrets for secure configuration. It also specifies memory and CPU resources, port mappings for network communication, a health check to ensure operational integrity, and log configuration for output management. These elements collectively enable the container to efficiently collect and forward telemetry data to SigNoz.

Update ECS Task Execution Role

The ECS Task Execution Role is an AWS IAM role that grants permissions to the ECS agent to make AWS API calls on behalf of the user. When integrating SigNoz as a sidecar container, the task execution role needs additional permissions to access the AWS Systems Manager Parameter Store, where the SigNoz configuration is stored and to access CloudWatch Logs for log management of the sidecar containers.

To modify the ECS Task Execution Role, follow these steps:

1. Identify the Role: Identify the IAM role used by your ECS tasks for execution. This role is specified when creating an ECS service or can be found in the ECS task definition. It's often named something like ecsTaskExecutionRole.

2. Edit the Role: Navigate to the IAM console in the AWS Management Console, find the role by name, and open its details page.

3. Attach Policies or Add inline Policy:

   There are two ways to grant access to the Parameter store:

   • Attach AWS Managed Policies: If the role doesn't already have the following policies, attach them:

     • AmazonSSMReadOnlyAccess
     • CloudWatchLogsFullAccess

   • Add Inline Policy: Alternatively, for more granular control, you can create an inline policy that specifically grants access to only the necessary resources in the Parameter Store. The JSON for the inline policy will be:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Action": [
                   "ssm:GetParameter"
               ],
               "Resource": [
                   "arn:aws:ssm:<aws-region>:<aws-account-id>:parameter/ecs/signoz/otelcol-sidecar.yaml"
               ],
               "Effect": "Allow"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "logs:CreateLogStream",
                   "logs:CreateLogGroup",
                   "logs:PutLogEvents",
                   "logs:DescribeLogStreams",
                   "logs:DescribeLogGroups"
               ],
               "Resource": "*"
           }
       ]
   }
   

Alternatively, you can add the policy AmazonSSMReadOnlyAccess and CloudWatchLogsFullAccess to the ECS Task Execution Role.

Update ECS Task Role

The ECS Task Role is distinct from the ECS Task Execution Role. While the execution role grants the ECS agent permission to manage the lifecycle of containers (like pulling images and storing logs), the task role is assumed by the task itself, including your application and any sidecar containers. This role provides the permissions necessary for the application to make AWS API calls directly.

To update the ECS Task Role, follow these steps:

1. Identify the Role: Determine the IAM role your ECS tasks are currently using to interact with AWS services. This role is specified in the ECS task definition under the "taskRoleArn" field.

2. Edit the Role: Go to the IAM section of the AWS Management Console, locate the role by its name, and open its configuration.

3. Attach Policies or Add Inline Policy:

   There are two ways to grant access to the Parameter store:

   • Attach AWS Managed Policies: If the role doesn't already have the following policies, attach them:

     • AmazonSSMReadOnlyAccess
     • CloudWatchLogsFullAccess

   • Add Inline Policy for Granular Access: For tighter security, you might opt to create an inline policy that specifies exactly which resources the tasks can access and what actions they can perform on those resources. This is particularly important for accessing specific resources like the Parameter Store parameters used by the SigNoz sidecar. The JSON for the inline policy will be:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Action": [
                   "ssm:GetParameter"
               ],
               "Resource": [
                   "arn:aws:ssm:<aws-region>:<aws-account-id>:parameter/ecs/signoz/otelcol-sidecar.yaml"
               ],
               "Effect": "Allow"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "logs:CreateLogStream",
                   "logs:CreateLogGroup",
                   "logs:PutLogEvents",
                   "logs:DescribeLogStreams",
                   "logs:DescribeLogGroups"
               ],
               "Resource": "*"
           }
       ]
   }
   

Step 3: Deploy the task definition

If your application runs as an ECS service, you update the service to use the new revision of your task definition. This tells ECS to start new tasks based on this updated definition and gracefully replace the old tasks with the new ones, ensuring minimal disruption to your application.

If you're not using a service and instead run tasks directly, you manually start a new task using the updated task definition. This approach is less common for long-running applications but might be used for testing or one-off executions.

Step 4: Verify data in SigNoz

To verify that your sidecar container is running, go to the Dashboard section of SigNoz and import the dashboard ECS - Container Metrics Dashboard from here.

Send Traces Data from Applications

In this section, we will see how to send traces data from applications deployed in ECS to SigNoz using sidecar container that we deployed in the previous section.

Add OpenTelemetry Instrumentation to your Application

Instrumentation involves modifying your application code to generate telemetry data. This could mean adding specific OpenTelemetry API calls or relying on auto-instrumentation provided by the SDK for common libraries and frameworks.

To add OpenTelemetry instrumentation to your application, follow the docs here.

Configure OTLP Endpoint

In your application task definition, you need to set the OTLP endpoint to the endpoint of the sidecar container. This can be done by setting the environment variable OTEL_EXPORTER_OTLP_ENDPOINT to the endpoint of the sidecar container.

Select the network mode of your ECS task definition:

{
    ...
    "containerDefinitions": [
        {
            "name": "<your-container-name>",
            "environment": [
                {
                    "name": "OTEL_EXPORTER_OTLP_ENDPOINT",
                    "value": "http://signoz-collector:4317"
                },
                {
                    "name": "OTEL_RESOURCE_ATTRIBUTES",
                    "value": "service.name=<your-service-name>"
                }
            ],
            "links": [
                "signoz-collector"
            ],
            ...
        }
    ]
}

Rebuild and Deploy Application Container

After instrumenting your application and configuring the OTLP endpoint, you'll need to rebuild your application container with these changes and deploy it to ECS cluster using the same task definition that we used in the previous section.

Verify data in SigNoz

To verify that the data is being sent to SigNoz, you will need to generate some traffic to your application. Now go to the Services section of SigNoz and you should be able to see your application in the services list.

Send Logs Data from Applications

In this section, we will see how to send logs data from applications deployed in ECS to SigNoz using sidecar container that we deployed in the previous section.

Configure Log Router

In your application code, you need to configure the Fluent Bit log router to your application to the sidecar otel-collector container.

{
    ...
    {
        "name": "signoz-log-router",
        "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:stable",
        "cpu": 250,
        "memory": 512,
        "essential": true,
        "dependsOn": [
            {
                "containerName": "signoz-collector",
                "condition": "HEALTHY"
            }
        ],
        "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
                "awslogs-create-group": "True",
                "awslogs-group": "/ecs/ecs-signoz-log-router",
                "awslogs-region": "us-east-1",
                "awslogs-stream-prefix": "ecs"
            }
        },
        "firelensConfiguration": {
            "type": "fluentbit",
            "options": {
                "enable-ecs-log-metadata": "true"
            }
        }
    }
}

Send Logs to Sidecar Container

In your application task definition, you need to use awsfirelens log driver to send logs to the sidecar otel-collector container via Fluent Bit log router.

{
    ...
    "containerDefinitions": [
        {
            "name": "<your-container-name>",
            "dependsOn": [
                {
                    "containerName": "signoz-log-router",
                    "condition": "START"
                }
            ],
            "logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "Name": "forward",
                    "Match": "*",
                    "Host": "signoz-collector",
                    "Port": "8006",
                    "tls": "off",
                    "tls.verify": "off"
                }
            },
            "links": [
                "signoz-collector"
            ],
            ...
        }
    ]
}

Rebuild and Deploy Application Container

Now you can rebuild your application container and deploy it to ECS cluster using the same task definition that we updated in the previous section.

Verify data in SigNoz

To verify that the logs are being sent to SigNoz, you will need to generate some logs from your application. After which you can go to the Logs page in SigNoz UI and you should see logs from your application.