Skip to main content

Send Cloudwatch Logs to SigNoz

Overview​

AWS CloudWatch is a monitoring service that helps users keep tabs on their AWS resources. There are some challenges that users encounter on Cloudwatch, such as the absence of a unified observability experience, slightly higher costs, a focus on AWS-centric environments, and user experience limitations.

SigNoz effectively addresses these challenges, and in the following steps, we'll outline how to forward logs from AWS CloudWatch to SigNoz seamlessly.

Setup​

You can choose from the two options below.

Step 1 : Setup the OTel Collector

Follow the instructions in the SigNoz Cloud section of this tutorial to setup the OpenTelemetry Collector.

Step 2 : Configure AWS

Create a ~/.aws/credentials file in the machine which should have aws_access_key_id and the aws_secret_access_key in the default section of credentials file.

The below snippet shows an example of the credentials file.

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

[user1]
aws_access_key_id=AKIAI44QH8DHBEXAMPLE
aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

The account corresponding to these credentials should have the below-mentioned AWS Identity and Access Management (IAM) policy. This policy allows the account associated with these permissions to describe and filter log events within all log groups in the specified AWS account, which is crucial for setting up the necessary permissions to forward CloudWatch logs to SigNoz.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups",
                "logs:FilterLogEvents"
            ],
            "Resource": "arn:aws:logs:*:090340947446:log-group:*"
        }
    ]
}
note

Make sure you have AWS configured on the machine where otel-collector is running

Step 3 : Configure the awscloudwatch receiver

We’ll add an awscloudwatch receiver inside the receivers section of the config.yaml that we created in Step 1 for the OTel collector.

You can configure your receiver to collect logs with different conditions. To see the different parameters, some sample configurations and more details about the awscloudwatch receiver, check out this GitHub link.

Here are two sample configurations:

  1. This configuration below will do autodiscovery and collect 100 log groups starting with prefix application.

receivers:
...
    awscloudwatch:
        region: us-east-1
        logs:
          poll_interval: 1m
          groups:
            autodiscover:
              limit: 100
              prefix: application
...
  1. This configuration below will not do autodiscovery and specifies the names of the log groups to collect.
receivers:
...
awscloudwatch:
  profile: 'my-profile'
  region: us-west-1
  logs:
    poll_interval: 5m
    groups:
      named:
        /aws/eks/dev-0/cluster:

...

Step 4 : Send logs to SigNoz
To test out the receiver, create a pipeline in the pipeline section of the config.yaml created in Step 1 for Otel Collector.

...
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics/internal:
      receivers: [prometheus, hostmetrics]
      processors: [resourcedetection, batch]
      exporters: [otlp]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp, awscloudwatch]
      processors: [batch]
      exporters: [otlp]

This will log out everything from the receiver and you should be able to see your Cloudwatch logs in the logs tabs of SigNoz Cloud UI.