Compute Engine Logging

Overview

This documentation provides a detailed walkthrough to send the Google Compute Engine logs directly to SigNoz.

Prerequisites

Google Cloud account with administrative privilege or Compute Instance Admin privilege.
Access to a project in GCP

Get started with Compute Engine Configuration

Create the Compute Engine instance using the following steps:

Step 1: Go to your GCP console and search for Compute Engine, go to Compute Engine service and click on CREATE INSTANCE.

Step 2: Give an apprpriate name for the Compute Engine instance. Select an appropriate region and zone. You can choose a machine as per your requirements. After choosing all the appropriate settings, click on CREATE button at the bottom of the page.

With this, the Compute Engine instance is created.

Create PubSub Topic

Follow the steps mentioned in the Creating Pub/Sub Topic document to create the Pub/Sub topic.

Create Log Router to Pub/Sub Topic

Follow the steps mentioned in the Log Router Setup document to create the Log Router.

To ensure you filter out only the Compute Engine logs, use the following filter conditions:

resource.type="gce_instance"

Setup OTel Collector

Follow the steps mentioned in the Creating Compute Engine document to create another Compute Engine instance. We will be installing OTel Collector on this instance.

Install OTel Collector as agent

Firstly, we will establish the authentication using the following commands:

Initialize gcloud:

gcloud init

Authenticate into GCP:

gcloud auth application-default login

Let us now proceed to the OTel Collector installation:

Step 1: Download otel-collector tar.gz for your architecture

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.135.0/otelcol-contrib_0.135.0_linux_amd64.tar.gz

Step 2: Extract otel-collector tar.gz to the otelcol-contrib folder

mkdir otelcol-contrib && tar xvzf otelcol-contrib_0.135.0_linux_amd64.tar.gz -C otelcol-contrib

Step 3: Create config.yaml in the folder otelcol-contrib with the below content in it. Replace <region> with the appropriate SigNoz Cloud region. Replace SIGNOZ_INGESTION_KEY with what is provided by SigNoz:

Note: The googlecloudpubsub receiver’s encoding was updated from the deprecated raw_text to text_encoding. This configuration has been tested and works with otelcol-contrib v0.135.0.

extensions:
  text_encoding:
    encoding: utf-8
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  googlecloudpubsub:
    project: <gcp-project-id>
    subscription: projects/<gcp-project-id>/subscriptions/<pubsub-topic's-subscription>
    encoding: text_encoding
processors:
  batch: {}
  resource/env:
    attributes:
    - key: deployment.environment
      value: prod
      action: upsert
exporters:
  otlp:
    endpoint: "ingest.<region>.signoz.cloud:443"
    tls:
      insecure: false
    headers:
      "signoz-ingestion-key": "<SigNoz-Key>"
service:
  extensions: [text_encoding]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp, googlecloudpubsub]
      processors: [batch, resource/env]
      exporters: [otlp]

Step 4: Once we are done with the above configurations, we can now run the collector service with the following command:

From the otelcol-contrib, run the following command:

./otelcol-contrib --config ./config.yaml

Run in background

If you want to run OTel Collector process in the background:

./otelcol-contrib --config ./config.yaml &> otelcol-output.log & echo "$!" > otel-pid

The above command sends the output of the otel-collector to otelcol-output.log file and prints the process id of the background running OTel Collector process to the otel-pid file.

If you want to see the output of the logs you’ve just set up for the background process, you may look it up with:

tail -f -n 50 otelcol-output.log

Invoking Compute Engine logs

Step 1: Go to the Compute Engine homepage, and click on the instance created at the beginning of this guide.

Step 2: Click on the EDIT button at the top of the Compute Engine instance's page.

Step 3: After scrolling a bit on the edit page, you will see the "Additional disks" section. Click on the ADD NEW DISK button.

Step 4: Give an appropriate name and description for the additional disk. You can reduce the size to 10. Click on SAVE button at the bottom.

Step 5: Click on SAVE button for the Compute Engine instance.

This will generate the Compute Engine logs which you can now see on the SigNoz Cloud.

Prerequisites

Google Cloud account with administrative privilege or Compute Instance Admin privilege.
Access to a project in GCP

For more details on how to configure Self-Hosted SigNoz for Logs, check official documentation by Self-Hosted SigNoz and navigate to the "Send Logs to Self-Hosted SigNoz" section.

Get started with Compute Engine Configuration

Create the Compute Engine instance using the following steps:

Step 1: Go to your GCP console and search for Compute Engine, go to Compute Engine service and click on CREATE INSTANCE.

With this, the Compute Engine instance is created.

Create PubSub Topic

Follow the steps mentioned in the Creating Pub/Sub Topic document to create the Pub/Sub topic.

Create Log Router to Pub/Sub Topic

Follow the steps mentioned in the Log Router Setup document to create the Log Router.

To ensure you filter out only the Compute Engine logs, use the following filter conditions:

resource.type="gce_instance"

Setup OTel Collector

Follow the steps mentioned in the Creating Compute Engine document to create the Compute Engine instance.

Install OTel Collector as agent

Firstly, we will establish the authentication using the following commands:

Initialize gcloud:

gcloud init

Authenticate into GCP:

gcloud auth application-default login

Let us now proceed to the OTel Collector installation:

Step 1: Download otel-collector tar.gz for your architecture

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.135.0/otelcol-contrib_0.135.0_linux_amd64.tar.gz

Step 2: Extract otel-collector tar.gz to the otelcol-contrib folder

mkdir otelcol-contrib && tar xvzf otelcol-contrib_0.135.0_linux_amd64.tar.gz -C otelcol-contrib

Step 3: Create config.yaml in the folder otelcol-contrib with the below content in it. We are using the clickhouselogsexporter in this case.

Note: The googlecloudpubsub receiver’s encoding was updated from the deprecated raw_text to text_encoding. This configuration has been tested and works with otelcol-contrib v0.135.0.

extensions:
  text_encoding:
    encoding: utf-8
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  googlecloudpubsub:
    project: <gcp-project-id>
    subscription: projects/<gcp-project-id>/subscriptions/<pubsub-topic's-subscription>
    encoding: text_encoding
processors:
  batch: {}
  resource/env:
    attributes:
    - key: deployment.environment
      value: prod
      action: upsert
exporters:
  clickhouselogsexporter:
    dsn: tcp://clickhouse:9000/
    timeout: 5s
    sending_queue:
      queue_size: 100
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
service:
  extensions: [text_encoding]
  pipelines:
    logs:
      receivers: [otlp, googlecloudpubsub]
      processors: [batch, resource/env]
      exporters: [clickhouselogsexporter]

Step 4: Once we are done with the above configurations, we can now run the collector service with the following command:

From the otelcol-contrib, run the following command:

./otelcol-contrib --config ./config.yaml

Run in background

If you want to run OTel Collector process in the background:

./otelcol-contrib --config ./config.yaml &> otelcol-output.log & echo "$!" > otel-pid

The above command sends the output of the otel-collector to otelcol-output.log file and prints the process id of the background running OTel Collector process to the otel-pid file.

If you want to see the output of the logs you’ve just set up for the background process, you may look it up with:

tail -f -n 50 otelcol-output.log

Invoking Compute Engine logs

Step 1: Go to the Compute Engine homepage, and click on the instance created at the beginning of this guide.

Step 2: Click on the EDIT button at the top of the Compute Engine instance's page.

Step 3: After scrolling a bit on the edit page, you will see the "Additional disks" section. Click on the ADD NEW DISK button.

Step 4: Give an appropriate name and description for the additional disk. You can reduce the size to 10. Click on SAVE button at the bottom.

Step 5: Click on SAVE button for the Compute Engine instance.

This will generate the Compute Engine logs which you can now see on SigNoz.

Overview

This documentation provides a detailed walkthrough to send the Google Compute Engine logs directly to SigNoz.

Prerequisites

Google Cloud account with administrative privilege or Compute Instance Admin privilege.
Access to a project in GCP

Get started with Compute Engine Configuration

Create the Compute Engine instance using the following steps:

Step 1: Go to your GCP console and search for Compute Engine, go to Compute Engine service and click on CREATE INSTANCE.

With this, the Compute Engine instance is created.

Create PubSub Topic

Follow the steps mentioned in the Creating Pub/Sub Topic document to create the Pub/Sub topic.

Create Log Router to Pub/Sub Topic

Follow the steps mentioned in the Log Router Setup document to create the Log Router.

To ensure you filter out only the Compute Engine logs, use the following filter conditions:

resource.type="gce_instance"

Setup OTel Collector

Follow the steps mentioned in the Creating Compute Engine document to create another Compute Engine instance. We will be installing OTel Collector on this instance.

Install OTel Collector as agent

Firstly, we will establish the authentication using the following commands:

Initialize gcloud:

gcloud init

Authenticate into GCP:

gcloud auth application-default login

Let us now proceed to the OTel Collector installation:

Step 1: Download otel-collector tar.gz for your architecture

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.135.0/otelcol-contrib_0.135.0_linux_amd64.tar.gz

Step 2: Extract otel-collector tar.gz to the otelcol-contrib folder

mkdir otelcol-contrib && tar xvzf otelcol-contrib_0.135.0_linux_amd64.tar.gz -C otelcol-contrib

Note: The googlecloudpubsub receiver’s encoding was updated from the deprecated raw_text to text_encoding. This configuration has been tested and works with otelcol-contrib v0.135.0.

extensions:
  text_encoding:
    encoding: utf-8
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  googlecloudpubsub:
    project: <gcp-project-id>
    subscription: projects/<gcp-project-id>/subscriptions/<pubsub-topic's-subscription>
    encoding: text_encoding
processors:
  batch: {}
  resource/env:
    attributes:
    - key: deployment.environment
      value: prod
      action: upsert
exporters:
  otlp:
    endpoint: "ingest.<region>.signoz.cloud:443"
    tls:
      insecure: false
    headers:
      "signoz-ingestion-key": "<SigNoz-Key>"
service:
  extensions: [text_encoding]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp, googlecloudpubsub]
      processors: [batch, resource/env]
      exporters: [otlp]

Step 4: Once we are done with the above configurations, we can now run the collector service with the following command:

From the otelcol-contrib, run the following command:

./otelcol-contrib --config ./config.yaml

Run in background

If you want to run OTel Collector process in the background:

./otelcol-contrib --config ./config.yaml &> otelcol-output.log & echo "$!" > otel-pid

The above command sends the output of the otel-collector to otelcol-output.log file and prints the process id of the background running OTel Collector process to the otel-pid file.

If you want to see the output of the logs you’ve just set up for the background process, you may look it up with:

tail -f -n 50 otelcol-output.log

Invoking Compute Engine logs

Step 1: Go to the Compute Engine homepage, and click on the instance created at the beginning of this guide.

Step 2: Click on the EDIT button at the top of the Compute Engine instance's page.

Step 3: After scrolling a bit on the edit page, you will see the "Additional disks" section. Click on the ADD NEW DISK button.

Step 4: Give an appropriate name and description for the additional disk. You can reduce the size to 10. Click on SAVE button at the bottom.

Step 5: Click on SAVE button for the Compute Engine instance.

This will generate the Compute Engine logs which you can now see on the SigNoz Cloud.

Prerequisites

Google Cloud account with administrative privilege or Compute Instance Admin privilege.
Access to a project in GCP

For more details on how to configure Self-Hosted SigNoz for Logs, check official documentation by Self-Hosted SigNoz and navigate to the "Send Logs to Self-Hosted SigNoz" section.

Get started with Compute Engine Configuration

Create the Compute Engine instance using the following steps:

Step 1: Go to your GCP console and search for Compute Engine, go to Compute Engine service and click on CREATE INSTANCE.

With this, the Compute Engine instance is created.

Create PubSub Topic

Follow the steps mentioned in the Creating Pub/Sub Topic document to create the Pub/Sub topic.

Create Log Router to Pub/Sub Topic

Follow the steps mentioned in the Log Router Setup document to create the Log Router.

To ensure you filter out only the Compute Engine logs, use the following filter conditions:

resource.type="gce_instance"

Setup OTel Collector

Follow the steps mentioned in the Creating Compute Engine document to create the Compute Engine instance.

Install OTel Collector as agent

Firstly, we will establish the authentication using the following commands:

Initialize gcloud:

gcloud init

Authenticate into GCP:

gcloud auth application-default login

Let us now proceed to the OTel Collector installation:

Step 1: Download otel-collector tar.gz for your architecture

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.135.0/otelcol-contrib_0.135.0_linux_amd64.tar.gz

Step 2: Extract otel-collector tar.gz to the otelcol-contrib folder

mkdir otelcol-contrib && tar xvzf otelcol-contrib_0.135.0_linux_amd64.tar.gz -C otelcol-contrib

Step 3: Create config.yaml in the folder otelcol-contrib with the below content in it. We are using the clickhouselogsexporter in this case.

Note: The googlecloudpubsub receiver’s encoding was updated from the deprecated raw_text to text_encoding. This configuration has been tested and works with otelcol-contrib v0.135.0.

extensions:
  text_encoding:
    encoding: utf-8
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  googlecloudpubsub:
    project: <gcp-project-id>
    subscription: projects/<gcp-project-id>/subscriptions/<pubsub-topic's-subscription>
    encoding: text_encoding
processors:
  batch: {}
  resource/env:
    attributes:
    - key: deployment.environment
      value: prod
      action: upsert
exporters:
  clickhouselogsexporter:
    dsn: tcp://clickhouse:9000/
    timeout: 5s
    sending_queue:
      queue_size: 100
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
service:
  extensions: [text_encoding]
  pipelines:
    logs:
      receivers: [otlp, googlecloudpubsub]
      processors: [batch, resource/env]
      exporters: [clickhouselogsexporter]

Step 4: Once we are done with the above configurations, we can now run the collector service with the following command:

From the otelcol-contrib, run the following command:

./otelcol-contrib --config ./config.yaml

Run in background

If you want to run OTel Collector process in the background:

./otelcol-contrib --config ./config.yaml &> otelcol-output.log & echo "$!" > otel-pid

The above command sends the output of the otel-collector to otelcol-output.log file and prints the process id of the background running OTel Collector process to the otel-pid file.

If you want to see the output of the logs you’ve just set up for the background process, you may look it up with:

tail -f -n 50 otelcol-output.log

Invoking Compute Engine logs

Step 1: Go to the Compute Engine homepage, and click on the instance created at the beginning of this guide.

Step 2: Click on the EDIT button at the top of the Compute Engine instance's page.

Step 3: After scrolling a bit on the edit page, you will see the "Additional disks" section. Click on the ADD NEW DISK button.

Step 4: Give an appropriate name and description for the additional disk. You can reduce the size to 10. Click on SAVE button at the bottom.

Step 5: Click on SAVE button for the Compute Engine instance.

This will generate the Compute Engine logs which you can now see on SigNoz.

Compute Engine Logging

Overview

Prerequisites

Get started with Compute Engine Configuration

Create PubSub Topic

Create Log Router to Pub/Sub Topic

Setup OTel Collector

Install OTel Collector as agent

Invoking Compute Engine logs

Prerequisites

Get started with Compute Engine Configuration

Create PubSub Topic

Create Log Router to Pub/Sub Topic

Setup OTel Collector

Install OTel Collector as agent

Invoking Compute Engine logs

Was this page helpful?

Is this page helpful?

Compute Engine Logging

Overview

Prerequisites

Get started with Compute Engine Configuration

Create PubSub Topic

Create Log Router to Pub/Sub Topic

Setup OTel Collector

Install OTel Collector as agent

Invoking Compute Engine logs

Prerequisites

Get started with Compute Engine Configuration

Create PubSub Topic

Create Log Router to Pub/Sub Topic

Setup OTel Collector

Install OTel Collector as agent

Invoking Compute Engine logs

Was this page helpful?

Is this page helpful?