Cloud Storage Metrics
Overview
This document provides a detailed walkthrough on how to send Google Cloud Storage metrics to SigNoz. By the end of this guide, you will have a setup that sends your Cloud Storage metrics to SigNoz.
Here's a quick summary of what we will be doing in this guide
- Create Cloud Storage bucket and objects
- Create and configure Compute Engine VM instance to deploy Telegraf and OpenTelemetry Collector
- Deploy OpenTelemetry Collector to scrape the metrics from Google Cloud monitoring
- Send and Visualize the metrics in SigNoz Cloud
Prerequisites
- Google Cloud account with administrative privilege or Compute Instance Admin privilege.
- SigNoz Cloud Account (we are using SigNoz Cloud for this demonstration, we will also need ingestion details. To get your Ingestion Key and Ingestion URL, sign-in to your SigNoz Cloud Account and go to Settings >> Ingestion Settings)
- Access to a project in GCP
Setup
Create Cloud Storage bucket and objects
Step 1: On the GCP console, search for Cloud Storage, go to Cloud Storage service.
Step 2: Click on the CREATE button at the top of the Cloud Storage page to create a bucket.
Step 3: On the Create a Bucket page, provide an appropriate name for the bucket in the Name your bucket section, and click on CONTINUE.
In the Choose where to store your data section, choose an appropriate location type, and click on CONTINUE. You can select an appropriate option in the rest of the sections as per your requirements, or let them be as default.
After filling the complete form, click on the CREATE button at the bottom of the page. This will create the new bucket.
Step 4: Optionally, you can also upload files to the newly created bucket. For that, click on the UPLOAD FILES button, and select the file you want to upload.
Deploy OpenTelemetry to scrape the metrics from Google Cloud Monitoring
Step 1: Install and configure OpenTelemetry for scraping the metrics from Google Cloud Monitoring. Follow OpenTelemetry Binary Usage in Virtual Machine guide for detailed instructions.
Note that googlecloudmonitoring
receiver is supported only from version 0.112.0 and above of opentelemetry-collector. So, ensure you download and use the appropriate release.
Step 2: Create config.yaml
.
Under the googlecloudmonitoring
receiver, you need to specify the metrics that you want to capture in the metrics_list
. You can include all the metrics that you would like to capture for Cloud Storage, see the available metrics for Cloud Storage.
Here is the config.yaml
file for capturing Cloud Storage metrics:
receivers:
googlecloudmonitoring:
collection_interval: 5m # Can be specified in seconds (s), minutes (m), or hours (h)
project_id: my-project-id
metrics_list:
- metric_name: "storage.googleapis.com/api/request_count"
- metric_name: "storage.googleapis.com/network/received_bytes_count"
- metric_name: "storage.googleapis.com/network/sent_bytes_count"
- metric_name: "storage.googleapis.com/storage/object_count"
- metric_name: "storage.googleapis.com/storage/total_bytes"
- metric_name: "storage.googleapis.com/storage/total_byte_seconds"
processors:
resource/env:
attributes:
- key: deployment.environment
value: prod
action: upsert
batch: {}
exporters:
otlp:
endpoint: "ingest.{region}.signoz.cloud:443"
tls:
insecure: false
headers:
"signoz-access-token": "<SIGNOZ_INGESTION_KEY>"
service:
pipelines:
metrics:
receivers: [googlecloudmonitoring]
processors: [resource/env,batch]
exporters: [otlp]
Depending on the choice of your region for SigNoz cloud, the otlp endpoint will vary according to this table.
Region | Endpoint |
---|---|
US | ingest.us.signoz.cloud:443 |
IN | ingest.in.signoz.cloud:443 |
EU | ingest.eu.signoz.cloud:443 |
After successful configuration start the OTel Collector using following command:
./otelcol-contrib --config ./config.yaml &> otelcol-output.log & echo "$!" > otel-pid
Step 3: If the configurations are configured correctly, you can see the output logs from OpenTelemtry as follows:
Send and Visualize the metrics obtained by OpenTelemetry in SigNoz
Step 1: Go to the SigNoz Cloud URL and head over to the dashboard.
Step 2: If not already created, create a new dashboard. You can create the dashboard and multiple panel under it by following the instructions here.
Step 3: While creating the panel, select metric for Cloud Storage.
All metrics starting with storage_googleapis_com_
have been collected from Cloud Storage instances.
For example, storage_googleapis_com_storage_total_bytes
is one of the metrics which is collected.
Here is the sample dashboard for Cloud Storage:
Troubleshooting
If you run into any problems while setting up monitoring for your Cloud Storage metrics with SigNoz, consider these troubleshooting steps:
- Verify Configuration: Double-check your
config.yaml
file to ensure all settings, including the ingestion key and endpoint, are correct. - Review Logs: Look at the logs of OpenTelemetry Collector to identify any error messages or warnings that might provide insights into what’s going wrong.
- Update Dependencies: Ensure all relevant packages and dependencies are up-to-date to avoid compatibility issues.
- Consult Documentation: Review the SigNoz and OpenTelemetry documentation for any additional troubleshooting of the common issues.