OpenTelemetry Binary Usage in Virtual Machine
Overview
This tutorial shows how you can deploy OpenTelemetry binary an agent, which collects telemetry data. Data such as traces, metrics and logs generated by applications most likely running in the same virtual machine (VM).
It can also be used for collecting data from other VMs in the same cluster, data center or region, however binary is not recommended in that scenerio but container or deployment which can be easily scaled.
In this guide, you will also learn to set up hostmetrics receiver to collect metrics from the VM and view in SigNoz.
Prerequisites
- SigNoz application up and running
- SigNoz endpoint accessible from the VM
- availability of ports:
4317
,4318
,8888
,1777
,13133
Installation
You can obtain OpenTelemetry collector binary in the assets of each releases:
open-telemetry/opentelemetry-collector-releases/releases.
There are two ways of installation with binary release assets: deb
as
systemd
and tar.gz
as plain binary.
Systemd
Using deb
file, OpenTelemetry Collector will be installed as a systemd
and
default configuration prepopulated at /etc/otelcol-contrib
path. This method
would be preferable in case you want the OpenTelemetry collector to always be
running in the background.
To download deb
file of release version 0.55.0
:
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.55.0/otelcol-contrib_0.55.0_linux_amd64.deb
In case of different OpenTelemetry collector version, replace 0.55.0
with respective version.
To install otelcol
as systemd
using dpkg
:
sudo dpkg -i otelcol-contrib_0.55.0_linux_amd64.deb
Plain Binary
Using tar.gz
release asset, we can extract the OpenTelemetry collector binary
and default configuration at our desired path. We can run the binary directly
with flags either use `tmux
To download tar.gz
file of release version 0.55.0
:
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.55.0/otelcol-contrib_0.55.0_linux_amd64.tar.gz
In case of different OpenTelemetry collector version, replace 0.55.0
with respective version.
To create otelcol
folder and extract files from tar.gz
to newly created folder:
mkdir otelcol-contrib && tar xvzf otelcol-contrib_0.55.0_linux_amd64.tar.gz -C otelcol-contrib/
OpenTelemetry Collector Configuration
Let's download standalone configuration for otelcol
binary running in the VM:
wget https://raw.githubusercontent.com/SigNoz/benchmark/main/docker/standalone/config.yaml
Replace <IP of machine hosting SigNoz>
with the address to SigNoz in configuration
highlighted below:
exporters:
otlp:
endpoint: "<IP of machine hosting SigNoz>:4317"
tls:
insecure: true
In the configuration above, we enable three receivers: OTLP
, hostmetrics
and prometheus
.
OTLP
receiver is configured to receive all types of telemetry data:
traces, metrics and logs. These data would be forwarded to SigNoz via
OTLP gRPC endpoint.
hostmetrics
receiver is configured to collect various metrics of the virtual
machine. It consists of metrics related to CPU, memory, disk, file system,
network, and others.
prometheus
receiver is configured to collect the internal metrics of the
otelcol
. You can update it as per your need to include additional scrape
targets accessible from the VM or remove existing targets.
OpenTelemetry Collector Usage
You copy the configuration file to the respective config paths as per your
installation methods. Followed by respective instructions to start, restart,
and view logs of the otelcol
binary.
Systemd
To copy the updated config.yaml
file:
sudo cp config.yaml /etc/otelcol-contrib/config.yaml
To restart otelcol
with updated config:
sudo systemctl restart otelcol-contrib.service
To check status of otelcol
:
sudo systemctl status otelcol-contrib.service
To view logs of otelcol
:
sudo journalctl -u otelcol-contrib.service
To stop of otelcol
:
sudo systemctl stop otelcol-contrib.service
Plain Binary
It is recommended to use the otelcol
binary inside terminal multiplexer
tools like tmux
or screen
, since plain binary usage is ephemeral.
To copy the updated config.yaml
file:
cp config.yaml ./otelcol-contrib/config.yaml
To change directory inside otelcol-contrib
folder:
cd otelcol-contrib
To start otelcol
with updated config:
./otelcol-contrib --config ./config.yaml &> otelcol-output.log & echo "$!" > otel-pid
To view last 50 lines of otelcol
logs:
tail -f -n 50 otelcol-output.log
To stop otelcol
:
kill "$(< otel-pid)"
Test Sending Traces
OpenTelemetry collector binary should be able to forward all types of telemetry data recevied: traces, metrics, and logs, to SigNoz OTLP endpoint via gRPC.
Let's send sample traces to the otelcol
using tracegen
.
To install tracegen binary:
go install github.com/open-telemetry/opentelemetry-collector-contrib/[email protected]
To send trace data using tracegen
, execute the command below:
tracegen -traces 1 -otlp-endpoint localhost:4317 -otlp-insecure
Output should look like this:
...
2022-09-30T14:23:20.439+0545 INFO channelz/funcs.go:340 [core][Channel #1] Channel Connectivity change to READY {"system": "grpc", "grpc_log": true}
2022-09-30T14:23:20.439+0545 INFO tracegen/config.go:105 generation of traces isn't being throttled
2022-09-30T14:23:20.440+0545 INFO tracegen/worker.go:91 traces generated {"worker": 0, "traces": 1}
2022-09-30T14:23:20.440+0545 INFO [email protected]/main.go:98 stop the batch span processor
2022-09-30T14:23:20.742+0545 INFO channelz/funcs.go:340 [core][Channel #1] Channel Connectivity change to SHUTDOWN {"system": "grpc", "grpc_log": true}
2022-09-30T14:23:20.742+0545 INFO channelz/funcs.go:340 [core][Channel #1 SubChannel #2] Subchannel Connectivity change to SHUTDOWN {"system": "grpc", "grpc_log": true}
2022-09-30T14:23:20.742+0545 INFO channelz/funcs.go:340 [core][Channel #1 SubChannel #2] Subchannel deleted {"system": "grpc", "grpc_log": true}
2022-09-30T14:23:20.742+0545 INFO channelz/funcs.go:340 [core][Channel #1] Channel deleted {"system": "grpc", "grpc_log": true}
2022-09-30T14:23:20.742+0545 INFO [email protected]/main.go:89 stopping the exporter
If the SigNoz endpoint in the configuration is set correctly and accessible,
you should be able to see the traces sent via OpenTelemetry collector in VM
from tracegen
in the SigNoz UI.
HostMetrics Dashboard
In this section, we will generate and import dashboard with VM HostMetrics. It involves two steps: generting dashboard JSON using bash script and importing dashboard JSON in SigNoz UI.
To generate HostMetrics dashboards for the VM:
curl -sL https://github.com/SigNoz/benchmark/raw/main/dashboards/hostmetrics/hostmetrics-import.sh | bash
Output should look similar to the following:
✅ Succesfully generated Host Metrics dashboard: signoz-hostmetrics-one-piece.json
After importing the dashboard JSON, we should see the following dashboard in SigNoz UI:
List of metrics
Virtual Machine Metrics - hostmetrics
- system_network_connections
- system_disk_weighted_io_time
- system_disk_merged
- system_disk_operation_time
- system_disk_pending_operations
- system_disk_io_time
- system_disk_operations
- system_disk_io
- system_filesystem_inodes_usage
- system_filesystem_usage
- system_cpu_time
- system_memory_usage
- system_network_packets
- system_network_dropped
- system_network_io
- system_network_errors
- system_cpu_load_average_5m
- system_cpu_load_average_15m
- system_cpu_load_average_1m