Collecting application Logs from Log file
With SigNoz you can collect your application logs from a log file and push it to SigNoz and perform different actions on your data.
In this tutorail configure OpenTelemetry Collector to read logs from a file and push them to SigNoz.
We will create a sample log file named app.log
and paste the following lines.
This is a log line 1
This is a log line 2
This is a log line 3
Collect Application Logs from Log file in SigNoz cloud
If you don’t already have a SigNoz cloud account, you can sign up here.
- Add otel collector binary to your VM by following this guide.
Add the filelog reciever to
config.yaml
.receivers:
...
filelog/app:
include: [ /tmp/app.log ]
start_at: beginning
...start_at: beginning
can be removed once you are done testing.For parsing logs of different formats you will have to use operators, you can read more about operators here.
For more configurations that are available for filelog receiver please check here.
Next we will modify our pipeline inside
config.yaml
to include the receiver we have created above.service:
....
logs:
receivers: [otlp, filelog/app]
processors: [batch]
exporters: [otlp]Now we can restart the otel collector so that new changes are applied.
The log will be exported, if you add more lines to the log file it will be exported as well
If there are no errors your logs will be visible on SigNoz UI.
Collect Application Logs from Log file in Self-Hosted SigNoz
Steps for collecting logs if SigNoz is running on the same host.
Modify the
docker-compose.yaml
file present insidedeploy/docker/clickhouse-setup
to expose to mount the log file to otel-collector....
otel-collector:
image: signoz/signoz-otel-collector:0.79.5
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ~/<path>/app.log:/tmp/app.log
....Here we are mounting the log file of our application to the
tmp
directory of SigNoz otel-collector. You will have to replace<path>
with the path where your log file is present.Add the filelog reciever to
otel-collector-config.yaml
which is present insidedeploy/docker/clickhouse-setup
receivers:
...
filelog:
include: [ /tmp/app.log ]
start_at: beginning
...start_at: beginning
can be removed once you are done testing.For parsing logs of different formats you will have to use operators, you can read more about operators here
For more configurations that are available for filelog receiver please check here.
Next we will modify our pipeline inside
otel-collector-config.yaml
to include the receiver we have created above.service:
....
logs:
receivers: [otlp, filelog]
processors: [batch]
exporters: [clickhouselogsexporter]Now we can restart the otel collector container so that new changes are applied.
The log will be exported, if you add more lines to the log file it will be exported as well
If there are no errors your logs will be visible on SigNoz UI.
Steps for collecting logs if SigNoz is running on a different host.
If you have a signoz running on a different host then you will have to run a otel-collector to export logs from your host to the host where SigNoz is running.
We will create a
otel-collector-config.yaml
receivers:
filelog:
include: [ /tmp/app.log ]
start_at: beginning
processors:
batch:
send_batch_size: 10000
send_batch_max_size: 11000
timeout: 10s
exporters:
otlp/log:
endpoint: http://<host>:<port>
tls:
insecure: true
service:
pipelines:
logs:
receivers: [filelog]
processors: [batch]
exporters: [ otlp/log ]For parsing logs of different formats you will have to use operators, you can read more about operators here
The parsed logs are batched up using the batch processor and then exported to the host where signoz is deployed. The
otlp/log
exporter here uses a http endpoint but if you want to use https you will have to provide the certificate and the key. You can read more about it hereFor finding the right host and port for your SigNoz cluster please follow the guide here.
We will start our otel-collector container and mount the logs file so that the logs can be read from log file.
docker run -d --name signoz-host-otel-collector --user root -v $(pwd)/app.log:/tmp/app.log:ro -v $(pwd)/otel-collector-config.yaml:/etc/otel/config.yaml signoz/signoz-otel-collector:0.79.0
If there are no errors your logs will be exported and visible on the SigNoz UI.