Overview
This documentation provides detailed instructions on configuring the OpenTelemetry Collector to read logs from a file and push them to SigNoz, enabling you to analyze your application logs effectively.
Sample Log File
As an example, we can create a sample log file called app.log with the following dummy data:
This is log line 1
This is log line 2
This is log line 3
This file represents a log file of your application. You can choose any file that contains your application's log entries.
Setup
1. Install OpenTelemetry Collector
The OpenTelemetry Collector provides a vendor-neutral way to collect, process, and export your telemetry data such as logs, metrics, and traces.
Follow this guide to install the OpenTelemetry Collector as an agent on your Virtual Machine.
2. Configure Filelog Receiver
Modify the config.yaml file created during the installation of the OpenTelemetry Collector to include the filelog receiver. Specify the path to your app.log file and set the start_at parameter.
receivers:
filelog/app:
include: [ /tmp/app.log ] # Include the full path to your log file
start_at: end
The start_at: end configuration ensures that only newly added logs are transmitted. To include historical logs, set start_at to beginning.
3. Update Pipeline Configuration
Update the pipeline settings in config.yaml to include the new filelog receiver:
service:
pipelines:
logs:
receivers: [otlp, filelog/app]
processors: [batch]
exporters: [otlp]
Restart the OpenTelemetry Collector for the changes to take effect. Follow this guide for restart instructions.
4. Verify Export
The logs will be visible in the Logs Tab of SigNoz. As more entires are added to app.log it will reflect Logs tab.
Scenarios
Scenario 1: SigNoz on the Same Host
Install SigNoz
Follow this guide to install self-hosted SigNoz.
Modify Docker Compose File
Edit the docker-compose.yaml file in the deploy/docker directory to mount your application's log file to the tmp directory of the OpenTelemetry Collector.
volumes:
- ~/<path>/app.log:/tmp/app.log
Replace <path> with the actual path of your log file.
Add Filelog Receiver
Update otel-collector-config.yaml with the following configuration:
receivers:
filelog:
include: [ /tmp/app.log ]
start_at: end
The start_at: end configuration ensures only newly added logs are transmitted. To include historical logs, set start_at to beginning.
Update Pipeline Configuration
Modify the pipeline to include the filelog receiver:
service:
pipelines:
logs:
receivers: [otlp, filelog]
processors: [batch]
exporters: [clickhouselogsexporter]
Restart the OpenTelemetry Collector for the changes to take effect.
Scenario 2: SigNoz on a Different Host
Create OTel Collector Configuration
Create an otel-collector-config.yaml file:
receivers:
filelog:
include: [ /tmp/app.log ]
start_at: end
processors:
batch:
send_batch_size: 10000
send_batch_max_size: 11000
timeout: 10s
exporters:
otlp/log:
endpoint: http://<host>:<port>
tls:
insecure: true
service:
pipelines:
logs:
receivers: [filelog]
processors: [batch]
exporters: [otlp/log]
To use https, provide the certificate and key. Refer to this guide for details.
Mount Log File and Run Collector
Run the following command to start the OpenTelemetry Collector in Docker:
docker run -d --name signoz-host-otel-collector --user root -v $(pwd)/app.log:/tmp/app.log:ro -v $(pwd)/otel-collector-config.yaml:/etc/otel/config.yaml signoz/signoz-otel-collector:0.88.11
Adding more entries to app.log will make them visible in SigNoz UI.
Application Deployed on Kubernetes
If your application is running on Kubernetes, you can use the k8s-infra-chart to get your logs directly to SigNoz.
The chart has presets.logsCollection.enabled property to true which adds a filelogreceiver. This receiver is configured to read the files where Kubernetes container runtime writes all containers console output (/var/log/pods///*.log).
Application running on Docker
If you application is running on Docker, you can follow this doc to get you container logs into SigNoz using the filelog receiver
