If you use FluentD to collect logs in your stack, this tutorial will help you send logs from FluentD to SigNoz.
SigNoz uses the OpenTelemetry collector to receive logs, which supports the fluentforward protocol. You can forward your logs from your FluentD agent to the OpenTelemetry collector.
Collect Logs Using FluentD in SigNoz Cloud
Add OpenTelemetry Collector Binary
Follow this guide to add the OpenTelemetry collector binary to your VM.
Configure FluentForward Receiver
Add the following to your
config.yaml:receivers: fluentforward: endpoint: 0.0.0.0:24224
You can change the port if needed. Learn more about the fluentforward receiver here.
Modify the Service Section
Update your
config.yaml:service: ... pipelines: logs: receivers: [otlp, fluentforward] processors: [batch] exporters: [otlp]Update FluentD Configuration
Add the following to your FluentD configuration to forward logs to the OpenTelemetry collector:
<match <directive>> @type forward send_timeout 60s recover_wait 10s hard_timeout 60s <server> name myserver1 host localhost port 24224 </server> </match>Replace
<directive>with your directive name. If FluentD is not running on the host, adjust thehostvalue accordingly.Restart Services
Restart FluentD and the OpenTelemetry collector binary. Logs should now appear in SigNoz.
Transform Logs to OpenTelemetry Model
Use processors in OpenTelemetry to transform logs. Example:
processors: logstransform: operators: - type: trace_parser trace_id: parse_from: attributes.trace_id span_id: parse_from: attributes.span_id - type: remove field: attributes.trace_id - type: remove field: attributes.span_id
Collect Logs Using FluentD in Self-Hosted SigNoz
- Configure FluentForward Receiver Add the following to your
otel-collector-config.yaml(insidedeploy/docker):receivers: fluentforward: endpoint: 0.0.0.0:24224
You can change the port if needed. Learn more about the fluentforward receiver here.
Update Exporter and Pipeline
Uncomment the exporter and pipeline for logs in
otel-collector-config.yamland make the following changes:exporters: ... clickhouselogsexporter: dsn: tcp://clickhouse:9000/ timeout: 5s sending_queue: queue_size: 100 retry_on_failure: enabled: true initial_interval: 5s max_interval: 30s max_elapsed_time: 300s service: ... pipelines: logs: receivers: [otlp, fluentforward] processors: [batch] exporters: [clickhouselogsexporter]Expose Collector Port
Modify
docker-compose-core.yamlto expose the port for the OpenTelemetry collector:otel-collector: ... ports: - "24224:24224"Update FluentD Configuration Update your FluentD configuration:
<source> @type sample sample [{"message": "my log data", "source": "myhost"}, {"message": "my log data 1", "source": "myhost1"}] tag sample rate 10000 </source> <match sample> @type forward send_timeout 60s recover_wait 10s hard_timeout 60s <server> name myserver1 host <otel-collector-host> port 24224 </server> </match>Replace
<otel-collector-host>with the host where the OpenTelemetry collector is running. For troubleshooting, check here.Restart Services
Restart FluentD and SigNoz. Logs should now appear in SigNoz.
Transform Logs to OpenTelemetry Model
Use processors in OpenTelemetry to transform logs. Example:
processors: logstransform: operators: - type: trace_parser trace_id: parse_from: attributes.trace_id span_id: parse_from: attributes.span_id - type: remove field: attributes.trace_id - type: remove field: attributes.span_id