Skip to main content

FluentD to SigNoz

If you use fluentD to collect logs in your stack with this tutotrial you will be able to send logs from fluentD to SigNoz.

At SigNoz we use opentelemetry collector to recieve logs which supports the fluentforward protocol. So you can forward your logs from your fluentD agent to opentelemetry collector.

Collect Logs Using FluentD in SigNoz cloud

  • Add otel collector binary to your VM by following this guide.

  • Add fluentforward reciever to your config.yaml

    receivers:
    fluentforward:
    endpoint: 0.0.0.0:24224

    Here we have used port 24224 for listening in fluentforward protocol, but you can change it to a port you want. You can read more about fluentforward receiver here.

  • Modify your config.yaml and add the above receiver

    service:
    ....
    logs:
    receivers: [otlp, fluentforward]
    processors: [batch]
    exporters: [otlp]
  • Add the following to your fluentD config to forward the logs to otel collector.

    <match <directive>>
    @type forward
    send_timeout 60s
    recover_wait 10s
    hard_timeout 60s

    <server>
    name myserver1
    host localhost
    port 24224
    </server>
    </match>

    In this config we are matching a directive and forwarding logs to the otel collector which is listening on port 24224. Replace <directive> with your directive name. Also we are assuming that you are running the fluentD binary on the host. If not, the value of host might change depending on your environment.

  • Once you make this changes you can restart fluentD and otel-binary, and you will be able to see the logs in SigNoz.

  • To properly transform your existing log model into opentelemetry log model you can use the different processors provided by opentelemetry. link

    eg:-

    processors:
    logstransform:
    operators:
    - type: trace_parser
    trace_id:
    parse_from: attributes.trace_id
    span_id:
    parse_from: attributes.span_id
    - type: remove
    field: attributes.trace_id
    - type: remove
    field: attributes.span_id

    The operations in the above processor will parse the trace_id and span_id from log to opentelemetry log model and remove them from attributes.

Collect Logs Using FluentD in Self-Hosted SigNoz

Steps to recieve logs from FluentD:

  • Add fluentforward reciever to your otel-collector-config.yaml which is present inside deploy/docker/clickhouse-setup

    receivers:
    fluentforward:
    endpoint: 0.0.0.0:24224

    Here we have used port 24224 for listening in fluentforward protocol, but you can change it to a port you want. You can read more about fluentforward receiver here.

  • Uncomment the exporter and pipleline for logs and make the following change in otel-collector-config.yaml

    exporters:
    ...

    clickhouselogsexporter:
    dsn: tcp://clickhouse:9000/
    timeout: 5s
    sending_queue:
    queue_size: 100
    retry_on_failure:
    enabled: true
    initial_interval: 5s
    max_interval: 30s
    max_elapsed_time: 300s

    ...

    service:
    ...

    logs:
    receivers: [ otlp, fluentforward ]
    processors: [ batch ]
    exporters: [ clickhouselogsexporter ]

    Here we are adding our clickhouse exporter and creating a pipeline which will collect logs from fluentforward receiver, processing it using batch processor and export it to clickhouse.

  • Expose the port in port for otel-collector in docker-compose.yaml file present in deploy/docker/clickhouse-setup

    otel-collector:
    ...
    ports:
    - "24224:24224"
  • Change the fluentD config to forward the logs to otel collector.

    <source>
    @type sample
    sample [{"message": "my log data", "source": "myhost"}, {"message": "my log data 1", "source": "myhost1"}]
    tag sample
    rate 10000
    </source>

    <match sample>
    @type forward
    send_timeout 60s
    recover_wait 10s
    hard_timeout 60s

    <server>
    name myserver1
    host <otel-collector-host>
    port 24224
    </server>
    </match>

    In this example we are generating sample logs and then forwarding them to the otel collector which is listening on port 24224. <otel-collector-host> has to be replaced by the host where otel-collector is running. For more info check troubleshooting.

  • Once you make this changes you can restart fluentD and SignNoz, and you will be able to see the logs in SigNoz.

  • To properly transform your existing log model into opentelemetry log model you can use the different processors provided by opentelemetry. link

    eg:-

    processors:
    logstransform:
    operators:
    - type: trace_parser
    trace_id:
    parse_from: attributes.trace_id
    span_id:
    parse_from: attributes.span_id
    - type: remove
    field: attributes.trace_id
    - type: remove
    field: attributes.span_id

    The operations in the above processor will parse the trace_id and span_id from log to opentelemetry log model and remove them from attributes.