Send Fluent Bit Logs to SigNoz from VMs and Kubernetes

Step 1. Add the fluentforward receiver to the Collector

Add the fluentforward receiver to the receivers section of the Collector configuration (/etc/otelcol-contrib/config.yaml). Then add it to the logs pipeline:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  # On Collector v0.151.0 and newer, use "fluent_forward" to avoid a deprecation warning.
  fluentforward:
    endpoint: 0.0.0.0:24224
service:
  pipelines:
    logs:
      receivers: [otlp, fluentforward]
      processors: [batch, resourcedetection]
      exporters: [otlphttp]

The receiver listens on port 24224 for the Fluent Forward protocol. Keep the other sections of the file from the install guide.

Restart the Collector:

sudo systemctl restart otelcol-contrib

The path and the command apply to the DEB and RPM packages. For macOS and Windows, use the configuration path and the restart steps in the install guide.

Step 2. Add a forward output to Fluent Bit

Add a forward output to the Fluent Bit configuration (fluent-bit.yaml):

pipeline:
  outputs:
    - name: forward
      match: '*'
      host: <collector-host>
      port: 24224
      retain_metadata_in_forward_mode: false

Verify these values:

• <collector-host>: The host where the Collector runs. Use 127.0.0.1 when Fluent Bit runs on the same host.

Set retain_metadata_in_forward_mode to false. Fluent Bit v5.0.4 and newer set it to true by default, and the fluentforward receiver cannot read that format. The receiver then drops every log, and Fluent Bit still reports a successful flush.

Restart Fluent Bit so that it loads the new output.

The Fluent Bit Helm chart runs Fluent Bit as a DaemonSet. Fluent Bit reads the container log files on each node, and its kubernetes filter adds the pod metadata to each log as one nested kubernetes map. This step sends the logs to the k8s-infra OpenTelemetry Collector agent, which splits that map into k8s.* resource attributes.

Step 1. Add the receiver and the mapping to k8s-infra

Add these values to the override-values.yaml file of your k8s-infra release:

presets:
  logsCollection:
    enabled: false
otelAgent:
  ports:
    fluentforward:
      enabled: true
      containerPort: 24224
      servicePort: 24224
      protocol: TCP
  config:
    receivers:
      fluentforward:
        endpoint: 0.0.0.0:24224
    processors:
      transform/fluentbit_k8s:
        error_mode: ignore
        log_statements:
          - set(log.attributes["k8s.pod.name"], log.attributes["kubernetes"]["pod_name"]) where log.attributes["kubernetes"] != nil
          - set(log.attributes["k8s.namespace.name"], log.attributes["kubernetes"]["namespace_name"]) where log.attributes["kubernetes"] != nil
          - set(log.attributes["k8s.pod.uid"], log.attributes["kubernetes"]["pod_id"]) where log.attributes["kubernetes"] != nil
          - set(log.attributes["k8s.container.name"], log.attributes["kubernetes"]["container_name"]) where log.attributes["kubernetes"] != nil
          - set(log.attributes["k8s.node.name"], log.attributes["kubernetes"]["host"]) where log.attributes["kubernetes"] != nil
          - delete_key(log.attributes, "kubernetes")
      groupbyattrs/fluentbit_k8s:
        keys: [k8s.pod.name, k8s.namespace.name, k8s.pod.uid, k8s.container.name, k8s.node.name]
    service:
      pipelines:
        logs:
          receivers: [otlp, fluentforward]
          processors: [transform/fluentbit_k8s, groupbyattrs/fluentbit_k8s, batch]

These values do the following:

• presets.logsCollection.enabled: false stops the k8s-infra agent from reading the container log files. Fluent Bit reads them, so without this line each log arrives twice.
• otelAgent.ports.fluentforward opens port 24224 on the agent pods and on the agent Service.
• The transform/fluentbit_k8s processor copies the fields of the kubernetes map to k8s.* attributes, then deletes the map.
• The groupbyattrs/fluentbit_k8s processor moves those attributes to the resource. SigNoz then shows the logs under the correct pod, namespace, and container.

The chart adds its own processors before these, so you only list the new ones.

Apply the values:

helm upgrade <release-name> signoz/k8s-infra -n <namespace> -f override-values.yaml

Verify these values:

• <release-name>: The name of your k8s-infra release.
• <namespace>: The namespace of your k8s-infra release.

Step 2. Install Fluent Bit with a forward output

Save these values as fluent-bit-values.yaml:

config:
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Exclude_Path /var/log/containers/*-otel-agent-*_<namespace>_*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
 
  outputs: |
    [OUTPUT]
        Name forward
        Match kube.*
        Host <release-name>-k8s-infra-otel-agent.<namespace>.svc.cluster.local
        Port 24224
        Retain_Metadata_In_Forward_Mode false

Verify these values:

• <release-name>: The name of your k8s-infra release. If the release name already contains k8s-infra, the Service is <release-name>-otel-agent. Run kubectl get svc -n <namespace> to see the name.
• <namespace>: The namespace of your k8s-infra release. The Exclude_Path line stops Fluent Bit from reading the logs of the agent pods. Other pods in that namespace still send their logs.

The chart keeps its default kubernetes filter, which adds the pod metadata. Set Retain_Metadata_In_Forward_Mode to false. Fluent Bit v5.0.4 and newer set it to true by default, and the fluentforward receiver drops logs in that format.

Install the chart:

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
helm install fluent-bit fluent/fluent-bit -n logging --create-namespace -f fluent-bit-values.yaml

If you already run the Fluent Bit chart, add the forward output and the Exclude_Path line to your values file instead, then upgrade that release:

helm upgrade <fluent-bit-release> fluent/fluent-bit -n <fluent-bit-namespace> -f <your-values-file>