Collecting syslogs

Collecting Syslogs

With SigNoz you can collect your syslogs logs and perform different queries on top of it. We will demonstrate how to configure rsyslog to forward system logs to tcp endpoint of otel-collector and use syslog receiver in OpenTelemetry Collector to receive and parse the logs. Below are the steps to collect syslogs.

Collect Syslogs in SigNoz cloud

If you don’t already have a SigNoz cloud account, you can sign up here.

• Add otel collector binary to your VM by following this guide.

• Add the syslog reciever to config.yaml to otel-collector.

  receivers:
    syslog:
      tcp:
        listen_address: "0.0.0.0:54527"
      protocol: rfc3164
      location: UTC
      operators:
        - type: move
          from: attributes.message
          to: body
  ...
  

  Here we are collecting the logs and moving message from attributes to body using operators that are available. You can read more about operators here.

  For more configurations that are available for syslog receiver please check here.

• Next we will modify our pipeline inside config.yaml of otel-collector to include the receiver we have created above.

  service:
      ....
      logs:
          receivers: [otlp, syslog]
          processors: [batch]
          exporters: [otlp]
  

• Now we can restart the otel collector so that new changes are applied and we can forward our logs to port 54527.

• Modify your rsyslog.conf file present inside /etc/ by running the following command:

  sudo vim /etc/rsyslog.conf
  

  and adding the this line at the end

  template(
    name="UTCTraditionalForwardFormat"
    type="string"
    string="<%PRI%>%TIMESTAMP:::date-utc% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
  )
  
  *.* action(type="omfwd" target="0.0.0.0" port="54527" protocol="tcp" template="UTCTraditionalForwardFormat")
  

  For production use cases it is recommended to use something like below:

  template(
    name="UTCTraditionalForwardFormat"
    type="string"
    string="<%PRI%>%TIMESTAMP:::date-utc% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
  )
  
  *.*  action(type="omfwd" target="0.0.0.0" port="54527" protocol="tcp"
          action.resumeRetryCount="10"
          queue.type="linkedList" queue.size="10000" template="UTCTraditionalForwardFormat")
  

  So that you have retries and queue in place to de-couple the sending from the other logging action. Also we are assuming that you are running the otel binary on the same host. If not, the value of target might change depending on your environment.

• Now restart your rsyslog service by running sudo systemctl restart rsyslog.service

• You can check the status of service by running sudo systemctl status rsyslog.service

• If there are no errors your logs will be visible on SigNoz UI.

Collect Syslogs in Self-Hosted SigNoz

• Modify the docker-compose.yaml file present inside deploy/docker/clickhouse-setup to expose a port, in this case 54527 so that we can forward syslogs to this port.

  ...
  otel-collector:
      image: signoz/signoz-otel-collector:0.88.11
      command: ["--config=/etc/otel-collector-config.yaml"]
      volumes:
        - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
      ports:
        - "54527:54527"
  ...
  

• Add the syslog reciever to otel-collector-config.yaml which is present inside deploy/docker/clickhouse-setup

  receivers:
    syslog:
      tcp:
        listen_address: "0.0.0.0:54527"
      protocol: rfc3164
      location: UTC
      operators:
        - type: move
          from: attributes.message
          to: body
  ...
  

  Here we are collecting the logs and moving message from attributes to body using operators that are available. You can read more about operators here

  For more configurations that are available for syslog receiver please check here.

• Next we will modify our pipeline inside otel-collector-config.yaml to include the receiver we have created above.

  service:
      ....
      logs:
          receivers: [otlp, syslog]
          processors: [batch]
          exporters: [clickhouselogsexporter]
  

• Now we can restart the otel collector container so that new changes are applied and we can forward our logs to port 54527.

• Modify your rsyslog.conf file present inside /etc/ by running sudo vim /etc/rsyslog.conf and adding the this line at the end

  template(
    name="UTCTraditionalForwardFormat"
    type="string"
    string="<%PRI%>%TIMESTAMP:::date-utc% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
  )
  
  *.* action(type="omfwd" target="0.0.0.0" port="54527" protocol="tcp" template="UTCTraditionalForwardFormat")
  

  For production use cases it is recommended to using something like

  template(
    name="UTCTraditionalForwardFormat"
    type="string"
    string="<%PRI%>%TIMESTAMP:::date-utc% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
  )
  
  *.*  action(type="omfwd" target="0.0.0.0" port="54527" protocol="tcp"
          action.resumeRetryCount="10"
          queue.type="linkedList" queue.size="10000" template="UTCTraditionalForwardFormat")
  

  So that you have retires and queue in place to de-couple the sending from the other logging action.

  The value of target might vary depending on where SigNoz is deployed, since it is deployed on the same host I am using 0.0.0.0 for more help you can visit here

• Now restart your rsyslog service by running sudo systemctl restart rsyslog.service

• You can check the status of service by running sudo systemctl status rsyslog.service

• If there are no errors your logs will be visible on SigNoz UI.