Fields in Logs

A log line contains different attributes attached to it. These attributes help you filter your logs so that you can write efficient queries and get results faster. These attributes are referred to as fields in SigNoz.

All resource and log attributes parsed by the OpenTelemetry Collector are automatically available for querying and are auto-suggested by the query builder.

Configuring the SigNoz Collector

For detailed instructions on configuring the OpenTelemetry Collector, refer to Collector Configuration. You can edit the collector configuration file to filter what logs are being stored after being received by the collector.

Adding Attributes

To add attributes to all logs processed by this collector, add a section to the processors to add an attribute. For example, to tag all logs with a client ID:

attributes/clientid:
    actions:
      - key: client_id
        value: 1123
        action: insert

Adding this mapping alone isn't enough — you also need to add this processor to your pipeline:

logs:
      receivers: [otlp, tcplog/docker]
      processors: [attributes/clientid, logstransform/internal, batch]
      exporters: [clickhouselogsexporter]

Creating Log Fields

By default, when you receive a log from non-OTLP receivers, it is stored directly in the body and you won't be able to filter logs based on fields/attributes. OpenTelemetry provides different ways to parse attributes from your logs using different processors. These parsed attributes are referred to as fields in SigNoz.

For example, if your logs are formatted as:

{"time": "2022-09-20,15:27:17 +0530", "message": "Logging test...", "service": "python"}

You can parse them in your OTel Collector config:

receivers:
    ...
    filelog:
    include: [ /tmp/app.log ]
    start_at: beginning
    operators:
        - type: json_parser
        timestamp:
            parse_from: attributes.time
            layout: '%Y-%m-%d,%H:%M:%S %z'
        - type: move
        from: attributes.message
        to: body
        - type: remove
        field: attributes.time
...

• The JSON parser parses the JSON log line and adds fields to the attributes key.
• The timestamp parser extracts the timestamp from attributes.time.
• The move operator moves the message value to the log body.
• The remove operator removes time from attributes since the timestamp has already been populated.

Transforming Attributes

Logs data can be transformed using the OpenTelemetry Transformation Language (OTTL). Here's a simple example that sets severity based on the log body:

  transform:
    log_statements:
    - context: log
      statements:
        - set(severity_text, "FAIL") where body == "request failed"

When faced with excessively high cardinality data, it may be useful to replace attribute values with generics:

logs:
  replace_match(attributes["http.target"], "/user/*/list/*", "/user/{userId}/list/{listId}")

Removing Sensitive Data from Logs

The collector is one more place where you can control potentially sensitive data from being collected or transmitted. You can remove attributes with simple regex-style matching:

transform:
    log_statements:
    - context: log
      statements:
        - set(severity_text, "FAIL") where body == "request failed"
        - replace_match(attributes["social_security_number"], "*", "{userSocial}")

Or use pattern matching to find similar strings:

  transform:
    log_statements:
    - context: log
      statements:
        - set(severity_text, "FAIL") where body == "request failed"
        - replace_all_patterns(attributes, "value", "^\\D*\\d{3}-\\d{2}-\\d{4}", "{ss_number}")

Remember that in these examples you'll need to add transform to the pipeline section of your config for these changes to take effect.