Logs Query Builder

We recently released an updated logs explorer page and query builder in SigNoz to make experience of our logs product much more intuitive and seamless.

Some of the key features:

1. Advanced filtering based on attributes and auto suggestions for filters
   

   You can now create more complex queries for how you match attributes, and the query field will automatically suggest both attributes and values for your query.

   

   After entering your query hit 'run query' to see a default bar chart and results down below:

   

   To explore additional ways to filter your query, click 'view details' on any log line to get a list of parameters on the log, and click any value to automatically add a {attribute} IN {value} to your query.

   

2. Aggregation options like Group By, ability to specificy aggregation intervals, etc.
   

   The Group By drop down is automatically populated with common attributes, and selecting one, like log level in this example, gets us a few things right away

   • A comparitive bar chart, with a legend (you can enter a format for the legend including explanatory text eg in this case you might enter The log level is {{level}} to make the chart more readable by others)

   • A timeseries comparing the relative volume of results by time

   

3. Plot multiple queries and formulae based on them in the same charts
   

   For comparing two timeseries or values, you can now add multiple queries or formulae on the same chart, which is especially useful if you are comparing similar data across two different services.

4. Modify your query with a click
   

   By clicking on the details of any log line, you can see the attributes for that item. Click any value, and your query will be modified to require that attribute and value.

   

5. Create Dashboards and Alerts in a single click from logs query builder
   

   Directly from results you can add the query to a dashboard and set up an alert. This makes the timeseries view especially useful, as you can now create an alert when a certain event is logged beyond a certain rate.

   

With the alert query you can perform sophisticated comparisons or other math on your measurement from your logs.

Writing JSON Filters In The New Logs Explorer

In the new logs explorer, you can query your JSON data present inside the body. JSON data in the body will be rendered like this:

You can click on one of the keys and then filter them out

If you want to write the query on your own in the filter bar then you can follow the following rules.

• For json query inside body it starts with a prefix body.
• To access a value of a key you can use the notation body.key_name
• If it is nested key then use . to signify nested keys like body.level1.level2.key
• If the type of value is array use [*]
• operators supported for arrays are HAS and NHAS

Example for JSON filter

Lets say we have this example data in your json log body

{
  "message": "hello",
  "request": {
    "services": [
      "service_1",
      "service_2"
    ],
    "service_meta": [
      {
        "name": "service_1",
        "latency": 101,
        "tags": [
          "tag1",
          "tag2"
        ]
      },
      {
        "name": "service_2",
        "latency": 200,
        "tags": [
          "tag1",
          "tag2"
        ]
      }
    ]
  }
}

• logs where value of message is hello

  body.message = hello
  

• logs where value of message is contains he

  body.message CONTAINS he
  

• logs where value of latency of service_1 is >100

  body.request.service_meta[*].latency > 100
  

• logs where tag1 is present in service tags

  body.request.service_meta[*].tags[*] HAS tag1
  

• logs where tag2 is not present in service tags

  body.request.service_meta[*].tags[*] NHAS tag2
  

Logs Query Builder in old Logs Explorer

This section will walk you through the query language that is used by SigNoz for filtering logs in the old logs explorer.

This query language for logs is a simplified version of SQL. The current state of the query language is good enough for daily uses. As of now we don't support nesting and parenthesis for explicit ordering due to added complexity.

If you have a use case which you are not able to fullfill with the current features please reach out to us on our slack community or Github issues. We plan to improve the query as we go forward while keeping it simple

Types of queries supported by SigNoz:

• Freehand query
  When a user writes a plan text query without using any kind of operators, the query is directly searched against the log body. ( inefficient over huge log data)

  eg:-Dispatch Successful

  

• Filtering queries
  When a user writes queries using a key, operator and a value separated by and , or operators it is a filtering query. This kind of queries are faster as they reduce the search space by using indexes.

  eg:- id IN ('2DCVZOsKHioCeuvbObzVzzL1eZ5') AND fulltext contains 'Dispatch Successful'

  

List of Operators supported by SigNoz

• Here is a list of all the operators that are supported:

  Operator	Multivalued	Examples	Meaning
  IN	yes	num in (1,2,3) strdata in ('a', 'b', 'c')	In
  NIN	yes	num nin (1,2,3) strdata nin ('a', 'b', 'c')	Not In
  GTE	no	num gte 10 dict_word gte 'cat'	Greater than or Equal to
  GT	no	num gt 10 dict_word gt 'cat'	Greater than
  LTE	no	num lte 10 dict_word lte 'cat'	Less than or Equal to
  LT	no	num lt 10 dict_word lt 'cat'	Less than
  CONTAINS	no	stream contains 'err'	Contains
  NCONTAINS	no	stream ncontains 'err'	Doesn't Contain

Fulltext Key

The fulltext key is used when we want to write freehand queries and combine them with filters.

eg:-

id IN ('2DCVZOsKHioCeuvbObzVzzL1eZ5') AND fulltext contains 'Dispatch Successful'

In this cases we are searching Dispatch Successful as fulltext along with the id filter.

Pointers to note while writing queries

• Text always needs to be enclosed in single quotes in filtering queries

  eg:- If you want to search for logs with stream error and which contains Mozilla in body, the corresponding query on the ui will be

  stream IN ('stderr') AND fulltext contains 'Mozilla'
  

  as we can see Mozilla is enclosed in single quotes as well as stderr.

• Order of execution is similar to sql i.e left to right and and has higher precedence over or , but currently SigNoz doesn’t support combining explicitly using parenthesis.

  correct ✅ :-

  stream IN ('stdout') and fulltext contains 'Mozilla' or stream IN ('stderr')
  

  incorrect ❌ :-

  (stream IN ('stdout') and fulltext contains 'Mozilla') or stream IN ('stderr')
  

  While the above to evaluates to the same expression, it’s not necessarily same for the one below

  correct ✅ :-

  stream IN ('stdout') and fulltext contains 'Mozilla' or stream IN ('stderr'
  

  incorrect ❌ :-

  stream IN ('stdout') and (fulltext contains 'Mozilla' or stream IN ('stderr'))
  

Here both the statements are not equivalent of each other i.e it is currently not supported

Query Examples

Here are a few examples of valid and invalid queries:

• Valid Queries

  Query	Description
  OPERATION in ('add') AND FULLTEXT contains 'search string'	fulltext with filtering query
  my fulltext search	fulltext search query
  status gte 200 AND FULLTEXT contains 'search string'	fulltext with filtering query
  service IN ('name>100') AND length GT 100	filtering query
  service IN ('name > 100') AND name GT 'myname'	filtering query
  hello in 2	fulltext search query
  hello in (2,3)	filtering query
  hello lt 2	filtering query

• Invalid Queries

  Query	Description
  OPERATION in ('bcd') AND 'abcd' FULLTEXT contains 'search string'	AND missing between 'abcd' and FULLTEXT
  OPERATION in ('bcd') AND 'search string'	Operator missing before 'search string'