Skip to main content

Implementing Distributed Tracing in a Nodejs application

· 11 min read
Selvaganesh

In this article, we will implement distributed tracing for a nodejs application based on microservices architecture. To implement distributed tracing, we will be using open-source solutions - SigNoz and OpenTelemetry, so you can easily follow the tutorial.

Cover Image

In modern microservices-based applications, it is difficult to understand how requests are performing across multiple services, infrastructure, and protocols. As companies began moving to distributed systems, they realized they needed a way to track requests in their entirety for debugging applications. Distributed tracing is a technology that was born out of this need. Let’s have a brief overview of distributed tracing.

What is distributed tracing?

In a microservices architecture, a user request travels through hundreds, even thousands of services before serving the user what they need. Engineering teams often responsible for maintaining single services have no visibility over how the system performs as a whole.

Read our complete guide on Distributed Tracing

Distributed tracing gives insights into how a particular service is performing as part of the whole in a distributed software system. It involves passing a trace context with each user request which is then passed across hosts, services, and protocols to track the user request.

In this article, we will use OpenTelemetry and SigNoz to enable distributed tracing in a sample nodejs application.

OpenTelemetry and SigNoz

OpenTelemetry is a vendor-agnostic set of tools, APIs, and SDKs used to instrument applications to create and manage telemetry data(Logs, metrics, and traces). It aims to make telemetry data(logs, metrics, and traces) a built-in feature of cloud-native software applications.

OpenTelemetry provides the instrumentation layer to generate and export your telemetry data to a backend. Then, you need to choose a backend tool that will provide the data storage and visualization for your telemetry data. That’s where SigNoz comes into the picture.

SigNoz is a full-stack open-source APM tool that provides metrics monitoring and distributed tracing.

We will demonstrate implementing distributed tracing in a nodejs application in two sections:

  • Running a sample nodejs application with OpenTelemetry
  • Visualizing traces data with SigNoz dashboards

Running a sample nodejs application with OpenTelemetry

The sample nodejs application will have three microservices:

  • user-service
  • orders-service
  • payment-service

Here’s the architecture of the sample application along with OpenTelemetry and SigNoz.

Sample nodejs app application architecture
Application architecture along with SigNoz and OpenTelemetry(OTel Collector)

Pre-requisites

Installing SigNoz

SigNoz can be installed on macOS or Linux computers in just three steps by using a simple install script.

The install script automatically installs Docker Engine on Linux. However, on macOS, you must manually install Docker Engine before running the install script.

git clone -b main https://github.com/SigNoz/signoz.git
cd signoz/deploy/
./install.sh

You can visit our documentation for instructions on how to install SigNoz using Docker Swarm and Helm Charts.

Deployment Docs

When you are done installing SigNoz, you can access the UI at http://localhost:3301

SigNoz dashboard
SigNoz dashboard - It shows services from a sample app that comes bundled with the application

Installing MySql

Download MySQL community version from here based on your operating system.

Once installation is complete, run the below commands to create a database for our sample nodejs app.

~ mysql -u root
mysql> create database signoz;
mysql> use signoz;

If you have MySQL already installed and is configured with a password, you might need to update the password in the sample app here.

Running sample application

Below are the steps to run the sample nodejs application with OpenTelemetry:

  1. Clone sample nodejs app repository and go to the root folder
    We will be using a sample nodejs app at this GitHub repo.
    git clone [email protected].com:SigNoz/distributed-tracing-nodejs-sample.git
    cd distributed-tracing-nodejs-sample
  1. Install the required dependencies
    You can check out the depencies required from package.json file. Install all the required dependencies for the sample application using npm. Also, install typescript and ts-node-dev globally.

    npm install
    npm install -g typescript
    npm install -g ts-node-dev

    OpenTelemetry needs the following packages to instrument the nodejs app.

    "@opentelemetry/api": "^1.0.3",
    "@opentelemetry/auto-instrumentations-node": "^0.25.0",
    "@opentelemetry/exporter-collector": "0.25.0",
    "@opentelemetry/exporter-collector-grpc": "^0.25.0",
    "@opentelemetry/exporter-otlp-grpc": "^0.26.0",
    "@opentelemetry/resources": "^0.24.0",
    "@opentelemetry/sdk-trace-base": "^1.0.1",
    "@opentelemetry/sdk-trace-node": "^1.0.1",
    "@opentelemetry/semantic-conventions": "^0.24.0",
  1. tracer.ts file
    In order to instrument our services, we will create a single tracer.ts file and use it to instrument all three services.

    We need to initialize OpenTelemetry before our application gets loaded. If your application begins requiring packages before OpenTelemetry is set up, it can create issues. You can initialize OpenTelemetry by using the code as shown below:

    import init from './tracer'
    const { sdk } = init('order-service') // provide service name to view in signoz dashboard

    You can check out the code sample here.

  2. Setting up SigNoz as the OpenTelemetry backend
    To set up OpenTelemetry to collect and export telemetry data, you need to specify OTLP (OpenTelemetry Protocol) endpoint. It consists of the IP of the machine where SigNoz is installed and the port number at which SigNoz listens.

    OTLP endpoint for SigNoz - <IP of the machine>:4317

    Here's the code to run a microservice along with the environment variables needed for OpenTelemetry and SigNoz:

    OTEL_EXPORTER_OTLP_ENDPOINT=<IP of the machine>:4317 \
    OTEL_RESOURCE_ATTRIBUTES=service.name=UserService \
    ts-node-dev -r ./src/tracer.ts ./src/user-service.ts

    If you have installed SigNoz on your local machine, then your endpoint is 127.0.0.1:4317.

    If you have installed SigNoz on some domain, then your endpoint is http://test.com:4317

  3. Run the microservices
    From the root folder of your application on your terminal, run each microservice. As we have installed SigNoz on the local host, we will be using its ip. Run users service:

    OTEL_EXPORTER_OTLP_ENDPOINT=127.0.0.1:4317 \
    OTEL_RESOURCE_ATTRIBUTES=service.name=UserService \
    ts-node-dev -r ./src/tracer.ts ./src/user-service.ts

    Open a new tab of your terminal, and run payment service:

    OTEL_EXPORTER_OTLP_ENDPOINT=127.0.0.1:4317 \
    OTEL_RESOURCE_ATTRIBUTES=service.name=PaymentService \
    ts-node-dev -r ./src/tracer.ts ./src/payment-service.ts

    Open a new tab of your terminal, and run orders service:

    OTEL_EXPORTER_OTLP_ENDPOINT=127.0.0.1:4317 \
    OTEL_RESOURCE_ATTRIBUTES=service.name=OrderService \
    ts-node-dev -r ./src/tracer.ts ./src/order-service.ts

    Ensure that the microservices are running on different ports. As earlier mentioned, you can set ports using the .env file. Your microservices ports are setup in the .env file as shown below:

    ORDER_PORT=8082
    USERS_PORT=8081
    PAYMENT_PORT=8080

    You can check the file in the cloned repo.

    Screenshot of terminal showing mircroservices running on different ports
    Running microservices on different ports
  4. Confirm table creation
    After running the services, check if the tables ORDERS and USERS are created using the commands below:

    mysql> use signoz;
    mysql> show tables;
    Checking creation of tables
    Checking creation of tables after running microservices

Visualizing traces data with SigNoz dashboards

To visualize the traces data with SigNoz, we first need to generate some user data by interacting with the sample nodejs application.

Generating user data by interacting with the sample app

You need to generate some user data to see how it appears in the SigNoz dashboard. The sample application comes with an UI to interact with the app. Use the below command in the root folder to launch the UI:

npm install -g serve
serve -l 5000 ui

This will run the app UI on port 5000. Open the UI at http://localhost:5000/.

Running sample app UI
Running sample app UI
  1. Create a new user
    Click Create User button to create a new user in the MySQL db.

    Create new user
    Create new user
  2. Transfer some amount
    Transfer some amount by clicking Transfer Fund button.

    Transfer some fund
    Transfer some fund
  3. Place an order
    Place an order by selecting a product from dropdown.

    Place an order
    Place an order

Now go to SigNoz dashboard, wait for sometime and refresh the dashboard. You will notice the list of service names that we configured:

  • user-service
  • order-service
  • payment-service
Sample app microservices monitored on the SigNoz dashboard
Microservices in our nodejs app being monitored in the SigNoz dashboard

You can play around with the dashboard to see what data is captured. Below is a handy guide on how to use the SigNoz dashboard to see the captured data.

Capturing MySQL traces

To view MySQL traces add below opentelemetry package:

npm install @opentelemetry/instrumentation-mysql2

Now, make the following changes in your tracer.ts file:

const { MySQL2Instrumentation } = require('@opentelemetry/instrumentation-mysql2')

.....
const sdk = new opentelemetry.NodeSDK({
traceExporter,
instrumentations: [
getNodeAutoInstrumentations(),
new MySQL2Instrumentation()
],
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: serviceName,
}),
})
.....

Restart the microservices, and generate new usage data in order to see traces related to MySQL events.

How to use SigNoz dashboard to analyze traces

The traces tab of SigNoz dashboard provides powerful filters to analyze the traces data. You can use a number of filters to see traces data across many dimensions.

  1. See the count of requests by service and HTTP Status code
Count of requests in traces filter tab
Count of requests by service and HTTP status code
  1. Identify latency issues with Flamegraphs and Gantt charts
    You can inspect each event in the table with flamegraphs and Gantt charts to see a complete breakdown of the request. Establishing a sequential flow of the user request along with info on time taken by each part of the request can help identify latency issues quickly. Let’s see how it works in case of our sample nodejs app.

    Go to operation filter on the left navigation apply two filters GET /payment/transfer/:id and service name payment-service . Click on the single event listed in the table as shown below:

    Events table in traces filter tab
    Use filters to inspect events that you want to investigate further.

    You will be able to see the flamegraph of the selected event which shows how the request traveled between the payment and the user-service. You can also use the Gantt chart to analyze each event in detail.

Flamegraph and Gantt chart for the selected event
Flamegraph and Gantt chart for the selected event
SigNoz also provides a detailed view of common semantic conventions like HTTP, network, and other attributes. The end-to-end tracing of user requests can help you to identify latency issues quickly.

Conclusion

Distributed tracing is a powerful and critical toolkit for developers creating applications based on microservices architecture. For nodejs applications based on microservices architecture, distributed tracing can enable a central overview of how requests are performing across services which is needed for quick debugging.

OpenTelemetry and SigNoz provide a great open-source combo to implement distributed tracing for your nodejs applications. You can check out SigNoz by visiting its GitHub repo 👇

SigNoz GitHub repo

If you are someone who understands more from video, then you can watch the below video tutorial on the same with SigNoz.

 

YouTube's thumbnail image for the video.

 

If you have any questions or need any help in setting things up, join our slack community and ping us in #support channel.

SigNoz Slack community


Read more about distributed tracing from SigNoz blog 👇

Spans - a key concept of distributed tracing

Context Propagation in distributed tracing