Manual Instrumentation for Traces - OpenTelemetry NodeJS

In this article, we will explore how to add custom spans using OpenTelemetry to enhance observability in our Order service. Specifically, we'll focus on the validate-order process. Manual spans allow us to track specific business logic and identify performance bottlenecks that automatic instrumentation might miss.

Prerequisites

Before we begin, ensure you have the sample app set up from the previous tutorials. You'll need the following OpenTelemetry libraries in your order service:

npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/resources @opentelemetry/semantic-conventions

Acquiring a Tracer

We need to acquire a tracer to create spans. Add the following to your order-service code:

import express, { json } from 'express';
import fetch from 'node-fetch';
import mongoose from 'mongoose';
const { connect, Schema, model } = mongoose;

// add the following
import { trace, SpanStatusCode} from '@opentelemetry/api';
const tracer = trace.getTracer('order-service');

const app = express();
const port = 3001;
...

Adding Custom Spans in the Validate-Order Process

We'll demonstrate how to add a custom span to the validate-order function, which ensures that all products in an order are available in the required quantity.

First, extract the validateOrder function in your order-service:

async function validateOrder(order) {
    // Start a new span for the validation process
    return tracer.startActiveSpan('validate-order', async (span) => {
      try {
        // Add an event indicating the start of validation
        span.addEvent('Order validation started');
  
        // Set attributes to provide more context
        span.setAttribute('order.id', order._id.toString());
        
        let total = 0;
  
        // Validate each product in the order
        for (const item of order.products) {
          // Fetch product details from the Product service
          const productResponse = await fetch(`http://product:3003/products/${item.productId}`);
          const product = await productResponse.json();
  
          // Check product availability
          if (!product || product.stock < item.quantity) {
            throw new Error(`Product ${item.productId} is out of stock or does not exist.`);
          }
  
          // Decrement product stock via the Product service
          const updateResponse = await fetch(`http://product:3003/products/${item.productId}/decrement-stock`, {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({ decrementBy: item.quantity })
          });
  
          // Check if stock update was successful
          if (!updateResponse.ok) {
            throw new Error(`Failed to update stock for Product ${item.productId}.`);
          }

          // Calculate the total amount
          total += product.price * item.quantity;
        }
  
        span.setAttribute('order.total', total);

        // Add an event indicating the completion of validation
        span.addEvent('Order validation completed');
        span.setStatus({ code: SpanStatusCode.OK });
      } catch (error) {
        // Record the error and set the span status to error
        span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
        span.recordException(error);
        throw error;
      } finally {
        // End the span
        span.end();
      }
    });
  }

Explanation:

• Start a new span: We start a new span named validate-order using tracer.startActiveSpan.
• Add an event for the start of validation: Using span.addEvent, we add an event to mark the start of the validation process.
• Set attributes: We use span.setAttribute to add attributes to the span, providing more context about the order.
  • We have added id and total as custom attributes.
• Validate each product: For each product in the order, we fetch details from the Product service and check availability. If the product is available, we proceed to decrement the stock.
• Add an event for the completion of validation: Using span.addEvent, we add another event to mark the completion of the validation process.
• Set span status: If the validation completes successfully, we set the span status to OK. If an error occurs, we set the span status to ERROR and record the exception.
• End the span: Finally, we end the span.

Update the POST /orders route to use this function:

app.post('/orders', async (req, res) => {
  try {
    const order = new Order(req.body);

    // Validate the order
    await validateOrder(order);

    // Save the order
    await order.save();
    res.status(201).json(order);
  } catch (error) {
    res.status(500).json({ message: error.message });
  }
});

Running the Application with Docker

Run the services with Docker:

docker-compose up --build

Verifying the Manual Spans

Run your application and check SigNoz for the manually created spans. Verify that attributes and events are correctly logged.

You can see the new span being created in the post request to /orders called validate-order with the added attributes and events.

When to Use Manual Spans

Manual spans are useful in the following scenarios:

• Detailed Business Logic Tracking: For tracing specific business processes and operations.
• Performance Bottlenecks: To identify and analyze performance issues in critical sections of your application.
• Error Handling and Debugging: For tracking and recording detailed error information and exceptions.

Conclusion

• Custom Spans: Provide detailed tracking for specific business logic and performance analysis.
• Error Handling: Enhance error handling and debugging with detailed span information.
• Flexible Instrumentation: Manual spans offer flexibility to instrument any part of your application as needed.
• Improved Observability: Gain deeper insights into your application's performance and behavior with custom spans.

By adding custom spans to the validate-order process, we enhanced our observability, enabling detailed tracking and improved debugging capabilities. Manual instrumentation allows us to gain deeper insights into specific parts of our application, ensuring better performance and reliability.