NGINX Tutorial: How to Use OpenTelemetry Tracing to Understand Your Microservices

ul.no-bullets {
list-style-type: none;
}

This post is one of four tutorials that help you put into practice concepts from Microservices March 2023: Start Delivering Microservices:

How to Deploy and Configure Microservices

How to Securely Manage Secrets in Containers

How to Use GitHub Actions to Automate Microservices Canary Deployments

How to Use OpenTelemetry Tracing to Understand Your Microservices (this post)

A microservices architecture comes with many benefits, including increased team autonomy and increased flexibility in scaling and deployment. On the downside, the more services in a system (and a microservices app can have dozens or even hundreds), the more difficult it becomes to maintain a clear picture of the overall operation of the system. As writers and maintainers of complex software systems, we know the vital importance of having that clear picture. Observability tooling gives us the power to build that picture across numerous services and supporting infrastructure.

In this tutorial, we highlight one very important type of observability for microservices apps: tracing. Before getting started, let’s define some terms commonly used when discussing observability:

Observability – The ability to understand the internal state or condition of a complex system (such as a microservices app) based only on knowledge of its external outputs (such as traces, logs, and metrics).
Monitoring – The ability to observe and check the progress or status of an object over a period of time. For example, you can monitor the traffic coming to your app during peak times, and use that information to scale it appropriately in response.
Telemetry – The act of gathering metrics, traces, and logs and transferring them from their point of origin to another system for storage and analysis. Also, the data itself.
Tracing/traces – An account of the journey of a request or an action as it moves through all the nodes of a distributed system.
Span – A record within a trace of an operation and its associated metadata. Traces are made up of many nested spans.
Event logging/logs – A time-stamped text record with metadata.
Metric – A measurement captured at runtime. For example, the amount of memory being used by an application at a certain point in time.

We can use all these concepts to attain insight into the performance of our microservices. Tracing is a particularly useful part of an observability strategy because traces offer a “big picture” of what’s happening across multiple, often loosely coupled, components when a request is made. It is also a particularly effective way to identify performance bottlenecks.

This tutorial uses the tracing tool kit from OpenTelemetry (OTel), an open source vendor-neutral standard for collecting, processing, and exporting telemetry that is rapidly gaining in popularity. In OTel’s conception, a trace slices up a data flow, which may involve multiple services, into a series of chronologically arranged “chunks” that can help you easily understand:

All the steps that happened in a “chunk”
How long all these steps took
Metadata about each step

If you’re unfamiliar with OTel, see What Is OpenTelemetry? for a thorough introduction into the standard and considerations for implementing it.

Tutorial Overview

This tutorial focuses on tracing the operations of a microservices app with OTel. In the four challenges in this tutorial, you learn how to track a request through your system and answer questions about your microservices:

Set up basic OTel instrumentation
Set up OTel instrumentation and trace visualization for all services
Learn to read OTel traces
Optimize instrumentation based on trace readings

These challenges illustrate our recommended process when setting up tracing for the first time. The steps are:

Understand the system as well as the particular operation you are instrumenting.
Decide what you need to know from the running system.
Instrument the system “naively” – meaning use a default configuration without trying to weed out information you don’t need or gather custom data points – and evaluate whether the instrumentation helps you answer your questions.
Tweak the information that is reported to allow you to more quickly answer those questions.

Note: Our intention in this tutorial is to illustrate some core concepts about telemetry, not to show the right way to deploy microservices in production. While it uses a real “microservices” architecture, there are some important caveats:

The tutorial does not use a container orchestration framework such as Kubernetes or Nomad. This is so that you can learn about microservices concepts without getting bogged down in the specifics of a certain framework. The patterns introduced here are portable to a system running one of these frameworks.
The services are optimized for ease of understanding rather than software engineering rigor. The point is to look at a service’s role in the system and its patterns of communication, not the specifics of the code. For more information, see the README files of the individual services.

Tutorial Architecture and Telemetry Goals

Architecture and User Flow

This diagram illustrates the overall architecture and flow of data among the microservices and other elements used in the tutorial.

The two microservices are:

The messenger service – A simple chat API with message storage capabilities
The notifier service – A listener that triggers events to alert users, based on their preferences

Three pieces of supporting infrastructure are:

NGINX Open Source – An entry point to the messenger service and the system at large
RabbitMQ – A popular open source message broker that enables services to communicate asynchronously
Jaeger – An open source end-to-end distributed tracing system for collecting and visualizing telemetry from components of the system that produce it

Taking OTel out of the picture for a moment, we can concentrate of the sequence of events that we’re tracing: what happens when a user sends a new chat message and the recipient is notified about it.

The flow breaks down like this:

A user sends a message to the messenger service. The NGINX reverse proxy intercepts the message and forwards it to one of many instances of the messenger service.
The messenger service writes the new message to its database.
The messenger service produces an event on a RabbitMQ message queue called chat_queue to indicate that a message was sent. The event is generic and has no specific target.
At the same time:
- 4a. The messenger service returns a response to the sender reporting that the message was sent successfully.
- 4b. The notifier service notices the new event on the chat_queue and consumes it.
The notifier service checks its database for the notification preferences of the recipient of the new message.
The notifier service uses the recipient’s preferred method to send one or many notifications (in this tutorial, the choices of method are SMS and email).

Telemetry Goals

When setting up telemetry instrumentation, it’s best to start with a set of goals for instrumentation more defined than “send everything and hope for insights”. We have three key telemetry goals for this tutorial:

Understand all the steps a request goes through during the new message flow
Have confidence that the flow is executing end-to-end within five seconds under normal conditions
See how long it takes the notifier service to begin processing the event dispatched by the messenger service (excessive delay might mean the notifier service is having trouble reading from the event queue and events are backing up)

Notice that these goals are related to both the technical operation of the system and the user experience.

Tutorial Prerequisites and Set Up

Prerequisites

To complete the tutorial in your own environment, you need:

A Linux/Unix‑compatible environment
Basic familiarity with the Linux command line, JavaScript, and bash (but all code and commands are provided and explained, so you can still succeed with limited knowledge)
Docker and Docker Compose
Node.js 19.x or later
- We tested version 19.x, but expect that newer versions of Node.js also work.
- For detailed information about installing Node,js, see the README in the messenger service repository. You can also install asdf to get exactly the same Node.js version used in the tutorial.
curl (already installed on most systems)
The technologies listed in Architecture and User Flow: messenger and notifier (you’ll download them in the next section), NGINX Open Source, Jaeger, and RabbitMQ.

Note: The tutorial uses the JavaScript SDK because the messenger and notifier services are written in Node.js. You also set up the OTel automatic instrumentation feature (also called auto‑instrumentation) so you can get a feel for the type of information available from OTel. The tutorial explains everything you need to know about the OTel Node.js SDK, but for more details, see the OTel documentation.

Set Up

Start a terminal session.
In your home directory, create the microservices-march directory and clone the GitHub repositories for this tutorial into it. (You can also use a different directory name and adapt the instructions accordingly.)

Note: Throughout the tutorial the prompt on the Linux command line is omitted, to make it easier to copy and paste the commands into your terminal. The tilde (~) represents your home directory.
```
mkdir ~/microservices-march
cd ~/microservices-march
git clone https://github.com/microservices-march/messenger --branch mm23-metrics-start
git clone https://github.com/microservices-march/notifier --branch mm23-metrics-start
git clone https://github.com/microservices-march/platform --branch mm23-metrics-start
```

Challenge 1: Set Up Basic OTel Instrumentation

In this challenge you start the messenger service and configure OTel auto‑instrumentation to send telemetry to the console.

Launch the messenger Service

Change to the platform repository and start Docker Compose:
```
cd ~/microservices-march/platform
docker compose up -d --build
```
This starts RabbitMQ and Jaeger, which will be used in subsequent challenges.
- The ‑d flag instructs Docker Compose to detach from containers when they have started (otherwise the containers will remain attached to your terminal).
- The --build flag instructs Docker Compose to rebuild all images on launch. This ensures that the images you are running stay updated through any potential changes to files.
Change to the app directory in the messenger repository and install Node.js (you can substitute a different method if you wish):
```
cd ~/microservices-march/messenger/app
asdf install
```
Install dependencies:
```
npm install
```
Start the PostgreSQL database for the messenger service:
```
docker compose up -d
```
Create the database schema and tables and insert some seed data:
```
npm run refresh-db
```

Configure OTel Auto-Instrumentation Sent to the Console

With OTel auto‑instrumentation, you don’t need to modify anything in the messenger codebase to set up tracing. Instead of being imported into the application code itself, all tracing configuration is defined in a script that is imported into the Node.js process at runtime.

Here you configure auto‑instrumentation of the messenger service with the most basic destination for traces, the console. In Challenge 2, you’ll change the configuration to send traces to Jaeger as an external collector.

Still working in the app directory of the messenger repo, install the core OTel Node.js packages:
```
npm install @opentelemetry/[email protected] 
            @opentelemetry/[email protected]
```
These libraries provide the following functionality:
- @opentelemetry/sdk-node – Generation and export of OTel data
- @opentelemetry/auto-instrumentations-node – Automatic setup with default configuration of all the most common Node.js instrumentations
Note: It is a quirk of OTel that its JavaScript SDKs are broken up into very, very small pieces. So you will be installing a few more packages just for the basic example in this tutorial. To understand which packages you might need to accomplish instrumentation tasks beyond those covered in this tutorial, peruse the (very good) OTel getting started guides and look through the OTel GitHub repository.
Create a new file called tracing.mjs to contain the setup and configuration code for OTel tracing:
```
touch tracing.mjs
```
In your preferred text editor, open tracing.mjs and add this code:
```
//1
import opentelemetry from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";

//2
const sdk = new opentelemetry.NodeSDK({
  traceExporter: new opentelemetry.tracing.ConsoleSpanExporter(),
  instrumentations: [getNodeAutoInstrumentations()],
});

//3
sdk.start();
```
The code does the following:
1. Imports the required functions and objects from the OTel SDK.
2. Creates a new instance of the NodeSDK and configures it to:
  - Send spans to the console (ConsoleSpanExporter).
  - Use the auto‑instrumenter as the base set of instrumentation. This instrumentation loads up all the most common auto‑instrumentation libraries. In the tutorial the relevant ones are:
    - @opentelemetry/instrumentation-pg for the Postgres database library (pg)
    - @opentelemetry/instrumentation-express for the Node.js Express framework
    - @opentelemetry/instrumentation-amqplib for the RabbitMQ library (amqplib)
3. Starts the SDK.

Start the messenger service, importing the auto‑instrumentation script you created in Step 3.

node --import ./tracing.mjs index.mjs

After a moment, a lot of output related to tracing starts appearing in the console (your terminal):

...
{
  traceId: '9c1801593a9d3b773e5cbd314a8ea89c',
  parentId: undefined,
  traceState: undefined,
  name: 'fs statSync',
  id: '2ddf082c1d609fbe',
  kind: 0,
  timestamp: 1676076410782000,
  duration: 3,
  attributes: {},
  status: { code: 0 },
  events: [],
  links: []
}
...

Note: Leave the terminal session open for reuse in Challenge 2.

Challenge 2: Set Up OTel Instrumentation and Trace Visualization for All Services

There are many tools that you can use to view and analyze traces, but this tutorial uses Jaeger. Jaeger is a simple, open source end-to-end distributed tracing framework with a built-in web-based user interface for viewing spans and other tracing data. The infrastructure provided in the platform repository includes Jaeger (you started it in Step 1 of Challenge 1), so you can focus on analyzing data instead of dealing with complex tooling.

Jaeger is accessible at the http://localhost:16686 endpoint in your browser, but if you access the endpoint right now, there’s nothing to see about your system. That’s because the traces you’re currently collecting are being sent to the console! To view trace data in Jaeger, you need to export the traces using the OpenTelemetry protocol (OTLP) format.

In this challenge you instrument the core user flow by configuring instrumentation for:

The messenger service, switching the destination for traces from the console to Jaeger
The notifier service
NGINX

Configure OTel Auto-Instrumentation Sent to an External Collector

As a reminder, using OTel auto‑instrumentation means you don’t modify anything in the messenger codebase to set up tracing. Instead, all tracing configuration is in a script that is imported into the Node.js process at runtime. Here you change the destination for traces generated by the messenger service from the console to an external collector (Jaeger in this tutorial).

Still working in the same terminal as in Challenge 1, and in the app directory of the messenger repo, install the OTLP exporter Node.js package:
```
npm install @opentelemetry/[email protected]
```
The @opentelemetry/exporter-trace-otlp-http library exports trace information in OTLP format via HTTP. It’s used when sending telemetry to an OTel external collector.
Open tracing.mjs (which you created and edited in Challenge 1) and make these changes:
- Add this line to the set of import statements at the top of the file:
```
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
```
- Change the “exporter” that you provide to the OTel SDK from the console exporter used in Challenge 1 to one that can send OTLP data over HTTP to an OTLP-compatible collector. Replace:
```
traceExporter:new opentelemetry.tracing.ConsoleSpanExporter(),
```
  with:
```
traceExporter: new OTLPTraceExporter({ headers: {} }),
```
Note: For simplicity’s sake, the tutorial assumes the collector lives at the default location, http://localhost:4318/v1/traces. In a real system, it’s a good idea to set the location explicitly.
Press Ctrl+c to stop the messenger service, which you started in this terminal in Step 4 of Configure OTel Auto-Instrumentation Sent to the Console. Then restart it to use the new exporter configured in Step 2:
```
^c
node --import ./tracing.mjs index.mjs
```
Start a second, separate terminal session. (Subsequent instructions call this the client terminal and the original terminal – used in Steps 1 and 3 – the messenger terminal.) Wait about ten seconds, then send a health‑check request to the messenger service (you can run this a few times if you want to see multiple traces):
```
curl -X GET http://localhost:4000/health
```
Waiting ten seconds before sending the request helps make your trace easier to find, because it comes after the many traces that the auto‑instrumentation generates as the service starts.
In a browser, access the Jaeger UI at http://localhost:16686 and verify the OTLP exporter is working as expected. Click Search in the title bar and from the drop‑down menu in the Service field select the service whose name starts with unknown_service. Click the Find Traces button:
Click a trace in the right side of the window to display a list of the spans in it. Each span describes the operations, sometimes involving multiple services, that ran as part of the trace. The jsonParser span in the screenshot shows how long it took to run the jsonParser portion of the messenger service’s request‑handling code.
As noted in the Step 5, the name of the service as exported by the OTel SDK (unknown_service) is not meaningful. To fix this, in the messenger terminal press Ctrl+c to stop the messenger service. Then install a couple more Node.js packages:
```
^c 
npm install @opentelemetry/[email protected] 
            @opentelemetry/[email protected]
```
These two libraries provide the following functionality:
- @opentelemetry/semantic-conventions – Defines the standard attributes for traces as defined in the OTel specification.
- @opentelemetry/resources – Defines an object (resource) that represents the source generating the OTel data (in this tutorial, the messenger service).

Open tracing.mjs in a text editor and make these changes:

Add these lines to the set of import statements at the top of the file:

import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";

Create a resource called messenger under the correct key in the OTel spec by adding the following line after the last import statement:
```
const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: "messenger",
});
```

Pass the resource object to the NodeSDK constructor by adding the line highlighted in orange between the lines in black:

const sdk = new opentelemetry.NodeSDK({
  resource,
  traceExporter: new OTLPTraceExporter({ headers: {} }),
  instrumentations: [getNodeAutoInstrumentations()],
});

Restart the messenger service:
```
node --import ./tracing.mjs index.mjs
```
Wait about ten seconds, then in the client terminal (which you opened in Step 4) send another health‑check request to the server (you can run the command a few times if you want to see multiple traces):
```
curl -X GET http://localhost:4000/health
```
Note: Leave the client terminal open for reuse in the next section and the messenger terminal open for reuse in Challenge 3.
Confirm that a new service called messenger appears in the Jaeger UI in the browser (this may take a few seconds and you may need to refresh the Jaeger UI):
Select messenger from the Service drop‑down menu and click the Find Traces button to see all the recent traces originating from the messenger service (the screenshot shows the 2 most recent of 20):
Click on a trace to display the spans in it. Each span is properly tagged as originating from the messenger service:

Configure OTel Auto-Instrumentation of the notifier Service

Now launch and configure auto‑instrumentation for the notifier service, running basically the same commands as in the two previous sections for the messenger service.

Open a new terminal session (called the notifier terminal in subsequent steps). Change to the app directory in the notifier repository and install Node.js (you can substitute a different method if you wish):
```
cd ~/microservices-march/notifier/app
asdf install
```
Install dependencies:
```
npm install
```
Start the PostgreSQL database for the notifier service:
```
docker compose up -d
```
Create the database schema and tables and insert some seed data:
```
npm run refresh-db
```

Install the OTel Node.js packages (for a description of what the packages do, see Steps 1 and 3 in Configure OTel Auto‑Instrumentation Sent to the Console):

npm install @opentelemetry/[email protected] 
  @opentelemetry/[email protected] 
  @opentelemetry/[email protected] 
  @opentelemetry/[email protected] 
  @opentelemetry/[email protected]

Create a new file called tracing.mjs:
```
touch tracing.mjs
```

In your preferred text editor, open tracing.mjs and add the following script to get the OTel SDK up and running:

import opentelemetry from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";

const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: "notifier",
});

const sdk = new opentelemetry.NodeSDK({
  resource,
  traceExporter: new OTLPTraceExporter({ headers: {} }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Note: This script is exactly the same as the one for in the messenger service, except that the value in the SemanticResourceAttributes.SERVICE_NAME field is notifier.

Start the notifier service with OTel auto‑instrumentation:
```
node --import ./tracing.mjs index.mjs
```
Wait about ten seconds, then in the client terminal send a health‑check request to the notifier service. This service is listening on port 5000 to prevent conflict with the messenger service listening on port 4000:
```
curl http://localhost:5000/health
```
Note: Leave the client and notifier terminals open for reuse in Challenge 3.
Confirm that a new service called notifier appears in the Jaeger UI in the browser:

Configure OTel Instrumentation of NGINX

For NGINX, you set up tracing manually instead of using the OTel auto‑instrumentation method. Currently, the most common way to instrument NGINX using OTel is to use a module written in C. Third‑party modules are an important part of the NGINX ecosystem, but they require some work to set up. This tutorial does the setup for you. For background information, see Compiling Third‑Party Dynamic Modules for NGINX and NGINX Plus on our blog.

Start a new terminal session (the NGINX terminal), change directory to the root of the messenger repository and create a new directory called load-balancer, plus new files called Dockerfile, nginx.conf, and opentelemetry_module.conf:
```
cd ~/microservices-march/messenger/
mkdir load-balancer
cd load-balancer
touch Dockerfile
touch nginx.conf
touch opentelemetry_module.conf
```

In your preferred text editor, open Dockerfile add the following (the comments explain what each line does, but you can build and run the Docker container without understanding it all):

FROM --platform=amd64 nginx:1.23.1

# Replace the nginx.conf file with our own
COPY nginx.conf /etc/nginx/nginx.conf

# Define the version of the NGINX OTel module
ARG OPENTELEMETRY_CPP_VERSION=1.0.3

# Define the search path for shared libraries used when compiling and running NGINX
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/opentelemetry-webserver-sdk/sdk_lib/lib

# 1. Download the latest version of Consul template and the OTel C++ web server module, otel-webserver-module
ADD https://github.com/open-telemetry/opentelemetry-cpp-contrib/releases/download/webserver%2Fv${OPENTELEMETRY_CPP_VERSION}/opentelemetry-webserver-sdk-x64-linux.tgz /