Microprofile OpenTracing: microservice monitoring

Microprofile initiative creates new specifications for Java/JAX-RS/CDI environments and mainly targets the cloud.

After the configuration, the security (JWT Auth), it now tackles the tracing concern.

As a reminder, the goal of a tracing solution is to be able to follow end to end a transaction ("end user" request). It is one of the most critical concern in a microservice architecture since a lot of motley applications will participate to the same transactions and one can impact another one or trigger errors in another sub-system.

Historically, the first major modern mainstream tool was Zipkin which uses B3 protocol to propagate the trace id (and optionally span ids).

Very high level, the idea is to send to any remote system - not only remote HTTP services but also databases for instance, some metadata (typically HTTP headers) describing the transaction ID (called Trace ID), the current (client) hop/middleware ID (called Span ID).

Side note: there is a mode in B3 where you can force a remote Span ID, in this case you send a Parent Span ID to represent the client ID. This mode is however not encouraged now.

Thanks to these incoming - client - information, you can build a new Span representing the "current operation" with link to the parent one.

All these data will be sent to a backend - often through a collector which is the remote facade of the tracing backend, and then stored to be visualized later.

A typical Zipkin trace view is the following one:

Each line is a "hop" (or operation), it has the timing information + a name representing the information and if you click on a block, you get some more details (tags) which allows to qualify the execution (like an error if some happent). The first block on each line is the service name which is an alias used to identify the machine in Zipkin.

All this solution is getting standardized through the OpenTracing initiative. The overall process is the same as in zipkin. The drawback - today, is that the exchange format is not yet really standardize. This is the reason why there is some reuse or creation on that level. If you take one of the reference implementation of opentracing, Jaeger, you can see it is able to reuse Zipkin format, Uber format (the product is coming from here ;)) or W3C header since it starts to be standardize at that level too.

So what's the link with OpenTracing in all that context? OpenTracing integrates with Microprofile to enable your application to be monitored this way automatically.

The first important point is to reuse io.opentracing API, and in particular the Tracer which is basically a way to handle the "context" (called scope in this API) which represents a Span and a way to create Spans. Then, it provides a Tracer in the CDI context so you can just inject it to start using it in custom operations or part of your applications. It also provides a @Traced annotation which allows to customize the operation name and to deactivate/activate the monitoring on any CDI bean and JAX-RS endpoint. Finally it enables by default the tracing on all JAX-RS endpoints.

An implementation is available at Apache Geronimo (called geronimo-opentracing). Once the dependency added to your CDI application, you are monitored. The default "backend" implementation converts the opentracing spans to zipkin spans and logs it to a particular logger. This sounds quite a cheap implementation but actually all logger framework enable to log to any backend (Apache Kafka, JDBC, Elasticsearch, Apache Cassandra, HTTP, ...) so it is actually quite efficient and avoids a lot of integration modules. However if you want to deactivate the zipkin integration or the logging, there are CDI event you can observe to implement your own handling.

The project is available on github so don't hesitate to experiment and give feedback!

In terms of usage, here what it looks like if you want to implement some custom monitoring:

public class MyMonitoredMethod {
  @Inject
  private Tracer tracer;

  public void monitorMe() {
      final Scope scope = tracer.buildSpan("myOperationName")
          .withTag("myMeta", "demo")
          .withTag(Tags.SPAN_KIND.getKey(), Tags.SPAN_KIND_SERVER)
          builder.startActive(true);
      try {
          // do something
      } catch (final Exception ex) {
          Tags.ERROR.set(span, true);
          // a span can host log entries, you can also put it as tags
          scope.span().log(new HashMap<String, Object>() {
            {
              put("event", Tags.ERROR.getKey());
              put("event.object", ex);
            }
          });
      } finally {
          scope.close();
      }
  }
}

As you can see it is a quite classical API. The interesting point is the startActive which "binds" the span to the current thread. If you just use start, then it is not bound and you can manage it as you want (very useful for reactive and asynchrronous libraries). If you want to link to a "parent" span the current one, you need to call asChildOf before making your span active and that's it :).

So no reason to not know what happens in your server anymore, right?

Happy tracing!