Camel: set up kubernetes healthcheck endpoints

Camel 3 was a big step forward to a more modern development model. Except Spring Boot integration, it also provides Kubernetes like integration like health checks which enables to create liveness and readyness probe endpoints. Let see how to set it up.

To set up camel healthcheck registry you must:

• Add to camel context a health check registry
• Create a healthcheck registry
• (optionally) Register the route health check repository if you want the route healthchecks
• Bind liveness and readyness to endpoints

Create a camel healthcheck registry

The first step is to ensure you have camel-health dependency:

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-health</artifactId>
  <version>${camel.version}</version>
</dependency>

Then you can instantiate a registry and bind it to the context:

final var checkRegistry = new DefaultHealthCheckRegistry();
checkRegistry.register(new RoutesHealthCheckRepository());
camelContext.setExtension(HealthCheckRegistry.class, checkRegistry);

This will enable you to use HealthCheckHelper which provides invokeReadiness and invokeLiveness endpoints which return the list of healthcheck results.

Tip: you can register custom HealthCheck if you want to enrich them with custom ones (datasource check for example) but previous code, thanks to RoutesHealthCheckRepository, will create one healthcheck per route.

Bind the healthchecks

There are multiple options to bind the healthchecks over HTTP. The simplest is likely to wire the healthchecks to existing endpoint. It can be the case if you run a Microprofile server. In such a case, you can add camel-microprofile-health module and wire the camel checks to microprofile endpoint automatically. However it is not that common and consistent to use a business oriented stack for a camel/orchestration/ESB oriented middleware.

The other nice option is to use camel-servlet - or any other camel "HTTP" module. With this option, the idea is to register two routes (one for liveness and one for readyness) and to return the check status when all is fine and a HTTP 503 (or any status notifying kubernetes/docker-compose/...) the check failed.

Here what it can look like with camel-servlet and camel-rest modules:

api.get("/api/health/ready")
        .id("app.health.readiness")
        .route()
        .setHeader(CONTENT_TYPE, constant("application/json"))
        .process(exchange -> setHealthCheckResult(exchange, HealthCheckHelper.invokeReadiness(getContext())));

api.get("/api/health/live")
        .id("app.health.liveness")
        .route()
        .setHeader(CONTENT_TYPE, constant("application/json"))
        .process(exchange -> setHealthCheckResult(exchange, HealthCheckHelper.invokeLiveness(getContext())));

Nothing crazy, the content type is forced to use JSON and we use setHealthCheckResult utility to convert the list of health check result we get for each endpoint to the right HTTP status in camel-rest and set the payload to return to the caller.

Here is one implementation of the utility method:

private void setHealthCheckResult(final Exchange exchange,
                                 final Collection<HealthCheck.Result> results) {
    if (results.stream().anyMatch(it -> it.getState() != HealthCheck.State.UP)) { // <1>
        exchange.getMessage().setHeader(HTTP_RESPONSE_CODE, 503);
    }
    exchange.getMessage().setBody(results); // <2>
}

• If any check result is not UP then we consider the overall status of the application is down and we set the response status to 503,
• We set the returned payload to the list of results (works well thanks the POJO-like model of Result). note that the payload does not have to be the results, it can be empty or wrapped in an aggregating payload ({globalStatus,results}) as in Microprofile Health specification. This is more for human beings than any automotion so ensure to create the body which works the best for you.

Result

Running, and calling the live endpoint, you will get something like that:

[
   {
      "check":{
         "configuration":{
            "enabled":true,
            "failureThreshold":0,
            "interval":0
         },
         "group":"camel",
         "id":"route:app.health.readiness",
         "liveness":false,
         "metaData":{
            "invocation.count":1,
            "invocation.attempt.time":"2020-11-02T11:30:46.761982+01:00[Europe/Paris]",
            "check.id":"route:app.health.readiness",
            "invocation.time":"2020-11-02T11:30:46.761982+01:00[Europe/Paris]",
            "check.group":"camel",
            "failure.count":0
         }
      },
      "details":{
         "route.id":"app.health.readiness",
         "invocation.count":1,
         "route.context.name":"app-camel-context",
         "invocation.time":"2020-11-02T11:30:46.761982+01:00[Europe/Paris]",
         "route.status":"Started",
         "failure.count":0
      },
      "state":"UP"
   },
   {
      "check":{
         "configuration":{
            "enabled":true,
            "failureThreshold":0,
            "interval":0
         },
         "group":"camel",
         "id":"route:app.health.liveness",
         "liveness":false,
         "metaData":{
            "invocation.count":1,
            "invocation.attempt.time":"2020-11-02T11:30:46.76289+01:00[Europe/Paris]",
            "check.id":"route:app.health.liveness",
            "invocation.time":"2020-11-02T11:30:46.76289+01:00[Europe/Paris]",
            "check.group":"camel",
            "failure.count":0
         }
      },
      "details":{
         "route.id":"app.health.liveness",
         "invocation.count":1,
         "route.context.name":"app-camel-context",
         "invocation.time":"2020-11-02T11:30:46.76289+01:00[Europe/Paris]",
         "route.status":"Started",
         "failure.count":0
      },
      "state":"UP"
   },
   {
      "check":{
         "configuration":{
            "enabled":true,
            "failureThreshold":0,
            "interval":0
         },
         "group":"camel",
         "id":"route:app.api.test",
         "liveness":false,
         "metaData":{
            "invocation.count":1,
            "invocation.attempt.time":"2020-11-02T11:30:46.762987+01:00[Europe/Paris]",
            "check.id":"route:app.api.test",
            "invocation.time":"2020-11-02T11:30:46.762987+01:00[Europe/Paris]",
            "check.group":"camel",
            "failure.count":0
         }
      },
      "details":{
         "route.id":"app.api.test",
         "invocation.count":1,
         "route.context.name":"app-camel-context",
         "invocation.time":"2020-11-02T11:30:46.762987+01:00[Europe/Paris]",
         "route.status":"Started",
         "failure.count":0
      },
      "state":"UP"
   }
]

You can now set it up in your Kubernetes spec:

apiVersion: v1
kind: Pod # would work the same for a deployment
metadata:
  labels:
    app: demo-camel-health
  name: demo-camel-health
spec:
  containers:
  - name: demo-camel-health
    image: demo-camel-health:latest
    livenessProbe:
      httpGet:
        path: /api/health/live
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 3
    readinessProbe
      httpGet:
        path: /api/health/ready
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 3

Conclusion

It is quite important to set up proper healthchecks in your application. Camel does not do it automatically out of the box but it also enables you to control where it is deployed.

In plain old bare metal deployments, you still use health checks to monitor your application but often want to bind them on a "localhost" server whereas your main application/gateway will bind another host (monitoring concern vs business one). With such a solution it is trivial to bind the health checks on another server than the business one.

When running Apache Camel in Spring Boot or a Microprofile server, it is also better to align Camel health report to the stack monitoring and this is what does Camel Microprofile Health integration module for example.

So at the end, no excuse to not know if your application is up and runnning or not and potentially to have an orchestrator (k8s) to respawn it when down ;).