Generate your adoc module/application configuration instead of maintaining it manually

, Romain Manni-Bucau, 2016-10-25, 8 min and 57 sec read

Configuration if a central piece of an application and it is one of the communication channels with other people of your team or other teams (if you have an operation team). For that reason it is insanely important to ensure it is up to date and accurate.

For that reason it is impossible to do it manually without adding an insane process of review which would be very costly for a poor gain.

So how to do? To answer that question I'll use an application configured with DeltaSpike as example but it will be easy to generalize it to any application: the very high level view is to generate the documentation at build time.

Generation setup

Your configuraton can be lucky and use a framework with a plugin for your build tool to generate the documentation but if not, don't worry, you can "make your own". For Gradle it is quite easy since you can script it but for Maven you would need to write a Mojo (another small project). Where it is possible it is often too much overhead for the goal and therefore we'll just use the groovy mojo to script our documentation generation:

<plugin>
  <groupId>org.codehaus.gmaven</groupId>
  <artifactId>groovy-maven-plugin</artifactId>
  <version>2.0</version>
  <executions>
    <execution>
      <id>config-doc</id>
      <phase>prepare-package</phase>
      <goals>
        <goal>execute</goal>
      </goals>
      <configuration>
       <source>${project.basedir}/src/main/documentation/Configuration.groovy</source>
      </configuration>
    </execution>
  </executions>
  <dependencies>
    <dependency>
      <groupId>org.apache.xbean</groupId>
      <artifactId>xbean-finder-shaded</artifactId>
      <version>4.5</version>
    </dependency>
  </dependencies>
</plugin>

This is a standard setup of the plugin, the Configuration.groovy script will be executed just before creating your artifact (jar/war). You can have noted I added xbean-finder as a dependency, we'll soon see why.

Code the documentation generation

Now we have the plugin setup we need to create the src/main/documentation/Configuratoin.groovy. What will do the script?

find the configuration
extract from each entry the configuration name (can be an xml path), the default if any and a small description
dump everything in a well formatted document

Find the configuration

Illustrating with DeltaSpike this post, we'll need to look for these annotations:

@Inject
@ConfigProperty(name ="....", defaultValue = "...")
private String value;

That's where xbean-finder will help us. This is a small library parsing a list of "archives" to find annotations.

To create an AnnotationFinder, we need to create a classloader usable for the application and define what we scan (the Archive). Thanks to the groovy plugin we inherit from maven components and can use the project to achieve it - not that with gradle it is globally the same using configurations:

def classes = new File(project.build.outputDirectory).toURI().toURL()

def urls = []
urls.add(classes)
project.artifacts.each { urls.add(it.file.toURI().toURL()) }

def configLoader = new URLClassLoader(urls as URL[])
def finder = new AnnotationFinder(new FileArchive(configLoader, classes), true)

In summary, we create a classloader composed of dependencies and the module classes, we define we want to scan only the module classes (FileArchive) and create build a finder from it.

Tip: of course you can scan the whole dependency set if you want, just ensure to not scan utilities or external modules cause it can increase the execution time significantly for no gain. If you are interested in that strategy see xbean JarArchive and CompositeArchive.

Then the finder API is quite simple and you generally do:

Collection<Field> fields = finder.findAnnotatedFields(MyAnnotation.class);

Here it is just a bit more tricky cause to respect the classloading so we need to load MyAnnotation with the same classloader than the application: our configLoader.

Concretey we'll just do:

def markerConfig = configLoader.loadClass('org.apache.deltaspike.core.api.config.ConfigProperty')
finder.findAnnotatedFields(markerConfig)

Note: using this finder we get a nice high level API but we need to respect the classloading (last boolean parameter we set creating the finder). It is also possible to not go through these constraints disabling it or just using directly asm library but API is a bit more complex so personally I find this trade off quite acceptable.

Side note: for this post we'll only handle field injections but with the finder you can find constructor and setters injections too. This would be the exact same kind of logic.

Description?

DeltaSpike API only answers to DeltaSpike needs and therefore doesn't provide a description member to its @ConfigProperty. Not a big deal, we'll create our own:

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

import static java.lang.annotation.ElementType.FIELD;
import static java.lang.annotation.RetentionPolicy.RUNTIME;

@Target(FIELD)
@Retention(RUNTIME)
public @interface Description {
    String value();
}

Then update your configuration to use it:

@Inject
@Description("this configures something important")
@ConfigProperty(name ="....", defaultValue = "...")
private String value;

Of course we need to use the same trick as for @ConfigProperty to find it so we'll do:

def markerDescription = configLoader.loadClass('com.company.Description')

Output the documentation: adoc to the rescue

We have our configuration marker, we can find all attributes we want (@ConfigProperty has name and default, out @Description has some human readable description) so we just need to dump it in a table.

Most of the time you will want to output some html to share it easily but also some PDF to be able to mail the documentation (mailing a html is not that friendly if you already tried ;)).

Asciidoc has the advantage to support both (and more) outputs and is an easy format to generate since it is a wiki like syntax. Here what we want to generate can look like:

= My Config

|===
| Name     | Default | Description
| sample 1 | -       | aweomse config
| sample 2 | true    | aweomse config too
|===

I'll not show you how to output that exactly but something a bit more advanced in next part.

Custom generation

I previously justified the fact to code the generation by the lack of plugin for my configuration API. This is not 100% the actual reason. The actual reason is code is simple and allows you to format it exactly as you want with the content you want.

If you want to add some static data or documentation blocks, you can include other .adoc anywhere in the generation, if you want to instantiate some objects to generate some samples (very nice for JSON/XML REST API documentation) you can do it too etc...An automatic tool wouldn't be that powerful.

To show that I will suppose our application uses a common pattern for its configuration keys: ${application prefix}.${configuration segment}+.${configuration attribute}.

Concretely if we need to configure a twitter client, and a database pool it can look like:

app.twitter.baseUrl = ...
app.twitter.http.socket_timeout = ...
app.twitter.http.read_timeout = ...
app.database.driver = ...
app.database.url = ...
app.database.username = ...
app.database.password = ...
app.database.pool.min = ...
app.database.pool.max = ...

Of course this is a simplified version but we identify clearly we have a configuratoin marker then multiple levels of configuration for this marker: twitter configuration has the endpoint configuration and the http client configuration, database configuration has the connection configuration and the pool configuration etc...

Merging all properties in a single table wouldn't make it very obvious and would make the configuration a bit harder to read. To solve that we'll spli the configuration and create a table for each part. And to keep it hierarchic as ou keys we'll increment the level of title for each added segment: twitter would be a level 2 (level 1 if the whole document title) and twitter.http would be a level 3 for instance.

There are a lot of ways to code that but I chose a lazy one: build a configuration tree, sort the keys by tree node and dump them all. Each node of the tree will be responsible to dump its level and its children (then we browse the whole tree at the end).

To build the tree we iterate over all config entries found thanks the finder and we split the key with dots. We ignore the application prefix and then just add the nodes to the tree respecting the tree path the split gives us.

Enough words, let see what it can look it:

import org.apache.xbean.finder.AnnotationFinder
import org.apache.xbean.finder.archive.FileArchive

import static java.util.Locale.ENGLISH

class Config {
    def name
    def defaultValue
    def description
}
class Node {
    def shortName
    def level
    def children = [:]
    def childrenLeaves = []

    def dumpNode(writer) {
        (1..level).each { writer.write '=' }
        writer.writeLine "= ${shortName.capitalize()}"
        writer.writeLine ''

        if (!childrenLeaves.isEmpty()) {
            writer.writeLine '|==='
            writer.writeLine '|Name|Default|Description'
            childrenLeaves.sort { it.name }.each {
                writer.writeLine "|${it.name}|${it.defaultValue}|${it.description}"
            }
            writer.writeLine '|==='
            writer.writeLine ''
        }

        children.values()
                .sort { it.shortName.toLowerCase(ENGLISH) }
                .each { it.dumpNode(writer) }
    }

}

// retrieves all @ConfigProperty on fields and generate doc from it
def generateConfigurationDoc() {
    def classes = new File(project.build.outputDirectory).toURI().toURL()

    def urls = []
    urls.add(classes)
    project.artifacts.each { urls.add(it.file.toURI().toURL()) }

    log.debug("using classloader ${urls}")

    def configLoader = new URLClassLoader(urls as URL[])
    def markerConfig = configLoader.loadClass('org.apache.deltaspike.core.api.config.ConfigProperty')
    def markerDescription = configLoader.loadClass('com.company.Description')
    def finder = new AnnotationFinder(new FileArchive(configLoader, classes), true)

    def tree = new Node(shortName: "${project.name} :: Configuration", level: 0)
    finder.findAnnotatedFields(markerConfig).each {
        def config = it.getAnnotation(markerConfig)
        def description = it.getAnnotation(markerDescription)
        if (description == null) {
            throw new IllegalStateException("No @Description for " + it)
        }

        def name = config.name()
        def current = tree;
        def values = name.replace('myapp.', '').tokenize('\\.')
        for (int i = 0; i < values.size() - 1; i++) {
            def key = values.get(i);
            def newNode = current.children[key]
            if (newNode == null) {
                newNode = new Node(shortName: key, level: current.level + 1)
                current.children[key] = newNode
            }
            current = newNode
        }
        current.childrenLeaves << new Config(
                name: name,
                defaultValue: config.defaultValue().replace('org.apache.deltaspike.NullValueMarker', "-"),
                description: description.value())
    }

    if (tree.children.isEmpty()) {
        throw new IllegalStateException("Something went bad, we didn't find any doc");
    }

    def configuration = new File(project.build.directory, "doc/src/configuration.adoc")
    configuration.parentFile.mkdirs()
    configuration.withWriter('utf-8') { w ->
        w.writeLine "= ${project.name} :: Configuration"
        w.writeLine ''
        tree.dumpNode(w)
    }

}

generateConfigurationDoc()

The nice thing doing it from the build is you can use project to generate some parts. See how the title is reusing project.name for instance or project.build to find directories.

Groovy allows us to write real classes and "standard" code which makes the tree handling quite easy.

At the end the documentation is dumped in target/doc/src/configuration.adoc and you just need to render it. About that you can add asciidoctor (and its pdf extension) to do the generation in the same build script if needed.

Attach the documentation

Last important point is to ensure the documentation is released with the project. To do that you just need to attach the documentation with the module artifacts. Maven has a helper to do that just requiring the file, classifier and extension representing the artifact to attach. Here how you can do it in the setup of this post:

container.lookup('org.apache.maven.project.MavenProjectHelper')
  .attachArtifact(project, 'adoc', 'configuration', configuration)

Conclusion

Documentation is a part of software which is important to do cause it is one of the reference for end users - and dev when time passes ;). However doing it manually is quite costly, hard and not nice for developpers so automating it is always a big win and not that hard. You can get very high level documentation this way

configuration as we seen in this post
application graph interactions (capture clients in the generation)
resource usage (which part of your application use a JMS queue, a database, ...)
REST API (which paths, which payloads, ...)
and much more....

Correctly setup this generation is generally fast and really does worth it so when bootstraping a project ensure you don't bypass this step :).

From the same author:

Romain Manni-Bucau

In the same category:

Java/EE/Microprofile