How to handle subprocesses
From time to time you need to handle external processes from your application. An example is to manage an internal store where you can spawn an "owned" voldemort store, an hazelcast node with specific tuning (avoids to share JVM settings with the application), a custom NodeJS process associated with your application or even a built-in SSH or FTP process based on the OS programs, avoiding you to rewrite it in java.
What's the point? Being able to make this other process down if your application is down avoiding an half working instance or simply making the deployment simpler (avoiding to write new scripts or to manage multiple processes). If you even go further, if you add quartz/@Schedule in the game you can replace cron with a nice frontend but this requires a bit more work.
ProcessBuilder to the rescue
Since Java 8 you have most of what you need to handle an external process in the JVM: ProcessBuilder! This class is actually there since more time but was lacking some critical methods.
This class allows you to setup the command (String[]), handle basic IO (stdin, stderr, stdout) and to create a Process. Then once you have the process you can wait it exits during a timeout (this is one java 8 addition) and try to kill it if it doesn't (also a java 8 addition).
How to integrate it with the application (CDI or Servlet)
So sounds like we have all we need, why should I need CDI?
Actually few things are still needed from the application point of view:
- lifecycle handling: when to start/stop?
- CDI can use @Observes @initialized(ApplicationScoped.class) and @Observes @Destroyed(ApplicationScoped.class)
- Servlets can use a ServletContextListener
- command parsing: how do you pass the command? Do you hardcode it as an array or do you parse a plain String?
- IO handling: do you hardcode how stderr/stdout are redirected or not?
- Provide some interpolation: suppose you want to launch a java process, do you inherit from the application java instance? If your application runs on java 7 and the forked program needs java 8 how do you handle it? Is that hardcoded? No it is nice to be able to use ${java8.home} and reuse a system property from the main application.
- Do you provide hooks to be able to execute pre/post tasks around start/stop phases of the process
Coding a ProcessLifecycle
In this part we'll code what was described in the previous part. The ProcessLifecycle class will handle a single fork but code can easily be adapted to handle multiple:
import org.apache.commons.lang3.text.StrSubstitutor;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.context.Initialized;
import javax.enterprise.event.Observes;
import javax.script.Bindings;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.Reader;
import java.io.StringReader;
import java.net.Socket;
import java.util.LinkedList;
import java.util.List;
import java.util.logging.Logger;
import static java.util.concurrent.TimeUnit.MILLISECONDS;
import static java.util.logging.Level.WARNING;
@ApplicationScoped
public class ProcessLifecycle {
// config start, can require some config injection like Deltaspike @ConfigProperty
// resource id, tomee magic
private String serviceId;
private String command;
// if set the process will poll for the port to be opened
private int watchPort;
private String watchHost = "localhost";
private long watchTimeout = 60000;
// io propagation
private boolean inheritIo = true;
private boolean redirectErrorStream = false;
// if not inherited the redirection paths
private String stderr;
private String stdout;
private String stdin;
// stop handling
private long shutdownTimeout = 300000;
// customization
private String preStartScript;
private String postStartScript;
private String preStopScript;
private String postStopScript;
// config end
// runtime variables
private volatile Process process;
private volatile Thread hook;
// start cdi specific
public void start(@Observes @Initialized(ApplicationScoped.class) final Object start) {
// just touched (postconstruct), predestroy will finish the job automatically then
}
// end cdi specific
@PostConstruct
public void start() {
if (command == null) {
throw new IllegalArgumentException("No command set on '" + serviceId + "' resource.");
}
final ProcessBuilder builder = new ProcessBuilder(splitCommand(filteredCommand()));
if (inheritIo) {
if (stderr != null || stdout != null || stdin != null) {
throw new IllegalArgumentException("when setting inheritUi to true (default) you can't set stdin, stderr or stdout");
}
builder.inheritIO();
} else {
if (redirectErrorStream) {
if (stderr != null) {
throw new IllegalArgumentException("redirectErrorStream can't be set if stderr is set");
}
builder.redirectErrorStream(true);
}
if (stderr != null) {
builder.redirectError(new File(stderr));
}
if (stdout != null) {
builder.redirectOutput(new File(stdout));
}
if (stdin != null) {
builder.redirectInput(new File(stdin));
}
}
executeScript(preStartScript, "builder", builder);
boolean started = false;
try {
process = builder.start();
hook = new Thread() {
{
setName(serviceId + "-shutdown-hook");
}
@Override
public void run() {
ProcessLifecycle.this.stop();
}
};
Runtime.getRuntime().addShutdownHook(hook);
doWatch();
started = true;
} catch (final IOException e) {
try { // wait a bit then test if process needs to be killed
Thread.sleep(500);
} catch (final InterruptedException e1) {
Thread.interrupted();
}
try {
process.exitValue();
} catch (final IllegalThreadStateException ise) {
stop();
}
throw new IllegalStateException(e);
} finally {
executeScript(postStartScript, "started", started, "process", process);
}
}
@PreDestroy
public void stop() { // coded to pass in java 7 but using java 8 nice API if running on java 8
try {
executeScript(preStopScript, "process", process);
} finally {
int exit = Integer.MIN_VALUE;
try {
exit = doStop();
} finally {
executeScript(postStopScript, "exitValue", exit);
}
}
}
public Process getProcess() {
return process;
}
private void executeScript(final String script, final Object... bindings) {
if (script == null) {
return;
}
final ScriptEngineManager mgr = new ScriptEngineManager();
final File file = new File(script);
final ScriptEngine engine;
Reader content = null;
try {
if (file.exists()) {
final int lastDot = script.lastIndexOf('.');
if (lastDot > 0) {
engine = mgr.getEngineByExtension(script.substring(lastDot + 1));
} else {
engine = mgr.getEngineByExtension("js");
}
content = new BufferedReader(new FileReader(file));
} else {
engine = mgr.getEngineByExtension("js");
content = new StringReader(script);
}
final Bindings variables = engine.createBindings();
if (bindings != null) {
variables.put("out", System.out);
for (int i = 0; i < bindings.length; i += 2) {
variables.put(bindings[i].toString(), bindings[i + 1]);
}
}
try {
engine.eval(content, variables);
} catch (final ScriptException e) {
throw new IllegalStateException(e);
}
} catch (final FileNotFoundException e) {
throw new IllegalArgumentException(e);
} finally {
try {
if (content != null) {
content.close();
}
} catch (final IOException e) {
// no-op
}
}
}
private int doStop() {
if (hook != null) {
try {
Runtime.getRuntime().removeShutdownHook(hook);
} catch (final IllegalStateException ise) {
// ok we are already shutting down
} finally {
hook = null;
}
}
if (process == null) {
return Integer.MIN_VALUE;
}
process.destroy();
try {
process.waitFor(shutdownTimeout, MILLISECONDS);
} catch (final InterruptedException e) {
Thread.interrupted();
}
int exitValue;
try {
exitValue = process.exitValue();
} catch (final IllegalThreadStateException itse) {
Logger.getLogger(ProcessLifecycle.class.getName()).log(WARNING,
command + " didn't exist properly, will force destruction", itse);
try {
process.destroyForcibly();
process.waitFor(shutdownTimeout, MILLISECONDS);
} catch (final InterruptedException e) {
Thread.interrupted();
}
exitValue = process.exitValue();
}
Logger.getLogger(ProcessLifecycle.class.getName())
.info("Stopped elasticsearch code: " + exitValue);
return exitValue;
}
private String filteredCommand() {
return StrSubstitutor.replaceSystemProperties(command);
}
private void doWatch() {
if (watchPort <= 0) {
return;
}
final long end = System.currentTimeMillis() + watchTimeout;
while (end - System.currentTimeMillis() >= 0) {
try (final Socket s = new Socket(watchHost, watchPort)) {
s.getInputStream();
return; // opened :)
} catch (final IOException e) {
try { // try again
Thread.sleep(500);
} catch (final InterruptedException e1) {
Thread.interrupted();
break;
}
}
}
throw new IllegalStateException("Process " + command + " didnt start in " + watchTimeout);
}
private static List<String> splitCommand(final String raw) {
final List<String> result = new LinkedList<>();
Character end = null;
boolean escaped = false;
final StringBuilder current = new StringBuilder();
for (int i = 0; i < raw.length(); i++) {
final char c = raw.charAt(i);
if (escaped) {
escaped = false;
current.append(c);
} else if ((end != null && end == c) || (c == ' ' && end == null)) {
if (current.length() > 0) {
result.add(current.toString());
current.setLength(0);
}
end = null;
} else if (c == '\\') {
escaped = true;
} else if (c == '"' || c == '\'') {
end = c;
} else {
current.append(c);
}
}
if (current.length() > 0) {
result.add(current.toString());
}
return result;
}
}
This class has been written as a TomEE resource, if you remove the "cdi specific" block at the beginning you can define it as a tomee resource this way:
<Resource id="myprocess" class-name="com.foo.ProcessLifecycle">
command = ${java.home}/bin/java -cp ${catalina.base}/myapp com.myapp.Main
postStartScript = out.println('Started myprocess')
</Resource>
In TomEE id is propagated as serviceId in the resource instance. In CDI you will need to replace all setters by some configuration injection I recommand you DeltaSpike Config) for it.
The part to note are:
-
filteredCommand which reuses commons-lang3 to filter the command and support placeholders
-
splitCommand which converts a String in a preprocessed command (String[]) handling quoting
-
doWatch which allows to wait for a port to be opened. If you start a server you likely want to wait it is running before continuing to start (otherwise if you application request it at startup it will fail)
-
doStop which tries to shutdown cleanly the process and if it doesn't manage to do it after some timeout it tries to just kill it
-
executeScript which is used to execute some custom code before/after the start/stop phases. It relies on the script JSR and out of the box provides javascript support. The nice thing there is to pass some potentially useful instances like the process builder, process, exist value, etc... depending in which phase you are. If you allow it, don't forget to give the user a way to log something (nashorn is pretty bad at it). In previous sample it just exposes System.out as out in the script.
What about Servlet?
In a servlet container the exact same code works but you replace the lifecycle handling (@Observes, @PostConstruct, @PreDestroy) by a ServletContextLitener or a Servlet with load on startup = 1 and no mapping (tolerated by most containers). So even without CDI or Spring you can reuse most of the previous code.
Conclusion
Embedding the lifecycle management of an external program in an application is no more a challenge and can be done quite easily in java. Suprisingly it comes with the microservice trend since each part of the application needs to be isolated which can means that an application embeds some other technology to create a consistent delivery.
From the same author:
In the same category: