Gradle Madness – ExecWait: A task that waits for command line calls to be ready

We’re using Gradle as a test runner for our fairly heterogenous project ( built with grails, dropwizard and yeoman ).

One of the problems we ran into is that we wanted to execute our tests when all our services were ready.

There doesn’t seem to be a way to do so with the standard Exec task. There are plugins that allow you to start, say, Tomcat, Vert.X or the Google App Engine.

But it felt really cumbersome to have to write a task that simply starts yeoman or grails. This seems to be a common question around testing. Both being asked before in the mailing list, StackOverflow and Gradle JIRA.

In this post, I write about a hack we wrote to allow Exec to wait for a service to be ready and ask for better ways of doing this type of work in Gradle.

Our Problem:

We have an app that has two services, one written in Grails and one in Dropwizard. We also use Yeoman for our static assets. In our end to end functional tests, driven by Gradle, we want to say:

1. Start Grails
2. Start Yeoman
3. Start DropWizard
4. Run Tests
5. Shut Down Everything

However, the standard Exec task provided by Gradle is not capable of even spawning a process. The Ant.exec task doesn’t do better, as we’re unable to easily check that our service is ready.

Our Hack:

The solution we came up with is as follows. We spawn a process in our task and wait for a certain string to appear in the command line. When it does, we know our task is ready and move on to the next gradle task. In practice, it looks like the following:

// Task to start Grails application

task startGrails(type: ExecWait) {
    command './grailsw run-app --plain-output'
    ready 'Server running.'
    directory '../grails'
}

The command and directory properties are pretty self explanatory. The ready property is a string that appears in the output of running that command. If I’m starting MongoDB, for example, my ready string would look something like ‘[websvr] admin web console waiting for connections on port 28017’. Another example is our DropWizard task, which looks like this:

task startDropWizard(type: ExecWait) {
    command './gradlew run'
    ready 'Started SocketConnector@0.0.0.0:8001'
    directory '../dropWizard'
}

You will notice that we use a Gradle task to run another Gradle task instead of using the includeFlat command for subprojects. One of the reasons for this is to keep a consistent way of launching apps, even if they are gradle. A more important reason, however, is to make sure that when the run task in our DropWizard app is executed, it only blocks our build until it is serving our web service.

So really that’s it. We use the Java ProcessBuilder to start the command. The implementation of the task looks something like this:

class ExecWait extends DefaultTask {
    String command
    String ready
    String directory

    @TaskAction
    def spawnProcess() {

        ProcessBuilder builder = new ProcessBuilder(command.split(' '))
        builder.redirectErrorStream(true)
        builder.directory(new File(directory))
        Process process = builder.start()

        InputStream stdout = process.getInputStream()
        BufferedReader reader = new BufferedReader(new
InputStreamReader(stdout))

        def line
        while ((line = reader.readLine()) != null) {
            println line
            if (line.contains(ready)) {
                println "$command is ready"
                break;
            }
        }
    }
}

Killing The Process

We started off by having the ExecWait task write out a PID and then killing the process at the end of the test execution. However, we found out there was no platform independent way of doing this, and it just lead to a lot of extra code to manage the PID. It also meant adding a lot of Kill tasks which were littering up our build.gradle file. Instead, we ended up writing a small task that will kill any process available on known ports that we bind to in our environment:

task freeAllPorts << {
    def ports = [3501, 8000, 8001, 8080]

    ports.each { port ->
        def cmd = "lsof -Fp -i :$port"
        def process = cmd.execute()
        process.in.eachLine { line ->
            def killProcess = "kill -9 ${ line.substring(1) }".execute()
            killProcess.waitFor()
        }
    }
}

Putting it all together:

Rob Fletcher pointed me to an earlier implementation he had done for the Vert.x Gradle task, which we then co-opted and modified during our pairing session. Basically, there was an issue with Gradle where a failed test will make the cleanup code not compile, so we added a Gradle Life Cycle listener around out test.

gradle.addListener new TestLifecycleListener()

class TestLifecycleListener implements TaskExecutionListener {

    def servers = ['DropWizard', 'Grails', 'Yeoman']

    @Override
    void beforeExecute(Task task) {
        if (task.name == 'test') {
            servers.each {
                println "calling start$it"
                task.project.tasks."start$it".execute()
            }
        }
    }

    @Override
    void afterExecute(Task task, TaskState taskState) {
        if (task.name == 'test') {
            task.project.tasks.freeAllPorts.execute()
        }
    }
}

Questions, Questions and more Questions:

I’ve only started looking at Gradle in earnest for a day and half, so the solution we have here is fairly rudimentary. The external processes seem to fail the build when the process cannot be bound. But working on this has raised a few questions, maybe someone with more Gradle knowledge can help make this better:

• What is the correct way to do this type of Around Interception in other tasks. Other than writing out a file to disk, how can we link the output of a startupProcess task to a shutdown process task. 
• Is there a better strategy for dealing with external build tools? Ted Naleid has written about calling Grunt from Gradle but it seems odd that there is no simpler way to integrate external heterogeneous builds into the Gradle environment.
• It feels like this is something that Gradle core should handle very easily. Is there a task that we’re simply missing?

Anyway, if you have any suggestions or comments on how this could be improved, please feel free to write in the comments box below.