A Script to run Grails Functional Tests in Parallel

The following post is about our effort to set up a way to run functional tests in Parallel for our grails application. It contains a script that will run functional tests based on X-number of instances of the application server, creating the right databases and the proper environment.

The problem

At Secret Escapes, we have been struggling with the amount of time it takes to run functional tests. Our tests written in Geb and Spock has grown substantially and it takes up to 25 minutes to run completely.

During Luke Daley’s talk at the GGUG in London, he mentioned that one solution to speed up running functional tests was to set up multiple environments and run functional tests against each of them.

We wanted a way to do this in developer machines so they could run in parallel. After a few days of head scratching and ‘WTF doesn’t this work!’, we ( Donovan Frew, Michael Stogowski and me) came up with the script below.

The Script

Save this script in your scripts directory. The name we use for it is TestFunc.groovy

import groovy.sql.*
import org.codehaus.groovy.grails.test.*
import org.codehaus.groovy.grails.test.support.*
import org.codehaus.groovy.grails.test.event.*

includeTargets << grailsScript("TestApp")

target(main: "Runs functional tests in parallel in sets of bucketSize") {
	def reportsDir = 'reports'
	def numberOfServers = 5

	def sql = Sql.newInstance('jdbc:mysql://localhost:3306/', 'root', '', 'com.mysql.jdbc.Driver')

	def tests = new SpecFinder(binding).getTestClassNames()
	new File(reportsDir).mkdirs()
	def commands = []
	def threads = []
	def results = ''

	numberOfServers.times { id ->

		def reportsFile = new File(reportsDir + '/' + 'test' + id).absolutePath

		sql.execute( "DROP DATABASE IF EXISTS parallelDB${id};" )
		sql.execute( "CREATE DATABASE parallelDB${id};" )

		def pattern = ''

		tests.eachWithIndex { test, index ->

			if (index % numberOfServers == id)
			{
				pattern += " ${ tests.get(index) }"
			}

		}

		def command = "grails -Dgrails.project.test.reports.dir=${reportsFile} -Dserver.port=909${id} -Ddb.name=parallelDB${id} test-app functional:  ${pattern}"

		threads << Thread.start {

			println command
			ProcessBuilder builder = new ProcessBuilder(command.split(' '));

			builder.redirectErrorStream(true);
			Process process = builder.start();

			InputStream stdout = process.getInputStream();
			BufferedReader reader = new BufferedReader(new InputStreamReader(stdout));

			while ((line = reader.readLine()) != null)
			{
				if( !line.contains( 'buildtestdata.DomainInstanceBuilder' ) ){
					System.out.println("Server ${id}: " + line);
				}

				if( line.contains( 'Tests passed:' ) || line.contains( 'Tests failed:' ) ){
					results += "Server ${id}: " + line + '\n'
				}
			}

		}

	}

	threads.each {
		it.join()
	}

	println '------------------------------------'
	println 'Tests FINISHED'
	println '------------------------------------'
	println results

}

setDefaultTarget(main)

class SpecFinder extends GrailsTestTypeSupport {

	SpecFinder(binding) {
		super('name', 'functional')
		buildBinding = binding
	}

	int doPrepare() {
		0
	}

	GrailsTestTypeResult doRun(GrailsTestEventPublisher eventPublisher) {
		null
	}

	def getTestClassNames() {
		findSourceFiles(new GrailsTestTargetPattern('**.*Spec')).sort{ -it.length() }.collect{ sourceFileToClassName(it) }
	}
}

How it works:

This script only works with Geb Specs. If you need to use some other sort of functional testing, you will have to modify the way you identify your tests.

In order to have parallel environments, we needed to set up:

1. Different Databases
2. Different Report Directories so they wouldn’t get overwritten by each other
3. Different Ports to run functional tests against.

Notes about the script

• We use the ProcessBuilder instead of the standard Groovy .execute() method so we can bind the output of the script straight to the console while the script is running.
• Funtional specs are sorted by file size and executed via pattern matching. It could be more sophisticated but seems to work.

Application Changes needed for the Test environment

• To get this to work, we needed to change places in our Config.groovy that had a hard-coded reference to 8080. Instead, we specified a port name like so:

  grails.serverURL = "http://localhost:${ System.getProperty("server.port")?:'8080' }/${appName}"

• You also need to change the way your test environment loads it’s datasource so it can be overwritten by the command line. We changed ours to

  def dbName = System.getProperty("db.name") ?: 'flashsales_test'
  url = "jdbc:mysql://localhost:3306/$dbName?autoReconnect=true"
  

• This way, whenever db.name was specified in the command line, the test Datasource would use this.
• Change our functional tests so that there are no hard references to port 8080.
• We use mysql and the groovy sql mechanism to create new databases for these environments. You might choose a different poison.

Running the Script

• You can change the number of instance you want to run by changing the numberOfServers value. A quick modification to the script would be to make this value one that the script gets, but we haven’t gotten there yet.
• To run, just type grails test-func in your console.

Results

After getting this to work correctly, we noticed that in some cases, our test execution time went down from 26 minutes to 12 minutes. It is a definitive improvement.

Things to Do Next

• We would like to use the Grails pattern matching for test-app here, but haven’t had a chance to dig deeply into the code to do so.
• Number of servers, database names should all be configurable or read from Config.groovy.
• Not sure if running something like GPars would give us a performance gain over running scripts again.