When I was interviewing a candidate recently, he asked about the hardware that we used at Secret Escapes. I replied that we used standard Linux boxen for our day to day development, but the question made me consider the vast amount of tools, plugins and services that our little website uses.
Secret Escapes is a private sales travel site. Our members sign up to our site and get the opportunity of booking hotels and vacation packages at the guaranteed lowest prices on the web in sales that only last for a week. You can read up more on what we do in this post written by one of our investors and in this article.
The heart of the Secret Escapes technology is Grails and Groovy. We are using Grails 1.3.7 and all our support and migration scripts are written in Groovy. It doesn’t seem like a lot of other Grails shops are sharing the tools and processes they use, so I thought it might be useful to catalog the ones we find useful in our every day development.
In this post, I want to highlight some of the tools we use, why we chose them, and how they help us solve problems.
Development Environment / Tools
The dev team at Secret Escapes uses, for the most part, Ubuntu. Our Jenkins server is running on Crunchbang and our graphic designer uses Windows for Photoshop / Illustrator.
At the beginning, we were all on Windows machines. However, we soon ran into a few Windows only issues that plagued some of the Grails extensions. The database migration plugin, for example, would include certain extra fields and parameters in changesets. We also ran into strange issues using webdriver with Geb, and some earlier versions of plugins like Remote Control were simply not coded with Windows in mind.
Gradually, our entire team switched to Linux and it proved to be a better development environment for Grails, where it seems most of the development team works on Macs, anyways ( we probably would if we had the budget ).
One of the pieces of software I pushed hard for us to license was IntelliJ Idea. We had a lot of issues with Eclipse / STS in its earlier incarnations, and the ease of use and Grails integration within Idea just seemed like a more friction-less decision to make. While IntelliJ seems to struggle with some newer features of Grails ( plugin-driven-development is particularly lacking, for example ), it is definitively a rich productivity enhancing tool for us. No other IDE in the market is easier to use nor more grails friendly than Idea.
Git Flow / Git
Secret Escapes launched in January. Over the next six months and thousand+ sales, we had to quickly add and improve the functionality of the site. Using Grails made this type of agile / quick enhancements fairly easy, but we quickly ran into roadblocks using SVN.
One of our main problems with SVN was keeping track of deployments and different versions of code. We would create a branch of our code to work on some feature, and then face great difficulties in changing our deployed codebase when a quick text or copy change was required. Fed up with the situation, we looked at alternatives and concluded that switching to Git would be a good idea.
However, our team had very little experience with Git. It felt daunting. Luckily, we came across the Git Flow branching model. Git Flow is a set of commands on top of git that enforce a master / develop / hotfix / feature model. It adds a layer of conventions over the total freedom provided by Git. It almost feels like the ‘Convention over Configuration’ brought into source control.
Overall, we’re very happy with Git and Git flow. The one place where it falls apart a bit is when you’re working with larger features and have to merge these back into development. But it is definitively more flexible than SVN and allows us to code fearlessly.
For our issue tracking, we chose to move away from JIRA and use Pivotal Tracker instead. Pivotal is a Kanban board that enables us to quickly track issues that are important. For our team, it allows our business counterparts to keep track of what we’re working on and, unlike a physical board, means that we can edit them outside of the office.
Hosting / Infrastructure
Our main hosting solution is EveryCity. They provide a cloud-based VPS service that enables us to quickly clone instances if our traffic ever spikes. They are a premium service but offer excellent speeds for our UK customers and have quite a good / responsive tech support and monitoring service.
Amazon S3 and CloudFront
For our CDN, we deploy to Amazon S3 and their CloudFront service using the JetS3t library (Although you might want to check out Lucas Teixeira’s Excellent Amazon AWS plugin for Grails).
Secret Escapes has a lot of quickly changing sales data. Imagine a product catalog that gets 4 new pages with glossy images added every two days. By putting this content in a CDN, we are able to ensure that static resources uploaded to our CMS system is immediately available to our developers as well.
By using a CDN, we no longer need to worry about moving static files from staging to production, and have the added bonus of being able to use the same data in our development / testing. ( We have a script for continous deployment that pulls down our database data ).
Additionally, we get all the quick download and headers expiry optimization available to CloudFront resources.
Gaelyk and the Google App Engine
Image resizing has always been an issue with any sort of content-rich website written in Java. We found that the default Grails solutions like Burning Image and ImageTools simply returned badly resized images. We wanted glossy sexy images.
For CruiselineFans.com, we implemented a resizing solution using ImageMagick. This solution required custom setup in our development environment as ImageMagick is not cross platform. There was no guarantee this wasn’t going to break across Windows / Mac / PCs. It was just really painful.
For Secret Escapes, we wrote a quick little resizing service using Gaelyk and the Google App Engine using the techniques described here. Gaelyk is a groovy-based framework specially designed for Google App Engine. Our code to resize images was less than 20 lines but it provided us with an immediate, cross platform way of creating high quality images. Given that we only resize a small amount of images in our CMS, this solution allowed us an elegant and portable way of solving the image size in grails problem.
Overall, we find that Gaelyk and the Google App Engine provides a great additional set of APIs and tools that might not be available otherwise. We would definitively not write our entire site with it, but if we needed to add features like Custom Handlers for Emails, it provides an excellent alternative.
Cloud Foundry is VMWare’s new Cloud based solution for deploying all sorts of applications. It competes in the same space as Google App Engine, Amazon Elastic Beanstalk and Heroku / Azure.
First, let me clarify that only a fool would put a production environment on VMWare’s new beta Cloud Foundry solution in its current state. You can’t access the machines from the outside, it lacks a clear pricing structure and can’t support the most basic things like Maintenance pages ( all upcoming, but who knows when ). As it stands, it feels like a gigantic step backwards from Cloud Foundry Classic.
However, the one thing it does excellently is allowing you quickly put up versions of your application with zero configuration. This makes it the perfect environment to put up versions of our application for external API integration or demos. Instead of relying on clunky processes like SSH Tunneling to make sites visible, we simply deploy our application into Cloud Foundry. It is a good place to quickly throw applications up to see if it works.
Another advantage of Cloud Foundry is quick deployment. While other solutions like Amazon Elastic Beanstalk and Cloud Foundry classic requires you to upload the entire war file again ( something that might take 20+ minutes for our app ), Cloud Foundry only uploads changes to the application. This means that if you find that certain values are not right in your application, you can quickly change and update your application. Recently, it has become an excellent tool for external development.
Jenkins is a continous integration system with an impressive and extensive set of plugins. Aside from automatic builds on checkins and running functional tests, we also have Jenkins jobs to check system status and build new environments.
LXC is a Linux Container System conceptually similar to VirtualPC or VirtualBox. However, the way it is implemented means that you are not dealing with simulated virtual hardware, but rather with isolated linux processes. We use LXC to quickly spin up demo / qa environments for internal testing and showing off to potential stake holders.
Geb is a functional testing and browser automation framework that easily integrates with Grails. It wraps around the Webdriver framework and provides a groovy way to defining Page Objects and navigate across pages. You can see the way in which Secret Escapes uses Geb in this talk I gave last year at the Grails Exchange. The main advantage of Geb is that it enables us to automate and simulate an user clicking around in our site, making sure that any new functionality that we build is automatically tested.
Spock is a specification framework written in Groovy. It allows us to define our tests in a way that closely match the business requirements and eases readability and encourages Behaviour Driven Development. We find it very useful in helping us structure and define the code that we’re writing.
Build Test Data
Build test data is a fixtures plugin for Grails that simplifies the definition of test data. It is very useful as you can define only the fields you need for your domain objects in your tests, and it fills in the rest.
The grails remote control plugin allows you to remotely execute test code into a running server. The main advantage of this is that grails applications take a long time to start up and be ready to be tested. By using Remote Control in our functional tests, we can modify and alter all our tests at our heart’s content, inserting and removing data without ever needing to restart the main Grails application. You can read more about our technique here.
Helpful Grails Plugins
Shiro is an authentication plugin similar to the Spring-Security-Core plugin. It enables us to easily add access control to our application and ensure that only users with the right permissions can access the sections in our application they use.
The console plugin allows us to run arbitrary grails commands on our production systems. It is very useful to quickly query system state and get status reports. We also found it very useful in Cloud Foundry deployments to gain access to the underlying Linux system that is not externally available.
The database migration plugin keeps track of changes made to the databases used by our application. This becomes very important in our production systems as we want to make sure that no data is lost during migrations and we can restore systems to a correct database state. Using the database migration plugin ensures that any sort of data changes happen in a controlled and predictable way and guarantees data integrity.
The executor plugin enables you to split off tasks to be completed later in separate threads safely. We this plugin in our CMS to upload images and delay their resizing to separate threads since it might take a while for things to return.
The bean fields plugin provides a readable, easy to customize way of dealing with form fields within domain objects and GSP pages. We use this plugin extensively in our CMS instead of scaffolding. It allows the CMS pages for our sales, offers, allocations, etc to become more easily customized and easy to read.
Blueprint is a grid and CSS reset style sheet. We use this mostly in our CMS to simplify the placement of elements within a grid. It makes it much easier to edit and modify.
Tools we’re considering
Elastic Beanstalk / Cloud Foundry Classic
The future of hosting seems to be heavily shifting towards the cloud. While we like our current hosting providers, the traditional VPS model seems clunky compared to easy to deploy solutions like Elastic Beanstalk and Cloud Foundry.
Moving to Amazon Elastic Beanstalk would make our deployment process easier and cheaper. However, you can currently only deploy applications to servers hosted in Virginia, which adds an unacceptable lag to our UK customers. Once this limitation is shifted and we can create servers in the EU, we can see ourselves moving to the easy to use Elastic Beanstalk solution. ( Or Cloud Foundry Beta / Classic, if it ceases to be beta / abandonware )
Less CSS extends standard CSS to provide hierarchy, variables, mixins and a lot of goodies not possible with traditional CSS. We’re currently evaluating Less to allow us to more easily add custom themes to our application and simplify the css written for the site. You can see more about Less and Grails here.
One of our issues with the ui-performance plugin is that the framework seems to have been abandoned for more pressing issues in Grails 1.4. Given the shift to the resources framework, I can foresee a shift for us to Resources shortly.
JQuery Mobile / Sencha Touch / Flex
One of the things we don’t yet do well is to add mobile views for people coming in from iPads and iPhones. In the near future, we would like to add mobile views that can be visible both on the web and on a potential iPhone / Android app. Libraries like JQuery Mobile, Sencha Touch, PhoneGap or Adobe Flex would be useful for this purpose.
As you can see, even in a standard Grails application, there can be a lot of choice and interesting solutions for tools and software choices. If you would like to work with similar technologies, Secret Escapes is still hiring for a Senior Java Developer, so get in touch at tomas [ at ] secretescapes.com.