Tools used at a Grails Startup – technologies / infrastructure used at Secret Escapes

When I was interviewing a candidate recently, he asked about the hardware that we used at Secret Escapes. I replied that we used standard Linux boxen for our day to day development, but the question made me consider the vast amount of tools, plugins and services that our little website uses.

Secret Escapes is a private sales travel site. Our members sign up to our site and get the opportunity of booking hotels and vacation packages at the guaranteed lowest prices on the web in sales that only last for a week. You can read up more on what we do in this post written by one of our investors and in this article.

The heart of the Secret Escapes technology is Grails and Groovy. We are using Grails 1.3.7 and all our support and migration scripts are written in Groovy. It doesn’t seem like a lot of other Grails shops are sharing the tools and processes they use, so I thought it might be useful to catalog the ones we find useful in our every day development.

In this post, I want to highlight some of the tools we use, why we chose them, and how they help us solve problems.

Development Environment / Tools

Linux
The dev team at Secret Escapes uses, for the most part, Ubuntu. Our Jenkins server is running on Crunchbang and our graphic designer uses Windows for Photoshop / Illustrator.

At the beginning, we were all on Windows machines. However, we soon ran into a few Windows only issues that plagued some of the Grails extensions. The database migration plugin, for example, would include certain extra fields and parameters in changesets. We also ran into strange issues using webdriver with Geb, and some earlier versions of plugins like Remote Control were simply not coded with Windows in mind.

Gradually, our entire team switched to Linux and it proved to be a better development environment for Grails, where it seems most of the development team works on Macs, anyways ( we probably would if we had the budget ).

IntelliJ Idea
One of the pieces of software I pushed hard for us to license was IntelliJ Idea. We had a lot of issues with Eclipse / STS in its earlier incarnations, and the ease of use and Grails integration within Idea just seemed like a more friction-less decision to make. While IntelliJ seems to struggle with some newer features of Grails ( plugin-driven-development is particularly lacking, for example ), it is definitively a rich productivity enhancing tool for us. No other IDE in the market is easier to use nor more grails friendly than Idea.

Git Flow / Git
Secret Escapes launched in January. Over the next six months and thousand+ sales, we had to quickly add and improve the functionality of the site. Using Grails made this type of agile / quick enhancements fairly easy, but we quickly ran into roadblocks using SVN.

One of our main problems with SVN was keeping track of deployments and different versions of code. We would create a branch of our code to work on some feature, and then face great difficulties in changing our deployed codebase when a quick text or copy change was required. Fed up with the situation, we looked at alternatives and concluded that switching to Git would be a good idea.

However, our team had very little experience with Git. It felt daunting. Luckily, we came across the Git Flow branching model. Git Flow is a set of commands on top of git that enforce a master / develop / hotfix / feature model. It adds a layer of conventions over the total freedom provided by Git. It almost feels like the ‘Convention over Configuration’ brought into source control.

Overall, we’re very happy with Git and Git flow. The one place where it falls apart a bit is when you’re working with larger features and have to merge these back into development. But it is definitively more flexible than SVN and allows us to code fearlessly.

Pivotal Tracker
For our issue tracking, we chose to move away from JIRA and use Pivotal Tracker instead. Pivotal is a Kanban board that enables us to quickly track issues that are important. For our team, it allows our business counterparts to keep track of what we’re working on and, unlike a physical board, means that we can edit them outside of the office.

Hosting / Infrastructure

EveryCity
Our main hosting solution is EveryCity. They provide a cloud-based VPS service that enables us to quickly clone instances if our traffic ever spikes. They are a premium service but offer excellent speeds for our UK customers and have quite a good / responsive tech support and monitoring service.

Amazon S3 and CloudFront
For our CDN, we deploy to Amazon S3 and their CloudFront service using the JetS3t library (Although you might want to check out Lucas Teixeira’s Excellent Amazon AWS plugin for Grails).

Secret Escapes has a lot of quickly changing sales data. Imagine a product catalog that gets 4 new pages with glossy images added every two days. By putting this content in a CDN, we are able to ensure that static resources uploaded to our CMS system is immediately available to our developers as well.

By using a CDN, we no longer need to worry about moving static files from staging to production, and have the added bonus of being able to use the same data in our development / testing. ( We have a script for continous deployment that pulls down our database data ).

Additionally, we get all the quick download and headers expiry optimization available to CloudFront resources.

Gaelyk and the Google App Engine
Image resizing has always been an issue with any sort of content-rich website written in Java. We found that the default Grails solutions like Burning Image and ImageTools simply returned badly resized images. We wanted glossy sexy images.

For CruiselineFans.com, we implemented a resizing solution using ImageMagick. This solution required custom setup in our development environment as ImageMagick is not cross platform. There was no guarantee this wasn’t going to break across Windows / Mac / PCs. It was just really painful.

For Secret Escapes, we wrote a quick little resizing service using Gaelyk and the Google App Engine using the techniques described here. Gaelyk is a groovy-based framework specially designed for Google App Engine. Our code to resize images was less than 20 lines but it provided us with an immediate, cross platform way of creating high quality images. Given that we only resize a small amount of images in our CMS, this solution allowed us an elegant and portable way of solving the image size in grails problem.

Overall, we find that Gaelyk and the Google App Engine provides a great additional set of APIs and tools that might not be available otherwise. We would definitively not write our entire site with it, but if we needed to add features like Custom Handlers for Emails, it provides an excellent alternative.

Cloud Foundry
Cloud Foundry is VMWare’s new Cloud based solution for deploying all sorts of applications. It competes in the same space as Google App Engine, Amazon Elastic Beanstalk and Heroku / Azure.

First, let me clarify that only a fool would put a production environment on VMWare’s new beta Cloud Foundry solution in its current state. You can’t access the machines from the outside, it lacks a clear pricing structure and can’t support the most basic things like Maintenance pages ( all upcoming, but who knows when ). As it stands, it feels like a gigantic step backwards from Cloud Foundry Classic

However, the one thing it does excellently is allowing you quickly put up versions of your application with zero configuration. This makes it the perfect environment to put up versions of our application for external API integration or demos. Instead of relying on clunky processes like SSH Tunneling to make sites visible, we simply deploy our application into Cloud Foundry. It is a good place to quickly throw applications up to see if it works.

Another advantage of Cloud Foundry is quick deployment. While other solutions like Amazon Elastic Beanstalk and Cloud Foundry classic requires you to upload the entire war file again ( something that might take 20+ minutes for our app ), Cloud Foundry only uploads changes to the application. This means that if you find that certain values are not right in your application, you can quickly change and update your application. Recently, it has become an excellent tool for external development.

Testing

Jenkins
Jenkins is a continous integration system with an impressive and extensive set of plugins. Aside from automatic builds on checkins and running functional tests, we also have Jenkins jobs to check system status and build new environments.

LXC
LXC is a Linux Container System conceptually similar to VirtualPC or VirtualBox. However, the way it is implemented means that you are not dealing with simulated virtual hardware, but rather with isolated linux processes. We use LXC to quickly spin up demo / qa environments for internal testing and showing off to potential stake holders.

Geb
Geb is a functional testing and browser automation framework that easily integrates with  Grails. It wraps around the Webdriver framework and provides a groovy way to defining Page Objects and navigate across pages. You can see the way in which Secret Escapes uses Geb in this talk I gave last year at the Grails Exchange. The main advantage of Geb is that it enables us to automate and simulate an user clicking around in our site, making sure that any new functionality that we build is automatically tested.

Spock
Spock is a specification framework written in Groovy. It allows us to define our tests in a way that closely match the business requirements and eases readability and encourages Behaviour Driven Development. We find it very useful in helping us structure and define the code that we’re writing.

Build Test Data
Build test data is a fixtures plugin for Grails that simplifies the definition of test data. It is very useful as you can define only the fields you need for your domain objects in your tests, and it fills in the rest.

Remote Control
The grails remote control plugin allows you to remotely execute test code into a running server. The main advantage of this is that grails applications take a long time to start up and be ready to be tested. By using Remote Control in our functional tests, we can modify and alter all our tests at our heart’s content, inserting and removing data without ever needing to restart the main Grails application. You can read more about our technique here.

Helpful Grails Plugins

Shiro
Shiro is an authentication plugin similar to the Spring-Security-Core plugin. It enables us to easily add access control to our application and ensure that only users with the right permissions can access the sections in our application they use.

Console
The console plugin allows us to run arbitrary grails commands on our production systems. It is very useful to quickly query system state and get status reports. We also found it very useful in Cloud Foundry deployments to gain access to the underlying Linux system that is not externally available.

Database Migration
The database migration plugin keeps track of changes made to the databases used by our application. This becomes very important in our production systems as we want to make sure that no data is lost during migrations and we can restore systems to a correct database state. Using the database migration plugin ensures that any sort of data changes happen in a controlled and predictable way and guarantees data integrity.

UI Performance
The UI Performance plugin minifies, versions and zips up CSS / JavaScript and image files. It is useful to help us minimize page load.

Executor
The executor plugin enables you to split off tasks to be completed later in separate threads safely. We this plugin in our CMS to upload images and delay their resizing to separate threads since it might take a while for things to return.

Bean-fields
The bean fields plugin provides a readable, easy to customize way of dealing with form fields within domain objects and GSP pages. We use this plugin extensively in our CMS instead of scaffolding. It allows the CMS pages for our sales, offers, allocations, etc to become more easily customized and easy to read.

Blueprint
Blueprint is a grid and CSS reset style sheet. We use this mostly in our CMS to simplify the placement of elements within a grid. It makes it much easier to edit and modify.

Tools we’re considering

Elastic Beanstalk / Cloud Foundry Classic
The future of hosting seems to be heavily shifting towards the cloud. While we like our current hosting providers, the traditional VPS model seems clunky compared to easy to deploy solutions like Elastic Beanstalk and Cloud Foundry.

Moving to Amazon Elastic Beanstalk would make our deployment process easier and cheaper. However, you can currently only deploy applications to servers hosted in Virginia, which adds an unacceptable lag to our UK customers. Once this limitation is shifted and we can create servers in the EU, we can see ourselves moving to the easy to use Elastic Beanstalk solution. ( Or Cloud Foundry Beta / Classic, if it ceases to be beta / abandonware )

Less
Less CSS extends standard CSS to provide hierarchy, variables, mixins and a lot of goodies not possible with traditional CSS. We’re currently evaluating Less to allow us to more easily add custom themes to our application and simplify the css written for the site. You can see more about Less and Grails here.

Resources Plugin
One of our issues with the ui-performance plugin is that the framework seems to have been abandoned for more pressing issues in Grails 1.4. Given the shift to the resources framework, I can foresee a shift for us to Resources shortly.

JQuery Mobile / Sencha Touch / Flex
One of the things we don’t yet do well is to add mobile views for people coming in from iPads and iPhones. In the near future, we would like to add mobile views that can be visible both on the web and on a potential iPhone / Android app. Libraries like JQuery Mobile, Sencha Touch, PhoneGap or Adobe Flex would be useful for this purpose.

Summary

As you can see, even in a standard Grails application, there can be a lot of choice and interesting solutions for tools and software choices. If you would like to work with similar technologies, Secret Escapes is still hiring for a Senior Java Developer, so get in touch at tomas [ at ] secretescapes.com.

About these ads

22 thoughts on “Tools used at a Grails Startup – technologies / infrastructure used at Secret Escapes

  1. Stephen Westfall

    Excellent Post. Our development environments are very similar. I was un-aware of the Bean Fields plugin, so thank you for that. I’m going to grab it now. :)

    Reply
  2. Jan

    Hi,

    great article. We were considering LESS, too, but eventually chose SASS because it seemed like more people use it.

    Reply
  3. simoncomley

    Great article. So useful to see what environments and plugins real world applications use, I am happy to see that for most part we have made similar choices.

    I am curious as to your choice of Shiro over the traditional Security plugin, what made you swing in that direction?

    LESS has caught my eye recently but I haven’t actually used it yet.

    Reply
    1. Tomas Lin Post author

      For us, it was really more a matter of familiarity vs. features. We had used Shiro / JSecurity for a few projects already. I would say that the nice modular nature of Spring Security Core + plugins might make it a better choice for future projects, specially if the Spring Social plugin delivers what it promises.

      Reply
    2. Stephane Maldini

      Great article Tomas, I nearly use the same tools and I very enjoy them. Sometimes I wonder why there isn’t a grails plugin megapack with all the must have plugins dependances + some goodies.

      @simoncomley
      One thing to consider when choosing btw Shiro or S2 is the ACL system. I often found myself having headhaches with the S2 ACL module. On the other hand, Shiro offers a very comprehensive permissions system. I would go for S2 for support, plugins and/or Role-centric security. I would choose Shiro for the permissions and its easily pluggable identity sources (“Realm”).

      Reply
  4. William

    Great article. Very interesting to see what development teams are using in the real world. I will be looking into Git Flow. Thanks.

    Reply
  5. Pingback: Blog bookmarks 06/27/2011 « My Diigo bookmarks

  6. Mel

    Nice write up!

    We have a family of customer destined applications based on grails (core functionality is a plugin so targeted applications just use the plugin).

    I used eclipse (STS) for a long time and switched to intellij for the same reasons as you. I’ve switched back to eclipse. Grails support in STS is excellent now that grails is part of SpringSource.

    Reply
  7. Pingback: GroovyMag - the magazine for Groovy and Grails developers about Groovy programming and Grails development

  8. John Rellis

    Awesome, thanks for sharing, as you have said, there’s not much of this real world info out there so it is appreciated. Pretty sweet that you use Gaelyk too, sounds like a great place to be developing :) considering using GIT for a personal project so GIT flow looks like a good place to start! Thanks!

    Reply
  9. simoncomley

    @Tomas Lin and @smaldini Thanks for your responses/warnings regards Security plugin and ACL

    I actually just read the article again and there is a particular line : “quickly clone instances if our traffic ever spikes” that keeps nagging at me because I don’t know how you do that.

    I would love to know more about your setup specifically regards that.
    Did you start off setup for that or is something that you introduced later when you found out this Secret Escapes was doing well?
    What DB are you using?
    Are you using Terracotta?

    Sorry Tomas, when it comes to clustering and things I’m a little in the dark

    Any useful resources you folks used would be great

    Reply
  10. Lucas Teixeira

    Awesome post Tomas.

    It is a great to see people sharing about they dev environment, this kind of post helps a lot beginners to understand how achieve a ‘real world’ environment with grails. I’ll follow you and post something similar about our environment here, although I bet will be much smaller than this.

    Thanks for referencing the AWS plugin.

    []s,

    Reply
  11. Antoine

    Great post thanks! I didn’t know LXC, I will definitely give it a look.

    @smaldini Internally we have a Grails plug-in that pulls all those interesting plug-ins (spock, maven-deploy, etc…). With that, I can upgrade the dependency in the plug-in once and everyone benefits from it. Your idea is good, that could be kind of “distributions” for Grails.

    Reply
  12. Pingback: An Army of Solipsists » Blog Archive » This Week in Grails (2011-26)

  13. Ssaraswat

    We are developing a grails based application and are looking for a plug in that would enable us to convert a list into a pipe limited format. Any idea if there is a custom converter availaible for the same .

    Thanks .. Smita

    Reply
  14. Felipe Nascimento

    If you may, share with us some info about your architecure and scale. Things like: how many boxes tgere are in production environment; do you separate web container from database; do you use any cache solution (memcached, ehcache, …); what data persistence solution do you guys use (SQL, NoSQL, Mysql, MongoDb, Cassandra, whatever, …); any load balancing?
    And for the production setup you have, what is the load it supports? (page views, request per second, or any metric about your audience).
    Tks a lot.

    Reply
  15. Wendy Chamier

    Hi Tomas – thanks for all the detail. Forgive my ignorance as not familiar with all the grails stuff, but have you built your own booking engine or are you interfacing with an off the shelf? If off the shelf, which do you use?

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s