This week ushers in our first CI (continuous integration) server at Stylight. With almost 300 tests running, we want to ensure the feedback process from end-user to developer is as automatic and fast as possible. Just how fast? How does under three minutes sound? We achieved this using a variety of tools, platforms and web services. If you’ve never heard of Jenkins, Selenium or HipChat read on for a short introduction to how we joined them together to build and deploy a high-quality website even faster!
Jenkins builds the codebase, executes tests and keeps score
We setup a github webhook for the “bootstrap” Jenkins job which causes a build to be kicked off with every commit to our development branch. Once the new development code is deployed, it’s time for junit and acceptance testing. As we want fast feedback cycles, we don’t run our entire Selenium suite with every deployment. Instead, we run the most business critical tests in order to alert the team within three minutes of a commit to any problems. The regression test suite is run via a Jenkins cronjob after 10pm. This way, we are free to improve test coverage but still quickly identify problems.
As Jenkins stores every test result, it’s easy for the team to track quality over time. Tricks like @Ignoring tests are also prominently displayed to keep everyone honest.
When the team can instantly and easily see their improvements to the codebase with each commit, it creates a natural, positive spiral out of technical debt.
Selenium tests run in the cloud for extensive browser coverage
Selenium browser tests have been around for almost a decade – a tried-and-true mechanism for automatically smoke-testing your website. As JavaScript becomes ever more prevalent on websites, selenium is the quickest way to dig in and make sure all those bells and whistles are still working. While other JavaScript acceptance test frameworks are catching up (PhantomJS comes to mind), the creator of selenium, Jason Huggins, took the next logical step with SauceLabs – a compelling webservice for scaling your acceptance tests across a variety of operating systems and browsers.
We use maven and surefire to control our junit based selenium tests and multi-threaded scaling awesomeness with SauceLabs can be had with the <parallel> and <threadCount> configuration items in your pom.xml:
[gist]<a href=”https://gist.github.com/7155984“>https://gist.github.com/7155984</a>[/gist]
This simple setting changed our total test time on Jenkins for the regression suite from 1hr 47mins down to 13mins – truly webscale!
Hipchat notifies the team by providing situation awareness
All the above automated processes mean nothing without notifications. Hipchat provides a super-integrated, team IIRC chat that runs in a browser as well as providing clients for Windows, Mac & Linux. With mature plugins to github, jenkins and JIRA it was a natural choice for us. The results of every build and test job, as well as the github commits that triggered them, are dropped into the online chat for the entire team to see (or not see if they decide).
What’s even nicer about this is the ability to search and “catch-up” with what the team has been doing from anywhere and at anytime. This increases the situational awareness and helps foster communication not only within the development team but whomever may want to drop in and see what’s going on.
Under the hood, there’s a lot of technical configurations in place to enable this process flow. We use Puppet to setup and maintain our Jenkins environment, but there are many manual steps required to get the setup running smoothly. For example, setting up a jenkins user in your Github organization with pull-only access to your repositories and cronjob considerations for regression tests and automated deployments. If you want more details, give a shout-out to @stylight_eng!
