Automated Cross Browser Testing: The Holy Grail

Update: One of the authors of Buster.js helpfully responded to this article on Twitter, pointing out that a new version had been released since I'd tried it. This new version addresses IE support, which was one of my major qualms as mentioned later in the post.

When writing module.js, I wanted to set up some automated cross-browser testing. Firing up VMs is tedious and slow, but for a front-end JavaScript library, making sure it works in all the browsers you intend to support is paramount.

It's probably worth saying now that I'm focussing on functional testing here, not the inspection of styles. As far as debugging your CSS is concerned, I'm afraid my best suggestion is still to fire up a VM or use service that does that for you and lets you interact (i.e. BrowserStack, Browserling).

Testing by hand (i.e just running code and inspecting that it did what was expected) is a start, but after every tiny change to your front-end JavaScript, can you honestly have the discipline (and the time) to fire it up in every browser and fully test its functionality? If you do, I congratulate you, and apologise because the rest of this article might bore you.

Unit and acceptance tests help to catch regressions – bugs that are introduced by changes to the code: adding features, fixing other bugs or refactoring. They help to sanity check that your program does what it is intended to do. They mean that you as the author have to use your program, casting scrutiny over its API.

Importantly for open source projects, tests promote trust in your program – their existence alone is added assurance for most users, while the ability to run them enables the more sceptical user to verify that your program fulfils its claims. If you're releasing your OS project on GitHub, hoping for community input, then tests can help make sure pull requests don't break things. Also, well written tests can provide great example usage for your code, augmenting any documentation, readme and examples you might write.

Now that I've convinced you that testing is a good idea, and that opening up different browsers is a pain, hopefully now you can share my desire for automated cross-browser testing.

Testling

My first port of call was a service called Testling by StackVM. StackVM's main product (it seems) is Browserling: a service which lets you operate different browsers from within your own. Testling is Browserling's little brother – a service which sits on HTTP, allowing you to send a payload of tests to run in the selection of browsers that you specify.

Let me say now, that if Testling had worked for me, it would have been the holy grail (and this article would be called something 'TDD: Testling Driven Development', or some other witty pun).

However, I was fraught with silent hanging tests (Had they failed? Why? Had they not failed? Why were they taking so long?). These munched up my minutes, of which Testling's free tier gets you 30/month, and I wasn't ready to jump to $20/month for something that wasn't yet working for me. Another show-stopping issue that occurred when asking Testling to run tests files with more than two or three tests in various versions of IE was 'Failed to compile – Instrumentation may have failed'. There is an option to switch instrumentation off, so I did, resulting in a terse message of 'Syntax error' – leaving me baffled, since running the same file in different browsers worked, and each of the tests alone worked in IE. Anyway, I digress–I'd love to hear that I'm doing something wrong, but unfortunately I think these problems are a sign of Testling being a young and fledgling service. Again, let me re-iterate: I believe that a mature Testling would be the ideal solution. For now though, it's frustratingly close.

BusterJS

Buster is another new, 'beta' labelled approach to cross browser testing. Currently Buster requires that you start a server on your local machine, fire up all of the browsers that you want to test in (including from within VMs) and have them 'captured'. Once captured, a browser becomes a 'slave' to the Buster server, and they can be used to repeatedly run your tests. This is a pretty laborious thing to do compared with Testling, plus you have to stock up on your own array of VMs and browsers. However, in their roadmap, the team behind Buster aim to hook up with the BrowserStack API (currently in beta) so that you can leave the nitty-gritty of browser acquisition and OS virtualisation up to the experts, and concentrate on running your tests.

Getting up and running with Buster was pretty straightforward (especially being a Node developer – as the Buster server is a Node module); things were going swimmingly. Then the major blocker was discovered: the browser capture feature doesn't work in < IE8. Reading up on the issue tracker, it seems they are keen to deliver good browser support, but it's just not been achieved yet. An alternative 'static' server is offered, which instead of capturing the browser, will just serve up the tests to run within the browser. This works in said browsers, but requires manually navigating to the page and hitting refresh–that doesn't sound automated to me! Once Buster integrates with BrowserStack, I think it'll be a real rival for Testling (though I think as a CLI junkie I'd prefer Testling) but for now, no cigar.

Next…

My next port of call will be to try the vanilla BrowserStack API for myself, as the kind people there have given me Beta access.

This is where I need you. Have you solved this problem for yourself? If so, share the wisdom so fellow developers like myself can achieve enlightenment! Thanks for reading, and I look forward to responding to your comments.