For the Payments Team's front-end projects we've just landed support for running the front-end unit tests using Sauce Labs.
About Sauce Labs
In case you've never heard of Sauce Labs (unlikely!) they provide access to a multitude of browser/platform combinations via vms so you can run your automated test suite (or do manual testing) and get comprehensive cross browser coverage fairly easily.
Open source projects can get an 'Open Sauce' account for free. You can find out more on the 'Open Sauce' account here. Once you sign-up you will have a username and access key that are used to get access to the service.
Setting up the test runner
We're already using the excellent Karma as our test runner. Karma is essentially a test runner which browsers connect to in order to run a suite of unit tests.
As we're using React we're tending to use unit tests a lot more than before. This is because unit testing in React is really straightforward and it saves us having to worry about end-to-end (E2E) testing up front like we did before with Casper and PhantomJS etc. (We're still planning some integration tests but they can come later). Putting more effort into unit testing feels like the right thing to do. The tests are much more self-contained and therefore less brittle than E2E tests.
To get our existing Karma tests running with sauce labs we used the karma-sauce-loader which handles the setup of a tunnel and all the connections from each Sauce labs browser into the karma test runner.
To DRY things up we have a shared karma config which is just a module exporting the main options object. This is then used by a simple karma config karma.conf.js which just runs tests on Firefox and the sauce labs karma config file karma.conf.sauce.js.
Handling PRs conditionally
The access key and username are used via env vars in our karma config in order to allow our automated tests to run on Sauce labs.
Using TravisCI as our CI you can encrypt the SAUCE_ACCESS_KEY and SAUCE_USERNAME env vars so you don't publically expose them (Which would otherwise anyone to run tests on your account). See this doc for more details: Encryption Keys (docs.travis-ci.com).
But, if you do that, when you propose a PR from your fork of the project it won't be able to run the tests on Sauce Labs as encrypted secrets aren't exposed to pull requests that aren't made from branches on the main repo.
To work around this we can run the tests conditionally by looking at the TRAVIS_SECURE_ENV_VARS env var like so:
// Conditionally run saucelabs tests if we have the
// secure vars or a normal test run if not.
grunt.registerTask('test-ci', function() {
if (process.env.TRAVIS_SECURE_ENV_VARS === 'true') {
grunt.log.writeln('Running full SauceLabs test suite');
grunt.task.run('test-sauce');
} else {
grunt.log.writeln('Running limited test suite');
grunt.task.run('test');
}
});
If it's a PR from a fork we just run the standard tests on Firefox on Travis (because TRAVIS_SECURE_ENV_VARS is 'false').
Otherwise if the PR is from a branch on the main repo or a commit on master, the tests will be run by our pre-defined set of browsers on Sauce Labs.
Some projects just commit their public access keys but I feel it's not a good idea to propagate a bad practice like that. This workaround feels like a fair compromise.
If any of the team end up working on something that warrants extra testing before landing we can always push the branch to the main repo first and make a PR from that.
Sharing the browser config
To make it easier to share the browser config (which tells Sauce labs which browser versions and platform combinations we want) we have a node package which contains a simple config object. This module can be shared across all the projects that need to run tests on the same set of browsers.
For now we're testing the last two versions of Firefox, Chrome, Safari and IE but we can tweak this as we move forward.
Conclusion
Using Sauce labs makes running your existing testsuite on a broad set of platform and browser combinations really straight-forward for a minimal investment in setup time.
In our case karma-sauce-launcher made it really easy.