Taming SlimerJS

We've been using CasperJS for E2E testing on Marketplace projects for quite some time. Run alongside unittests, E2E testing allows us to provide coverage for specific UI details and to make sure that users flows operate as expected. It also can be used for writing regression tests for complex interactions.

By default CasperJS uses PhantomJS (headless webkit) as the browser engine that loads the site under test.

Given the audience for the Firefox Marketplace is (by a huge majority) mainly Gecko browsers, we've wanted to add SlimerJS as an engine to our CasperJS test runs for a while.

In theory it's as simple as defining an environment var to say which Firefox you want to use, and then setting a flag to tell casper to use slimer as the engine. However, in practice retrofitting Slimer into our existing Casper/Phantom setup was much more involved and I'm now the proud owner of several Yak hair jumpers as a result of the process to get it working.

In this article I'll list out some of the gotchas we faced along with how we solved them in-case it helps anyone else trying to do the same thing.

Require Paths and globals

There's a handy list of the things you need to know about modules in the slimer docs.

Here's the list they include:

• global variables declared in the main script are not accessible in modules loaded with require
• Modules are completely impervious. They are executed in a truly javascript sandbox
• Modules must be files, not folders. node_modules folders are not searched specially (SlimerJS provides require.paths).

We had used this pattern throughout our casperJS tests at the top of each test file:

var helpers = require('../helpers');

The problem that you instantly face using Slimer is that requires need an absolute path. I tried a number of ways to work around it without success. In the end the best solution was to create a helpers-shim to iron out the differences. This is a file that gets injected into the test file. To do this we used the includes function via grunt-casper.

An additional problem was that modules are run in a sandbox and have their own context. In order to provider the helpers module with the casper object it was necessary to add it to require.globals.

Our shim looks like this:

var v;
if (require.globals) {
  // SlimerJS
  require.globals.casper = casper;
  casper.echo('Running under SlimerJS', 'WARN');
  v = slimer.version;
  casper.isSlimer = true;
} else {
  // PhantomJS
  casper.echo('Running under PhantomJS', 'WARN');
  v = phantom.version;
}
casper.echo('Version: ' + v.major + '.' + v.minor + '.' + v.patch);
/* exported helpers */
var helpers = require(require('fs').absolute('tests/helpers'));

One happy benefit of this is that every test now has helpers on it by default. Which saves a bunch of repetition \o/.

Similarly, in our helpers file we had requires for config. We needed to workaround those with more branching. Here's what we needed at the top of helpers.js. You'll see we deal with some config merging manually to get the config we need, thus avoiding the need to workaround requires in the config file too!

if (require.globals) {
  // SlimerJS setup required to workaround require path issues.
  var fs = require('fs');
  require.paths.push(fs.absolute(fs.workingDirectory + '/config/'));
  var _ = require(fs.absolute('node_modules/underscore/underscore'));
  var config = require('default');
  var testConf = require('test');
  _.extend(config, testConf || {});
} else {
  // PhantomJS setup.
  var config = require('../config');
  var _ = require('underscore');
}

Event order differences

Previously for phantom tests we used the load.finished event to carry out modifications to the page before the JS loaded.

In Slimer this event fires later than it does in phantomjs. As a result the JS (in the document) was already executing before the setup occured which meant the test setup didn't happen before the code that needed those changes ran.

To fix this I tried alot of the other events to find something that would do the job for both Slimer and Phantom. In the end I used a hack which was to do it when the resource for the main JS file was seen as received.

Using that hook I fired used a custom event in Casper like so:

casper.on('resource.received', function(requestData) {
 if (requestData.url.indexOf('main.min.js') > -1) {
   casper.emit('mainjs.received');
 }
});

Hopefully in time the events will have more parity between the two engines so hacks like these aren't necessary.

Passing objects between the client and the test

We had some tests that setup spys via SinonJS which passed the spy objects from the client context via casper.evaluate (browser) into the test context (node). These spys were then introspected to see if they'd been called.

Under phantom this worked fine. In Slimer it seemed that the movement of objects between contexts made the tests break. In the end these tests were refactored so all the evaluation of the spys occurs in the client context. This resolved that problem.

Adding an env check to the Gruntfile

As we use grunt it's handy to add a check to make sure the SLIMERJSLAUNCHER env var for slimer is set.

if (!process.env.SLIMERJSLAUNCHER) {
	grunt.warn('You need to set the env var SLIMERJSLAUNCHER to point ' +
    		   'at the version of Firefox you want to use\n See http://' +
               'docs.slimerjs.org/current/installation.html#configuring-s' +
               'limerjs for more details\n\n');
}

Sending an enter key

Using the following under slimer didn't work for us:

this.sendKeys('.pinbox', casper.page.event.key.Enter);

So instead we use a helper function to do a synthetic event with jQuery like so:

function sendEnterKey(selector) {
  casper.evaluate(function(selector) {
    /*global $ */
    var e = $.Event('keypress');
    e.which = 13;
    e.keyCode = 13;
    $(selector).trigger(e);
  }, selector || 'body');
}

Update: Thursday 19th Feb 2015

I received this info:

@muffinresearch about your blog post: for key event, use key.Return, not key.Enter, which is a deprecated key code in DOM

— SlimerJS (@slimerjs) February 19, 2015

Using casper.back() caused tests to hang.

For unknown reasons using casper.back() caused tests to hang. To fix this I had to do a hacky casper.open() on the url back would have gone to. There's an open issue about this here: casper.back() not working script appears to hang.

Exit code is always 0

An eagle-eyed colleagure spotted that the exit code was always 0 via casper when setting SlimerJS up for the marketplace frontend tests. Fortunately the patch for this is coming soon in 0.10. The workaround for this today is to use the 0.10pre version. Hattip @markstrmr

The travis.yml

Getting this running under travis was quite simple. Here's what our file looks like (Note: Using env vars for versions makes doing updates much easier in the future):

Yes that is Firefox 18! Gecko18 is the oldest version we still support as FFOS 1.x was based on Gecko 18. As such it's a good Canary for making sure we don't use features that are too new for that Gecko version.

In time I'd like to look at using the env setting in travis and expanding our testing to cover multiple gecko versions for even greater coverage.