A quick tip. If you're running karma tests via Firefox on Travis CI by default the Firefox version used is quite old (31 at time of writing).

To always use the latest (current) version you can add this snippet to your .travis.yml.

addons:  
  firefox: latest

This will keep your firefox up-to-date as new versions are released. Two other special release aliases are available:

• latest-esr - The latest extended support release.
• latest-beta - The latest beta version.

You can of course specify a specific version like so:

addons:  
  firefox: "44.0"

For for info see the Travis docs on setting firefox versions here.

JavaScript has always made it hard to deal with long strings, especially when you need to format them across multiple lines. The solutions have always been that you need to concatenate multiple strings like so:

var foo = "So here is us, on the " +  
  "raggedy edge. Don't push me, " +
  "and I won't push you.";

Or use line-continuations:

var foo = "So here is us, on the \  
  raggedy edge. Don't push me, \
  and I won't push you.";

Neither of these are pretty. The latter was originally strictly illegal in ECMAScript (hattip Joseanpg):

A 'LineTerminator' character cannot appear in a string literal, even if preceded by a backslash \. The correct way to cause a line terminator character to be part of the string value of a string literal is to use an escape sequence such as \n or \u000A.

There's a rather wonderful bug on bugzilla discussing the ramifications of changing behaviour to strictly enforce the spec back in 2001. In case you're wondering it was WONTFIXED.

In ECMAScript 5th edition this part was revised to the following:

A line terminator character cannot appear in a string literal, except as part of a LineContinuation to produce the empty character sequence. The correct way to cause a line terminator character to be part of the String value of a string literal is to use an escape sequence such as \n or \u000A.

Anyway, enough with the history lesson, we now have features in ES6 on the horizon and if you're already starting to use ES6 via the likes of babel or as features become available then it provides some interesting possibilities.

Diving in with ES6 template strings

Having seen template strings I jumped right in only to find this happening.

var raggedy = 'raggedy';  
console.log(`So here is us, on the  
             ${raggedy} edge. Don't push
             me, and I won't push you.`);

So that looks neat right? But hang-on what does it output?

So here is us, on the  
             raggedy edge. Don't push
             me, and I won't push you.

Dammit - so by default template strings preserve leading whitespace like heredoc in PHP (which is kind of a memory I'd buried).

Ok so reading more on the subject of es6 template strings, processing template strings differently is possible with tag functions. Tags are functions that allow you to process the template string how you want.

Making a template tag

With that in mind I thought OK, I'm going to make a tag to make template strings have no leading whitespace on multiple lines but still handle var interpolation as by default.

Here's the code (es6) (note a simpler version is linked to at the end of this post if you're not using full-fat es6 via babel):

export function singleLineString(strings, ...values) {  
  // Interweave the strings with the
  // substitution vars first.
  let output = '';
  for (let i = 0; i < values.length; i++) {
    output += strings[i] + values[i];
  }
  output += strings[values.length];

  // Split on newlines.
  let lines = output.split(/(?:\r\n|\n|\r)/);

  // Rip out the leading whitespace.
  return lines.map((line) => {
    return line.replace(/^\s+/gm, '');
  }).join(' ').trim();
}

The way tags work is they're passed an array of strings and the rest of the args are the vars to be substituted. Note: as we're using ES6 in this example we've made values into an array by using the spread operator. If the original source string was setup as follows:

var two = 'two';  
var four = 'four';  
var templateString = `one ${two} three ${four}`;  

Then strings is: ["one ", " three ", ""] and values is ["two", "four"]

The loop up front is just dealing with interleaving the two lists to do the var substitution in the correct position.

The next step is to split on newlines to get a list of lines. Then the leading whitespace is removed from each line. Lastly the list is joined with a space between each line and the result is trimmed to remove any trailing space.

The net effect is that you get an easy way to have a long multiline string formatted nicely with no unexpected whitespace.

var raggedy = 'raggedy';  
console.log(singleLineString`So here is us, on the  
            ${raggedy} edge. Don't push
            me, and I won't push you.`);

Outputs:

So here is us, on the raggedy edge. Don't push me, and I won't push you.  

Trying this out in the browser

As template strings are becoming more and more available - here's a version without the fancy ES6 syntax which should work in anything that supports template strings:

See http://jsbin.com/yiliwu/4/edit?js,console

All in all it's not too bad. Hopefully someone else will find this useful. Let us know if you have suggestions or alternative solutions.

The tests looked like this:

def test_user_identification_is_valid_option(self):  
   with self.assertRaises(ValueError):
       example_seller(products=[
           'id': 'hai',
           'description': 'a description',
           'recurrence': None,
           'user_identification': True,
       }])


def test_user_identification_is_required(self):  
     with self.assertRaises(ValueError):
         example_seller(products=[{
             'id': 'hai',
             'description': 'a description',
             'recurrence': None,
         }])

I realised that it was really easy to be tricked by other parts of the code that might raise the same exception. For example, in this case the required code is raising ValueError separately.

In JavaScript using assertions in chai, assert.throws takes a function that should throw an exception, the 2nd arg is the exception instance you're expecting and the third arg can be a regex to match the string passed to the exception.

I wanted to do the same thing in Python.

assertRaisesRegexp

In Python (2.7+) assertRaisesRegexp does exactly what we want so we can fix our tests to be more explicit like so:

def test_user_identification_is_valid_option(self):  
    with self.assertRaisesRegexp(ValueError, 'user_identification must'):
       example_seller(products=[
           'id': 'hai',
           'description': 'a description',
           'recurrence': None,
           'user_identification': True,
       }])


def test_user_identification_is_required(self):  
     with self.assertRaisesRegexp(ValueError, 'Missing user_ident'):
         example_seller(products=[{
             'id': 'hai',
             'description': 'a description',
             'recurrence': None,
         }])

The second argument can either be a a regular expression object or a string containing a regular expression suitable for use by re.search().

With this in place we can be sure our tests are asserting on the actual part of the code that raises the exception not just asserting that any old ValueError is good enough.

I've just recently finished shaving about a billion yaks * to convert a React app over to use ES6 modules and classes so we can start living in the future that is ES6 with a sprinkling of ES7.

* Might not be true

Transpiling back to the present

We're using babel via webpack to transpile our ES6+ code into ES5. Babel exposes the various stages of ECMAScript proposals so you can choose whatever stages are appropriate for your project. For example Stage 0 will give you the most features, but that will include things like strawman proposals that may never actually see the light of day.

For our project we've tried to stick to Stage2 which is Babel's default (Stage 2 features Draft specs), and then add to that a couple of more modern features which should hopefully make it to prime time.

The two features we've specifically added are:

• es7.objectRestSpread from Stage 1. (A must for JSX e.g. {...otherProps})
• es7.classProperties from Stage 0 - more on this one in a bit

ES6 Modules

Es6 Modules are very powerful, it takes a bit of getting used to coming from commonJS or AMD modules.

One of the really great things is that you can export more than one thing by using a default export and exporting something else.
In this example the instance will be the default export that you'll get when importing this module:

E.g.

export class Tracking {  
    // Whatever
}

export default new Tracking();  

So now I have two options:

// Get an instance
import tracking from tracking;  
tracking.init();

// Get the class (handy for tests)
import { Tracking } from tracking;  
var tracking = new Tracking();  
tracking.init();  

There's lots of great info on the full range or what's possible with es6 modules here.

React components with ES6 classes

When converting from ES5 there's quite a number of things to be aware of.

Setting default state

You can easily do this in the constructor rather than using a separate getInitialState method.

import React, { Component } from 'react';

class MyComponent  extends Component {  
    constructor() {
        super();
        this.state = {
            // Default state here.
        }
    }

    //... Everything else  
}

Setting defaultProps and propTypes

When using defaultProps and propTypes are defined as static class properties.

If you have es7.classProperties you can do this like so:

import React, { Component, PropTypes } from 'react';

class MyComponent  extends Component {

    static propTypes = {
        foo: PropTypes.func.isRequired,
        bar: PropTypes.object,
    }

    static defaultProps = {
        foo: () => {
               console.log('hai');
        },
        bar: {},
    }

    //... Everything else
}

The alternative to this (when you don't have es7 shiny enabled) is to just set props directly on the class e.g:

class MyComponent  extends Component {  
    //... Everything else
}
MyComponent.propTypes = {foo: PropTypes.func.isRequired};  
MyComponent.defaultProps = {foo: function() { console.log('hai'); }};  

I found this very ugly though, so static class props seems to be a good way to go for now.

isMounted() isn't available

If you'd used isMounted() then this isn't availabe on ES6 classes extended from Component. If you really need it you can set a flag in componentDidMount and unset it again in componentWillUnMount.

Method binding

When passing methods around as props you'll find with ES6 classes this doesn't refer to the component like it used to with React.createClass.

To get back the same behaviour without using something ugly like:

<Foo onClick={this.handleClick.bind(this)} />  

You can use an arrow function instead:

class MyComponent  extends Component {  
    handleClick = (e) => {
        // this is now the component instance.
        console.log(this);
    }
}

This works because arrow functions capture the this value of the enclosing context. In otherwords an arrow function gets the this value from where it was defined, rather than where it's used.

If you want to get really fancy you can use "function bind syntax".

Which you get by doing this:

<Foo onClick={::this.handleClick} />  

Which is equivalent to the previous bind() example. But to do that you'd need to enable es7.functionBind. For me this was a step too far and I'm happy to stick with the arrow functions.

Testing with ES6/7

Our previous tests had made some good use of rewire.js to modify requires and vars that weren't exported from the module (It does this with some cheeky injection techniques).

Unfortunately due to some changes in Babel rewire isn't compatible (at the moment) with ES6 modules. There's an alternative babel-plugin-reqire but that injects getters and setters into every module.

Dependency Injection in React Classes

Instead of using module mangling it was easiest to fall-back to dependency injection for the places where we were changing Components to fake ones.

In React props look like a good fit for dep injection:

import DefaultOtherComponent from 'components/other';

class MyComponent extends Component {

    static defaultProps: {
        OtherComponent: DefaultOtherComponent,
    }

    //... Snip

    render() {
        var OtherComponent = this.props.OtherComponent;
        return (
          <OtherComponent {...this.props} />
        );
    }
}

Now in the test we can inject a fake component:

TestUtils.renderIntoDocument(  
  <MyComponent OtherComponent={FakeOtherComponent} />
);

getDOMNode() is deprecated

In ES6 React classes getDOMNode() is not present so you'll need to refactor all calls to MyComponent.getDOMNode() to React.findDOMNode(MyComponent) instead. The docs also mention it's deprecated so it might be removed completely in the future.

Summary

There's quite a bit of work in making the conversion depending on how big your code-base is and depending on how much use of ES6 you're making already.

A lot of the new features in ES6 are making things much easier for developers so I'd definitely say it's a direction worth moving in assuming you're comfortable with all the aspects of transpilation. Webpack + babel makes this pretty straightforward. If you're already using babel for JSX (or some other similar loader for JSX) then you're already most of the way there.

FWIW: if you're not using babel, switching to it is now an official recommendation.

I'm very impressed with the currently ecosystem around JS e.g. tools like babel and webpack alongside forward thinking libs like React + Redux. All of these things sit well with each other and allow projects like ours to be able to step into the future today.

Now we've just got to get used to all the new syntax and start using more of it!

Further reading

• React docs on ES6 classes
• Component API docs - has lots of info on what parts of the API are different for ES6 classes
• Babel blog post on using ES6+ - useful blog post with old and new code examples.

Credits

• Image By Terabass (Own work) CC BY-SA 4.0, via Wikimedia Commons

For the Payments Team's front-end projects we've just landed support for running the front-end unit tests using Sauce Labs.

About Sauce Labs

In case you've never heard of Sauce Labs (unlikely!) they provide access to a multitude of browser/platform combinations via vms so you can run your automated test suite (or do manual testing) and get comprehensive cross browser coverage fairly easily.

Open source projects can get an 'Open Sauce' account for free. You can find out more on the 'Open Sauce' account here. Once you sign-up you will have a username and access key that are used to get access to the service.

Setting up the test runner

We're already using the excellent Karma as our test runner. Karma is essentially a test runner which browsers connect to in order to run a suite of unit tests.

As we're using React we're tending to use unit tests a lot more than before. This is because unit testing in React is really straightforward and it saves us having to worry about end-to-end (E2E) testing up front like we did before with Casper and PhantomJS etc. (We're still planning some integration tests but they can come later). Putting more effort into unit testing feels like the right thing to do. The tests are much more self-contained and therefore less brittle than E2E tests.

To get our existing Karma tests running with sauce labs we used the karma-sauce-loader which handles the setup of a tunnel and all the connections from each Sauce labs browser into the karma test runner.

To DRY things up we have a shared karma config which is just a module exporting the main options object. This is then used by a simple karma config karma.conf.js which just runs tests on Firefox and the sauce labs karma config file karma.conf.sauce.js.

Handling PRs conditionally

The access key and username are used via env vars in our karma config in order to allow our automated tests to run on Sauce labs.

Using TravisCI as our CI you can encrypt the SAUCE_ACCESS_KEY and SAUCE_USERNAME env vars so you don't publically expose them (Which would otherwise anyone to run tests on your account). See this doc for more details: Encryption Keys (docs.travis-ci.com).

But, if you do that, when you propose a PR from your fork of the project it won't be able to run the tests on Sauce Labs as encrypted secrets aren't exposed to pull requests that aren't made from branches on the main repo.

To work around this we can run the tests conditionally by looking at the TRAVIS_SECURE_ENV_VARS env var like so:

// Conditionally run saucelabs tests if we have the
// secure vars or a normal test run if not.
grunt.registerTask('test-ci', function() {  
  if (process.env.TRAVIS_SECURE_ENV_VARS === 'true') {
    grunt.log.writeln('Running full SauceLabs test suite');
    grunt.task.run('test-sauce');
  } else {
    grunt.log.writeln('Running limited test suite');
    grunt.task.run('test');
  }
});

If it's a PR from a fork we just run the standard tests on Firefox on Travis (because TRAVIS_SECURE_ENV_VARS is 'false').

Otherwise if the PR is from a branch on the main repo or a commit on master, the tests will be run by our pre-defined set of browsers on Sauce Labs.

Some projects just commit their public access keys but I feel it's not a good idea to propagate a bad practice like that. This workaround feels like a fair compromise.

If any of the team end up working on something that warrants extra testing before landing we can always push the branch to the main repo first and make a PR from that.

Sharing the browser config

To make it easier to share the browser config (which tells Sauce labs which browser versions and platform combinations we want) we have a node package which contains a simple config object. This module can be shared across all the projects that need to run tests on the same set of browsers.

For now we're testing the last two versions of Firefox, Chrome, Safari and IE but we can tweak this as we move forward.

Conclusion

Using Sauce labs makes running your existing testsuite on a broad set of platform and browser combinations really straight-forward for a minimal investment in setup time.

In our case karma-sauce-launcher made it really easy.

Recently I've been working on a UI for credit-card payments. As part of this we needed some credit-card icons to show the card type as the user types in the number to the input field.

In looking for existing iconography I either couldn't find exactly what I was looking for, or the license terms were unknown. In the end I built up a small set of icons using a mix of Sketch and InkScape and provided them as svg on github.

You can see a preview of all the current icons here.

The repository is at https://github.com/muffinresearch/payment-icons (MPL 2.0).

Contributions

If you want to contribute updates or improvements ti these icons, please feel free to submit a pull request with any changes.

If you have new icons you want to be included then please follow the examples given and add your icon for each of the existing styles and make a pull request on the github repo.

Using docker and docker-compose to run a development environment can be a good way to have all your services connected and running together.

I'm going to assume you've got a basic understanding of docker and how it works if not there's a good overview of what docker is here.

We're currently using docker-compose alongside docker to tie together several services for the Mozilla payments development environment.

Having used docker-compose (née fig) on the marketplace development environment. We learnt a lot about what did and didn't work.

Two of the biggest issues we hit were:

• Dependencies and dep updates
• Front-end tools

Handling deps offline

Note: I've written about this problem before in detail: Docker and dependencies

If you're running a development environment it's likely you'll be running pip or npm in the Dockerfile. The problem here is that installing all those deps is one layer of the cache. As soon as you update the requirements.txt or package.json that step needs to run again - if you're running the build yourself that can be painful.

I think there's still room for a better solution, but the best way to avoid deps becoming a problem, is to use the hub to build your docker images from a branch in github.

This way you can just pull new images to update your entire environment and they have the latest deps. If you need a new dep, you can just manually install it on the running container (see docker-exec) and when you commit the deps file update the newly built image will have your new dep in it.

Front end toolchain

The next problem that was a big one for us was performance of front-end tooling in a docker container. If you're running anything other than Linux then you'll likely be running boot2docker in virtualbox. The problem with Virtualbox is that you're probably sharing the code into the vm with vboxfs. Unfortunately this doesn't provide the low-level file-change notifications into the vm from the host. So things that we typically use for front-end to watch for file-changes tend to work very slowly and cause virtualbox to eat CPU. From my experience NFS is better from a perf point of view, but still not great.

The trick to solving this one is again to leverage building images on the docker hub. Most of your front-end code is going to just be creating static files. So from your dev-env's perspective you just need to provide the files and serve them from your favorite web server e.g. nginx. The other neat thing is, if you do this right, you're not going to need to remember to commit built files to your tree. \o/

The way we do this is with a data volume container (or data-only container) that just contains the files. For developing locally we do a switcheroo and point that volume to our local source code in the docker-compose.yml.

All the file-watching tools (grunt/gulp and friends) and npm deps are installed and run on the host not in a container.

To have the docker env still work out of the box, we set-up hooks on travis to publish the built files (when the tests have successfully passed) to a different branch and the docker hub builds the image from that.

In summary

When things don't work you have to go back to the drawing board. Sometimes if you feel like you're fighting a tool then you're probably doing it wrong ™.

In our case leveraging the docker hub has made a real difference, Thanks to @AndyMckay for pushing us in that direction.

All in all I think we're starting to think more like we would for using docker in production. Which is not a bad thing, and it would mean that going from having a working development environment to something that could be running in prod would be less of a leap.

I've glossed over a lot of details - so if you'd like to know more take a look at the payments repos or let me know in the comments.

var inst = new MyConstructor();

The context object when new isn't used

The code contained a safeguard to bail if the new keyword wasn't used. Which is a good idea because if the the new keyword is omitted, the context will be window.

This outputs:

"[object Window]"
"[object Object]"

Making a flexible constructor

To move away from the need to require new we ended up changing the constructor to allow instance creation both with and without new. This meant that we could create our objects more directly. In our case the code was passing the objects as args so getting rid of new simplified things quite a bit.

This outputs:

"foo1"
"foo2"

So now:

var inst = MyConstructor();

and

var inst = new MyConstructor();

Are equivalent. At least if you forget the new keyword you're not going to get burned by having the context object be window.

Using arguments to DRY up the signature

One issue that my colleague Kumar raised with this approach is it means you need to update the signature in two places when it changes. Not a huge deal but it would be nice to get away from that.

Immediately I thought of arguments. But you can't use apply and new together to create an instance.

So here's an alternative that deals with that:

Here we create an object with the MyConstructor prototype and then call the constructor using that object as the context. As arguments is used you avoid having 2 places to update the signature.

Here's the output:

["foo1", "bar1"]
["foo2", "bar2"]
["whatever", "bar1"]
["foo2", "bar2"]
true
true

Here's that example on JSBin so you can see it in action.

Over to you

Found a better, more elegant way to do this? Leave a comment with a code example.

We've been using CasperJS for E2E testing on Marketplace projects for quite some time. Run alongside unittests, E2E testing allows us to provide coverage for specific UI details and to make sure that users flows operate as expected. It also can be used for writing regression tests for complex interactions.

By default CasperJS uses PhantomJS (headless webkit) as the browser engine that loads the site under test.

Given the audience for the Firefox Marketplace is (by a huge majority) mainly Gecko browsers, we've wanted to add SlimerJS as an engine to our CasperJS test runs for a while.

In theory it's as simple as defining an environment var to say which Firefox you want to use, and then setting a flag to tell casper to use slimer as the engine. However, in practice retrofitting Slimer into our existing Casper/Phantom setup was much more involved and I'm now the proud owner of several Yak hair jumpers as a result of the process to get it working.

In this article I'll list out some of the gotchas we faced along with how we solved them in-case it helps anyone else trying to do the same thing.

Require Paths and globals

There's a handy list of the things you need to know about modules in the slimer docs.

Here's the list they include:

• global variables declared in the main script are not accessible in modules loaded with require
• Modules are completely impervious. They are executed in a truly javascript sandbox
• Modules must be files, not folders. node_modules folders are not searched specially (SlimerJS provides require.paths).

We had used this pattern throughout our casperJS tests at the top of each test file:

var helpers = require('../helpers');  

The problem that you instantly face using Slimer is that requires need an absolute path. I tried a number of ways to work around it without success. In the end the best solution was to create a helpers-shim to iron out the differences. This is a file that gets injected into the test file. To do this we used the includes function via grunt-casper.

An additional problem was that modules are run in a sandbox and have their own context. In order to provider the helpers module with the casper object it was necessary to add it to require.globals.

Our shim looks like this:

var v;  
if (require.globals) {  
  // SlimerJS
  require.globals.casper = casper;
  casper.echo('Running under SlimerJS', 'WARN');
  v = slimer.version;
  casper.isSlimer = true;
} else {
  // PhantomJS
  casper.echo('Running under PhantomJS', 'WARN');
  v = phantom.version;
}
casper.echo('Version: ' + v.major + '.' + v.minor + '.' + v.patch);  
/* exported helpers */
var helpers = require(require('fs').absolute('tests/helpers'));  

One happy benefit of this is that every test now has helpers on it by default. Which saves a bunch of repetition \o/.

Similarly, in our helpers file we had requires for config. We needed to workaround those with more branching. Here's what we needed at the top of helpers.js. You'll see we deal with some config merging manually to get the config we need, thus avoiding the need to workaround requires in the config file too!

if (require.globals) {  
  // SlimerJS setup required to workaround require path issues.
  var fs = require('fs');
  require.paths.push(fs.absolute(fs.workingDirectory + '/config/'));
  var _ = require(fs.absolute('node_modules/underscore/underscore'));
  var config = require('default');
  var testConf = require('test');
  _.extend(config, testConf || {});
} else {
  // PhantomJS setup.
  var config = require('../config');
  var _ = require('underscore');
}

Event order differences

Previously for phantom tests we used the load.finished event to carry out modifications to the page before the JS loaded.

In Slimer this event fires later than it does in phantomjs. As a result the JS (in the document) was already executing before the setup occured which meant the test setup didn't happen before the code that needed those changes ran.

To fix this I tried alot of the other events to find something that would do the job for both Slimer and Phantom. In the end I used a hack which was to do it when the resource for the main JS file was seen as received.

Using that hook I fired used a custom event in Casper like so:

casper.on('resource.received', function(requestData) {  
 if (requestData.url.indexOf('main.min.js') > -1) {
   casper.emit('mainjs.received');
 }
});

Hopefully in time the events will have more parity between the two engines so hacks like these aren't necessary.

Passing objects between the client and the test

We had some tests that setup spys via SinonJS which passed the spy objects from the client context via casper.evaluate (browser) into the test context (node). These spys were then introspected to see if they'd been called.

Under phantom this worked fine. In Slimer it seemed that the movement of objects between contexts made the tests break. In the end these tests were refactored so all the evaluation of the spys occurs in the client context. This resolved that problem.

Adding an env check to the Gruntfile

As we use grunt it's handy to add a check to make sure the SLIMERJSLAUNCHER env var for slimer is set.

if (!process.env.SLIMERJSLAUNCHER) {  
    grunt.warn('You need to set the env var SLIMERJSLAUNCHER to point ' +
               'at the version of Firefox you want to use\n See http://' +
               'docs.slimerjs.org/current/installation.html#configuring-s' +
               'limerjs for more details\n\n');
}

Sending an enter key

Using the following under slimer didn't work for us:

this.sendKeys('.pinbox', casper.page.event.key.Enter);  

So instead we use a helper function to do a synthetic event with jQuery like so:

function sendEnterKey(selector) {  
  casper.evaluate(function(selector) {
    /*global $ */
    var e = $.Event('keypress');
    e.which = 13;
    e.keyCode = 13;
    $(selector).trigger(e);
  }, selector || 'body');
}

Update: Thursday 19th Feb 2015

I received this info:

@muffinresearch about your blog post: for key event, use key.Return, not key.Enter, which is a deprecated key code in DOM

— SlimerJS (@slimerjs) February 19, 2015

Using casper.back() caused tests to hang.

For unknown reasons using casper.back() caused tests to hang. To fix this I had to do a hacky casper.open() on the url back would have gone to. There's an open issue about this here: casper.back() not working script appears to hang.

Exit code is always 0

An eagle-eyed colleagure spotted that the exit code was always 0 via casper when setting SlimerJS up for the marketplace frontend tests. Fortunately the patch for this is coming soon in 0.10. The workaround for this today is to use the 0.10pre version. Hattip @markstrmr

The travis.yml

Getting this running under travis was quite simple. Here's what our file looks like (Note: Using env vars for versions makes doing updates much easier in the future):

Yes that is Firefox 18! Gecko18 is the oldest version we still support as FFOS 1.x was based on Gecko 18. As such it's a good Canary for making sure we don't use features that are too new for that Gecko version.

In time I'd like to look at using the env setting in travis and expanding our testing to cover multiple gecko versions for even greater coverage.

Along the way I've learnt a few things that made a difference to how I use irc and allowed me to get the most out of it.

Be on irc if it's a part of the culture

If your role is non-technical it can be easy to not take part in irc e.g. I've heard in the past people dismiss as "just for geeks" :). However, if irc is a big part of how the company communicates, it's important for everyone to get involved. Especially if remoties form any part of the work force they'll appreciate you being there.

irc is easy to learn and people will generally be very happy to show new people the ropes.

Here's a great wiki page with a ton of information on how to get started with irc.

Stay connected whilst offline

If you're working with people that span timezones chances are there are going to be times when you need to talk to them and they're not awake and vice versa.

When you shutdown or put your computer to sleep you're not available on irc the moment your typical irc client disconnects from the server.

This means that if someone wants to tell you something they've got to wait till you're online or drop you an email.

An irc bouncer or server-based client allows you to stay connected permanently regardless of when you are offline.

Typically this means that when you reconnect the next day the backlog from when you were offline is replayed, thus any questions or messages people are sending you you'll see.

If you're not a developer and you don't want to run your own bouncer then you could always look at a hosted solution like irccloud which is a web client it costs a minimal amount per-month and you might even find your company can cover the cost.

If you're a developer then take a look at something like this bip howto or checkout this list of bouncers.

Register your nickname

It's always a good idea to register your nick to stop someone coming along and taking your name. Especially if you want to be reliably available using a specific nickname. See this howto for how to set it up.

Highlight people you're talking to

If you know who you expect to get an answer to a question from it's a good idea to highlight them when talking to them.

So instead of:

[02:00:39] <Yondu> I can't seem to reproduce bug 123456

try this:

[02:00:39] <Yondu> muffinresearch: I can't seem to reproduce bug 123456

In this example because I've been mentioned by name I'll get notified of the question. This will also work if I'm replaying a bouncer backlog.

Most irc clients have auto completion for irc nicknames. This will also tell you if that person is connected. If they're not on the channel they won't be able to ever get the message (apart from looking at channel logs (unlikely)).

Configure additional highlights

Most clients have an option to allow you to be highlighted for more than just your nick. I tend to add things like my username for bugzilla so if someone links a bug and a bot displays the bug and I'm assigned I'll get highlighted.

As always choose carefully and don't have highlight set up for any words that will come up often or it will drive you crazy!

Ask questions at the start

Sometimes people will do this kind of thing on irc:

[02:00:39] <Yondu> muffinresearch: ping
...
[09:01:21] <muffinresearch> Yondu: pong

And now I have to wait for the question.

Here's a much better way

[02:00:39] <Yondu> muffinresearch: ping, how do I do x,y and z.
...
[09:01:21] <muffinresearch> Yondu: pong - ah you just need to do this, that and the other.

Now when Yondu's next online the answer is already waiting. You might find asking questions to people in this way is not the best fit for irc which brings us onto the next point.

Use private messages only when needed

Private messages are handy if you need to speak to someone in private (outside of a channel). However if you're asking someone a question PM'ing them is not the best solution unless it's something that you don't want in a public channel.

If you have a question it's best to ask out in the open. This way everyone gets to see the question and hopefully an answer too.

A trick you can use if on the receiving end of someone using PMs for something that should be public is reply to it from a relevant channel to pull the conversation into a public place. (Hattip @beuno for this one)

Know when irc isn't the best medium

If you want to ask a question and don't need a reply immediately then using email can be a better solution - I'd generally take irc to be a more reliable form of communication than relying on using irc for an async conversation.

Another aspect to this is being mindful of interrupting a developer. For an example see this brilliant comic by Jason Heeris. Personally I'm OK with being interupted as long as it's not something that if someone had spent two seconds to search for they would have found it for themselves. A culture where people are afraid to ask questions is not what you want so tread lightly on this point.

Mark yourself as away if you are

If you're not at your desk you can use away to tell people you're not around like so:

/away lunch

Good clients will usually indicate away status.

Sometime people change their nicks to show they're away. This is OK, but it does create noise in the channel due to the notification that the user has changed their nick. So most people prefer the use of away.

Remember, if you set an away status you'll want to clear it when you're back.

/away 

Is enough to do that. This is especially important when using a bouncer. You want other people to be able to highlight you when you're away but you also want to set expectations so they know you're not around.

I personally do something else at the start and end of each day. I always say good morning when I start. And I always end the day with

/me eods

This does 2 things. It tells people I'm gone for the day and a side benefit is I can always search logs and work out what hours I'm keeping by looking for the "eods" alongside my nick.

Setup logging

If you're using irc make sure you enable logs. They can be really useful if you need to recall a previous conversation.

Don't judge people by their irc persona.

If you've only talked to someone on irc, it's important to note that how they come across on irc may not be representative of who they are. I've worked with people before who seemed very terse on irc. I sometimes wondered if it was just me. Then when meeting them face to face it all made sense it was part of their character and we actually got on brilliantly.

This is why for remote people getting together face to face from time to time is worth it's weight in gold.

Use non text-based comms

irc is a great tool. It can also be a difficult medium to use if you're trying to convey lots of information.

All the companies I've worked at remotely tended to have a solution to this problem by providing either audio or video chat solutions. Personally I like using video chats for some face to face communication, this way you can probably convey far more information.

Use pastebins

If you need to show someone a traceback of a snippet of code. Never copy it directly into irc. A large chunk of text will tie up the channel for a while as it gets added line by line.

Instead use a pastebin service like dpaste.de and drop the link in the channel.

It's always a good idea to check that you're not posting sensitive content before you save it.

Stay safe

As above - don't paste sensitive things into irc. Be aware if the channel you're conversing in is private or public. Generally if you always assume you're speaking in a public channel you won't go far wrong.

If you do accidentally paste something like a password into a channel be sure to change it!

Always use secure connections to irc services where available.

Turn off automatic image display

My irc client supports automatically showing images. I thought this was brilliant until someone started dropping really unsavoury images into a public channel. Even if this hasn't yet happened to you I'd recommend turning it off before you get a nasty surprise.

Over to you

What other tips do you have?

Using current versions of Firefox with Charles Proxy you might run into a problem where Firefox see Charles' own CA certificate as expired when you try to install it.

After talking to some colleagues at Mozilla the cause is due to the way that Firefox validates certs. The more recent versions of Firefox only allow certs with start dates after the unix epoch (1st January 1970). As the Charles CA cert has a start year of 1899 it's seen as expired when you try to import it.

The easiest way to workaround it is to generate your own CA certificate and use that instead.

I've already filed a bug report with the author of Charles and will update this post with any updates in due course.

Having dealt with enabling Proxying on FFOS devices I recently needed to proxy code for Firefox for Android too. I'd recommend reading that post for all the background info.

Fortunately the process for adding the CA cert your proxy uses to the certs db on Firefox Android is identical to how you do it for FFOS.

Note: As always take care and carry out these steps at your own risk

Why?

Proxying SSL connections is really useful for debugging and development. I often use proxying to rewrite JS with my own built JS files so I can test a bugfix with real data. This can be especially helpful on devices where pointing at local development environments can take time.

SSL proxying is needed otherwise you can't de-crypt the request to do something useful with it.

Once you have the necessary rewrite rules setup in Charles and as long as you can use the proxy on your user-agent (Phone/Desktop browser etc), switching out dev/stage/prod JS with your own is trivial.

For more info on a Charles Proxy specific setup to rewrite files see: Using Charles proxy to debug live code

Requirements

The pre-requisite steps are as follows:

• You've got adb installed
• You know how to set-up the proxy settings for your android phone.
• You have a rooted phone android phone with Firefox Android installed.
• You've generated a cert for your proxy. (If not you can read how to do that here)
• You've got certutil installed from nss. Be careful if you use homebrew to install it. This can break sync.

Finding the profile

First I use adb shell to get a shell and then type su to get root.

adb shell
shell@m0:/ $ su
root@m0:/ #

Now to modify the db you need to find your profile.

For FF Android Nightly this will be:

/data/data/org.mozilla.fennec/files/mozilla

and for FF Android it will be in:

/data/data/org.mozilla.firefox/files/mozilla

And you'll be looking for a file that ends with .default

Moving the cert files

Once you have the paths. You'll want to move the cert9.db and key4.db onto the sdcard so you can more easily pull them onto your machine.

e.g:

cp /data/data/org.mozilla.fennec/files/mozilla/fkhfkjsh.default/cert9.db /sdcard/cert/cert9.db
cp /data/data/org.mozilla.fennec/files/mozilla/fkhfkjsh.default/key4.db /sdcard/cert/key4.db

Now you can grab them from another terminal tab:

mkdir ffa-certs && cd ffa-certs
adb pull /sdcard/certs/cert9.db
adb pull /sdcard/certs/key4.db

Now back them up in-case you want to restore them or something goes awry.

cp cert9.db{,.bck}
cp key4.db{,.bck}

Adding your CA to the cert db

Finally add your cert. I'm using the same CA cert I generated for Charles. So that's the following:

# Hit enter to remove the password
certutil -W -d sql:. 
# Add the cert
certutil -A -n 'Charles Custom Cert' -i /usr/local/CharlesCA/certs/ca_cert.pem -t 'TC,,' -d sql:.
# Check it validates
certutil -V -u 'V' -n 'Charles Custom Cert' -d sql:. 

Pushing certs onto the device

Now lets throw the certs db back on the device:

 adb push cert9.db /sdcard/certs/cert9.db
 adb push key4.db /sdcard/certs/key4.db

And then back on your root adb shell you had earlier:

cp /sdcard/cert/cert9.db /data/data/org.mozilla.fennec/files/mozilla/fkhfkjsh.default/cert9.db 
cp /sdcard/cert/key4.db /data/data/org.mozilla.fennec/files/mozilla/fkhfkjsh.default/key4.db

Assuming you've pointed your network (wifi) connection at the proxy you should be good to go.

As always don't forget to turn off the proxy when you're not using it.

Last week I posed this question on Twitter:

Docker experts. How do you deal with non-os deps e.g. pip/npm/bower and avoiding cache-busting the dep install layer on dep changes? #docker

— Stuart Colville (@muffinresearch) January 30, 2015

Unfortunately Twitter isn't the best medium to provide the necessary context so here's a bit more info on what I meant.

One of the problems we've seen using docker to build out a development environment is that it's required a lot of working around limitations of the environment. Dependencies aren't the only problem but they're proven to be something of a pain.

The problem

Dep changes require the dependency installation to be run from scratch.

The FFOS marketplace as an app comprises lots of services that are loosely coupled. This means we can run separate containers and use links to tie things together. We also have a lot of modular code and libs that are not OS packages.

We specify our non-OS packaged deps using requirements.txt files for Python and package.json for nodejs modules. We then add those files via the Dockerfile using an ADD instruction e.g:

ADD requirements /pip/requirements

Next the deps are installed:

RUN pip install -r /pip/requirements/dev.txt

Now due to how the AUFS works the ADD is cached so if the requirements.txt file stays the same re-building that part is a no-op. Great!

But, when we do bump a package version or add a new dependency the cache is invalidated and the entire installation of deps is re-run from scratch.

As we have quite a large collection of deps this can a) take a long time and b) can break if the package repository is busy/down/having a bad day.

The goal

We'd like a new dep or a dep change to not require all the network access and time it takes to re-install all the dependencies from scratch.

After-all this is a development environment and having to build the world after some dep changes/version bumps gets in the way.

Alternatives

I've been thinking about some alternatives but nothing yet has seemed particularly attractive.

Proxying npm/pypi etc

One possibility would be to use a proxy to cache the package downloads to minimise the network overhead. This would help but there's a few nits. It's another moving part. You still need to pull the packages and install them. So this would alleviate the network part of the problem but that's really only half the problem.

Caching the deps on the filesystem

I looked to see if caching the deps would make a difference. Unfortunately persistent storage via volumes isn't any use at build time. As volumes are only added at run-time you can't utilise the cache when the deps installation is re-run.

Externalizing the deps

One more recent thoughts I had was whether we could move the non-OS packaged deps out of the build-step and into run-time. This way we could use a data-only container for the deps and if the deps change you'll only pull in what's changed.

The downside is that this feels like it's starting to defeat the point of a container-based approach.

Offline builds

Andy Mckay has been trying out using automated hub builds as the trees are updated. This works quite well but there's still a few nits. Bumps in deps require downloading new layers to images. Larger files instead of lots of packages is a step in the right direction but it still feels like it gets in the way. We also need a good way to keep builds aligned with the src revision otherwise you might have the wrong deps for the branches src revision.

What's everyone else doing?

I'm sure we're not the only people trying to do this. How do you overcome these issues in your use of Docker? Or do you just accept the hit in terms of builds taking a long time? Have you found some clever way to manage deps so devs can get up to date quickly without getting buried in a ton of docker-shaped yaks?

Could Docker solve this?

Maybe there could be a way to allow certain paths to not be cache-busted? E.g. could we have a way to allow the pip cache to remain intact so that it can be used when the deps installation is run when the requirements.txt changes?

After my post on Live debugging with Charles, Nick Desaulniers asked if I knew if it was possible to proxy device traffic via Charles. I didn't, but I did know that if it was possible it was sure to come in handy.

10 freshly shaved yaks later…

This post details getting FFOS (I'm using a flame with FFOS 2.1) to use Charles Proxy for all requests so we can debug requests on the phone. I'm using OSX for the rest of the commands etc, but this should work similarly on Linux just be prepared to changes a few paths here and there as required. You'll need the usual adb etc before you start.

As always, respect the yaks and run these steps at your own risk.

Setting up FFOS to use a proxy

First you need to know your local network ip address. You'll typically be running all this on wifi. The expectation is that your phone and computer are running on the same wifi network.

This command should give you the primary interface (but will depend on that being the only interface try en1 if that doesn't provide what you expect.)

ipconfig getifaddr en0 

If in doubt you can always check network prefs via your system settings for your OS.

Once you know that you can set up the prefs file for your firefox OS device.

The proxy prefs are as follows (It goes without saying you'll need to replace 192.168.0.2 with whatever your local ip is. Change the ports if your charles port is not 8888.

pref("network.proxy.backup.ftp", "192.168.0.2");
pref("network.proxy.backup.ftp_port", 8888);
pref("network.proxy.backup.socks", "192.168.0.2");
pref("network.proxy.backup.socks_port", 8888);
pref("network.proxy.backup.ssl", "192.168.0.2");
pref("network.proxy.backup.ssl_port", 8888);
pref("network.proxy.ftp", "192.168.0.2");
pref("network.proxy.ftp_port", 8888);
pref("network.proxy.http", "192.168.0.2");
pref("network.proxy.http_port", 8888);
pref("network.proxy.no_proxies_on", "");
pref("network.proxy.share_proxy_settings", true);
pref("network.proxy.socks", "192.168.0.2");
pref("network.proxy.socks_port", 8888);
pref("network.proxy.ssl", "192.168.0.2");
pref("network.proxy.ssl_port", 8888);
pref("network.proxy.type", 1);

If you're already using your own user prefs then append these lines to the file. If not create a file called custom-prefs.js and then push it onto your device:

adb push custom-prefs.js /data/local/user.js

Now restart:

adb reboot

This should get the device talking to charles. However at this point, anything that uses SSL you won't be able to see the requests for.

Before you can do that, first you're going to need to generate your own cert for charles. It seems that recent Firefox versions see charles' CA cert as expired which can be a big red-herring to getting this working.

Update 29th November 2014

After talking to some colleagues the cause of the expired certificate is due to the way that Firefox validates certs. The more recent versions of Firefox only allow certs with start dates after the unix epoch (1st Jan 1970). As the Charles CA cert has a start year of 1899 it's seen as expired.

Generate your own CA cert for charles

It's a good idea to make your own CA cert for Charles - so this step isn't so bad in the long run. As long as you keep your cert and keys safe no-one can abuse it.

There's an excellent post with both the rationale and the low-down on how to generate your own cert here: Securing Charles Proxy with a Personal CA

Note: This post assumes openssl.conf is under /opt/local/etc/openssl which it will be if you've installed openssl with homebrew e.g brew install openssl.

I'll reproduce the broad strokes here to keep all the instructions relevant to this post in one place:

# Setup a directory to hold this info.
mkdir -p /usr/local/CharlesCA
cd /usr/local/CharlesCA
mkdir certs private newcerts
echo 01 > serial
touch index.txt

# Next create the CA cert
openssl req -new -x509 -days 3650 -extensions v3_ca \
-keyout private/ca_key.pem -out certs/ca_cert.pem \
-config /opt/local/etc/openssl/openssl.cnf

At this point you'll need to set a passphrase and enter some basic info. Charles will ask for this passphrase when it needs to.

Next we are going to run a command to generate the PKCS12 format that charles expects.

openssl pkcs12 -export -out ca_cert.pfx -inkey \
private/ca_key.pem -in certs/ca_cert.pem

So the files we get from this process are:

• ca_cert.pfx - this is what we need to point charles at.
• ca_cert.pem - this is the ca certificate to add to clients
• ca_key.pem - this is the key (keep this safe).

Now we have this we need to point charles at the ca_cert.pfx ca cert we just generated. When we do this Charles will ask for the passphrase.

Next it's a good idea to test out running Firefox with charles and makes sure SSL proxying is working. To make that happen you'll need to enable SSL proxying for a host of your choice. See above for an example.

To make firefox work with Charles we'll need to add the ca_cert.pem into firefox. I just use the Firefox -> Preferences -> Advances -> Certificates and the click View certificates and then import. Find the ca_cert.pem generated before and import that and check This certificate can identify websites.

Next try and view a site using the ssl host you set up. You should be able to look at responses.

Adding the custom CA to FFOS

Assuming that went well we're ready to add the custom cert to FFOS. There's an interface under the device's wifi settings that allows the import of certs (if they're pushed to /sdcard/downloads first) but this didn't seem to work for me. It might just be that you can't set trust correctly (there's no interface for that) via this route, which stops it working.

To add our own cert we'll need to use the nss tools. In particular certutil is the command we'll use to add a cert to your device.

You can install the nss-tools on OSX with brew install nss however be aware this might cause havoc with Firefox sync see my post on this issue. If that's the case you can build the nss tools yourself (I'll leave that as a yak to shave for the reader).

First we'll run some commands to get the existing cert9.db and key4.db off the device.

Here's a little script that will pull the certs into the same dir (or you can just run the commands one at a time if you prefer).

#!/usr/bin/env bash

adb wait-for-device
PROF_DIR=$(adb shell echo /data/b2g/mozilla/*.default | tr -d '\r')

echo $PROF_DIR

echo Copying ${PROF_DIR}/cert9.db
adb pull ${PROF_DIR}/cert9.db .

echo Copying ${PROF_DIR}/key4.db
adb pull ${PROF_DIR}/key4.db .

To back them up the first time you can run:

cp cert9.db{,.bck}
cp key4.db{,.bck}

This will give you a backup just in-case.

Next we need to clear the existing password, so we don't get errors trying to add our CA cert.

certutil -W -d sql:.

When the prompts appear just hit enter twice.

Now let's add our ca_cert.pem file.

certutil -A -n 'Charles Custom Cert' -i /usr/local/CharlesCA/certs/ca_cert.pem -t 'TC,,' -d sql:.

This assumes the cert file is in the location as per the custom CA generation steps above.

Now the next step is to validate that the cert added is good for our purposes:

certutil -V -u 'V' -n 'Charles Custom Cert' -d sql:. 

You're looking for output like so:

certutil: certificate is valid

If that's good then you're all set.

The final step is to push back the files:

Assuming you're in the same terminal session you can use the following commands.

adb push cert9.db ${PROF_DIR}
adb push key4.db ${PROF_DIR}
adb reboot

If not run this again first and then re-run the push commands above:

adb wait-for-device
PROF_DIR=$(adb shell echo /data/b2g/mozilla/*.default | tr -d '\r')

Next try hitting the same URL you verified that SSL proxying worked for with Firefox and you should see that FFOS is now both using the proxy and SSL proxying is taking place on the host you specified.

The traffic should show up with the files under the host like so:

Turning off the proxy

When you're done with the proxy remember to remove the prefs. Hopefully at some point they'll be a way to turn this on or off from the device settings.

Notes and references

mozTCPSocket requests don't seem to work via the proxy if they require SSL. You'll see bad_certificate in charles. I don't know of a workaround for this presently (so I'd suggest avoiding proxying hosts that use that).

This post wouldn't have been possible without the following - my thanks goes to the authors for sharing their wisdom:

• Adding CA to FirefoxOS
• Securing Charles Proxy with a Personal CA
• Intercepting Firefox OS traffic using a proxy

Over to you

It's likely the steps to do this might get shorter in due course or short-cuts will be found. If you find a better way to do any of this or have extra OS specific notes or additional useful references please let me know in the comments.

Also if you manage to shave all the yaks and get this working based on these instructions please let me know too.

sync encountered an error while syncing: Unknown error.

Sync has logs in about:sync-log

I was seeing this:

Crypto check failed: [Exception... "PK11_GenrateRandom failed"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/services-crypto/WeaveCrypto.js :: WeaveCrypto.prototype.generateRandomBytes :: line 550"  data: no]

Looking for a fix I found comment 33 in bug 1034979

|brew uninstall nss| and restart Fx seems to fix the problem for me.

And sure enough removing the brew installed nss package did the trick.