Photo CC licensed by oedipusphinx

I’ve been working for canonical for about four and a half months now. One of the things that I’ve been really impressed with is the workflow for development and the process for how new features end up in trunk.

The workflow is as follows:

Feature Development

Our primary development focus is trunk as this is what’s deployed on a weekly basis. So the first step is to pull the local copy of trunk to ensure it’s up to date. Next, the developer makes a branch of trunk for the feature/bugfix.

cd trunk
bzr pull
cd ../
bzr branch trunk branches/my-awesome-feature

The feature is developed with TDD (tests first, followed by code to make the tests pass). Once the feature or bugfix is finished the branch is pushed up to launchpad under the project.

# hack hack hack
# Ensure tests pass
bzr push lp:~<USER>/<PROJECT>/<BRANCHNAME>

A nice feature of bzr that I’d not used prior to working at canonical is the way you can automatically set push targets for branches based on the directory that contains them. So for example we have a config in ~/.bazaar/locations.conf that looks something like this (obviously substituting the variables in angled brackets for your username and project etc):

[/path/to/branches/]
push_location = bzr+ssh://<USERNAME>@bazaar.launchpad.net/~<USERNAME>/<PROJECT>
push_location:policy = appendpath

So if I have a new branch at /path/to/branches/foo-bar-baz

This will automatically set the push location to:

bzr+ssh://<USERNAME>@bazaar.launchpad.net/~<USERNAME>/<PROJECT>/foo-bar-baz

Meaning all I have to do is use bzr push to send my branch into launchpad.

Note: At any time you can check the push location with bzr info

Code Review

The next stage is for the developer to make a “merge proposal” via the launchpad interface. This puts the merge request in a “to be reviewed queue” which provided a merge proposal page with a diff of the changes from trunk (the target branch is customisable). The rule is that every branch has to be approved by a minimum of two devs. Reviewers can make recommendations or assert that something needs to be fixed and the developer can then push subsequent changes to the same branch and those changes are made visible.

The focus of reviews are on coding style, standards e.g. adherence to PEP 8, and reviewers will run the test suite and run pylint to check for any glaring violations. Merging with trunk is also carried out to check that no conflicts occur.

Landing branches

Once the code has been successfully reviewed code is submitted via PQM (Patch Queue Manager) which then runs the test suite and only merges the code with trunk if the tests pass. We’re still using PQM but most projects have already begun the process to move to tarmac which is tightly integrated to launchpad.

I like the rigorous review process. By having code reviews a mandatory part of the process it becomes just part of a routine, everyone is used to it and so are happy to receive and give criticism. Reviews are definitely seen as a route to better code rather than making people feel small which is exactly how it should be.

The process might sound quite lengthy but because of the ease of branching in bzr (like any other DVCS) the process is very quick and easy and branches are continually landed in trunk every day. It’s actively encouraged that branches are kept around 500 lines of diff where possible, to make it easier for code to get reviewed. Branches aren’t worked on for weeks on end, they are developed, reviewed and merged within what’s usually at most a couple of days.

Future changes

One of the things we’re currently looking at is using hudson to automate the running of the test-suite for merge-proposals. This would mean that hudson could run the tests as soon as a merge proposal is made and provide an “approval” or “needs fixing” status based on the results of running the test suite. The net effect would be that we’d then require 3 approvals rather than 2.

Summary

Obviously our workflow revolves around the use of launchpad, but there’s plenty of ways to adopt a similar workflow without requiring the use of launchpad. For example on personal projects I’ve adopted pretty much the same process except branches are pushed to remote server instead of launchpad and rietveld has been used for reviews in place of the merge proposal feature. Rietveld is fairly vcs-agnostic however, if you’re using git as your vcs then something like gerrit might be worth looking at. (gerrit was originally a fork of rietveld but has more recently been completely re-written using java instead of python). Also review-board is another code review tool that works for all popular VCSs – review-board is django-based and is therefore relatively easy to set-up.

In summary:

• Use a DVCS (e.g: git, mecurial, bzr) so branching is easy. I remember at Yahoo when CVS was the VCS everyone had to use, branching was explicitly avoided as much as possible because merging was seen as such a pain. Worst case, if you’re stuck with something like svn be comfortable with merging so branching is not a hurdle.
• Branch as often as necessary.
• Review everything and make it a part of the code routine.
• Merge as early as possible.
• Keep Trunk pristine.

When I first ran the image it looked to book up fine – however when it got to mounting the root file system it hung saying “Waiting for root filesystem”

Fortunately whilst looking at the hard-disk settings (with the vm shutdown) I noticed that there were additional adapters for SATA and SCSI. Vmware by default uses SATA so getting it working was a simple as enabling the SATA adapter and setting each disk to the correct SATA ports 0 and 1 in my case as I have 2 hard disks.

Booting the VM resulted in the filesystem mounting correctly and it all working!!!

I then thought it would be a plan to remove the VMware tools so I could install the VirtualBoxGuestAdditions. This turned out to be a bit of a nightmare as running the vmware-tools-uninstaller.pl also deleted my /etc/fstab (WTF?) so when I rebooted everything was read-only.

To get around this I booted back in single user mode and ran the following to remount the root filesystem as rw.

mount -no remount,rw /

Next I recreated the /etc/fstab from a backup and saved it.

Lastly I returned everything back to readonly with

mount -no remount,ro /

and then rebooted. After that it was a case of mounting the VirtualBoxGuestAdditions.iso mounting it at /media/cdrom and then running the installer from there.

Job’s a good’un. If you’re doing the same steps I’d strongly recommend backing-up or checking /etc/fstab is still there after removing vmware tools in case it get’s binned by the vmware tools script of LOL, so you don’t have to go through the same pain to get it all working.

It was a great experience working at Yahoo, I got the chance to work with plenty of awesome people and learnt a huge amount in my time there. So to all of you (you know who you are), thanks for everything!

Having been part of the highly caffeinated team that worked on the site, it’s great to see recognition for the team’s hard work in implementing so many microformats into the site and in particular, Ben’s pioneering work in bringing 26.5 million hListings into the world.

To make our lives easier I wanted to use Django as I’ve been really enjoying using it for a few personal bits and pieces and I thought it would also make it possible to get much more done in the time. Spending 24 hours working with Django and building a fully working site was not only fun but proved how solid and well thought out the framework is. All of the features we had decided on up front were pretty straightforward to implement and get working straight-away. We even managed to start working on some of our wish-list before the time was up.

Another thing about this hack was that we were using mod_wsgi with Apache. Now I’ve used mod_python before but mod_wsgi really is the way to go if you are doing anything with Python and Apache. Setting it up is very straightforward and the documentation is first rate so I’d not hesitate to recommend it.

After the 24 hours were up James gave a 2 minute presentation on our hack and did a fine job of showing off that it was a fully working site and wowing everyone with his kick-ass graphical design-fu. Some of the other hacks displayed were stunning in terms of being not only great ideas and technically awesome. We thought we didn’t stand a chance.

After the judges deliberated, they came back and announced the winners. To our surprise our hack was awarded “Best hack for Quality” and everyone was really chuffed. This was certainly the icing on the cake for myself and Cyril after having had 24 hours of unrestricted time to play with Django and our favourite programming language! Python is certainly not something that we generally use at Yahoo! and I’d personally love be able to use it more where it makes sense to do so.

Now I’m going to have endless fun annoying people around the office with my remote control blimp Darth Vader Transformer that I took as a prize.

When I have a few spare moments I’ll write a post covering the details of the presentation. To view the deck in text format it is available on slideshare. Alternatively you can download a pdf of the presentation (16mb).

| View | Upload your own

With Foosball tables in the canteen I might be able to brush up my skills to be able to take on the mighty Mike Rundle at SXSW 2007 or at least be able to stay on the table longer than 5 seconds.

Last night a few of my work colleagues and a some of my friends got together for a Ruby at the Kirthon in Tunbridge Wells which was fantastic, great food, good company and a nice send-off. The chaps sorted me out with an Amazon voucher as a leaving present so I’ll be sure to spend it wisely. Thanks everyone!

Tuesday will see me start at Multimap and I look forward to working with everyone there. Being up in London will also be great and I’m going to be heading over to Pub Standards on the Tuesday night after my first day at Multimap to meet up with some other London based web developers.

Working at Rentokil has been a great experience and I feel like I’ve learnt a lot in the year and a half that I’ve been there. I’m also proud to have played my part as a standards advocate and now the seed is sown, I’m sure the webdev team will continue to build upon this in the future.

Commuting to London will be a big change of scene and I’m going to have to dust off my BMX and my walkie talkie with it’s unfeasibly long aerial for the journey to the station. Still the exercise will do me good; all those puddings in the Rentokil canteen have taken their toll.

Well done to all of the people that played a part in putting this site together. (Quick plug for everyone’s personal sites there!)

We have also been featured in Stylegala which is a great result.

After Tim and I got the chance to go to @media2005 we have been able to bring webstandards to the fore of what we do. Before this site was re-done, as a collective the web-team contributed to creating best practices for the development of this site, and I feel this approach has worked well in making sure everyone who worked on this site was on the same page.

Paul’s design supports the content well and though it’s not the most revolutionary design, that is merely because this is a corporate site and pink flamingos and butterflys just don’t get a look in.

Tim worked really hard to get the mark-up as pure as possible before any of the styles were applied and the attention to detail shows.

We are still slave to content generated through CMS and third-party services but hopefully these are wrinkles that can be ironed our over time.