I wanted to write a technical post about a project I’ve undertaken in my job at CBC Music this last year: the refactor of their mobile site codebase. Since starting at the CBC last May (2013) my chief responsibility has been to lead this project, so it was great to see it launch last month.
Maybe I’ll start by outlining the problem I was tasked to solve.
The problem
The mobile code – which was already out in the wild when I joined – was perfectly decent, but lacked scalability both in terms of the overall code architecture and in what was doled up to the client. The JS was arranged in a simple object namespace, which was generally quite readable but had already led to a great deal of unnecessary and unfortunate coupling.
Worse, all JS was sent to the client on initial page load. Each and every sprint (we were on a weekly sprint cycle at the time) led to 10KB, 20KB, 30KB+ increases of the resource size and they knew that this problem was only going to be compounded over time.
What we needed was to update the codebase for on-demand loading, and rethink the overall JS architecture for a simpler, more formal structure to improve the overall code quality, and to make it simpler for junior developers to work with it.
Existing tools and technologies
I was lucky in that the existing libraries / technologies already in place were by and large well chosen. The key libraries were jQuery, Backbone, Handlebars, Sass, C# for JS and CSS bundling and minification. The site was almost entirely client-side which got its data via a RESTful API. However, there were a few C#-isms: the base markup for the mobile site was generated server-side, the initial URL routing was handled via a C# controller, the Handlebars templates were compiled on the server, and the IIS Web.config file was being used for environment-specific settings when rolling out the site to the various environments. Sass was being precompiled to CSS using a Visual Studio plugin. They were already using HTML5 History for URL routing and other than the lack of the on-demand loading, it had everything you’d expect of a one-page web app.
Tools-wise, CBC Music is very much a C# shop – and thus very Visual Studio-centric. VS was very much integrated with the whole dev process and difficult to remove.
Performance
One other key consideration to the whole project was performance. The site was slooooow! The C# code that handled the bundling and minification was very basic: it simply grouped together all resources into a single file (well, URL) and attached the hash of the result in the query string for browser cache busting. That meant that every week users would have to re-download everything. Yikes. One of my co-workers had (or has, I should say) a healthy obsession with speed, which helped motivate me to keep performance in mind at every turn. :)
The Grand Plan
After spending long hours with the codebase to find out how it all fit together and understand what problems they’d already solved, I decided on the setting the following targets for the refactor:
1. Completely remove all server-side dependencies.
2. Write it so that the website could be run on IIS or Apache, and be developed on any IDE on Mac or Windows.
3. Move the code to AMD, using requireJS for improving the overall code organization and on-demand loading.
4. Move to Grunt to handle all the intricacies of the build process, including bundling, minification, file renaming to get around caching issues.
A few notes about each.
- Removing the server-side dependencies just seemed like good sense: mixing the two was complex and unnecessary. They’d already gone 95% of the way to separate the client and server-side logic via the REST API, so the remaining ties were… untidy. :)
- Getting the site to run on both Apache and IIS and removing the dependency on Visual Studio and Windows was really just a pet goal of my own. I’m a Mac guy and I *really* don’t like IIS and VS for web development. I wanted to pave the way for any of us to use other IDEs and environments down the road.
- I’d used requireJS for several other projects and knew it well.
- Grunt was all new for me, but based on a few tests and what I’d read about, it seemed like it was going to be a good fit (spoiler: it was. Grunt rocks).
At the back of my mind the whole time was the idea of ultimately open-sourcing the codebase (This hasn’t happened yet, but I’ll continue to annoy my bosses about it). Like anything, I wanted it to be as simple as possible to understand, install and use. The fewer dependencies of non-client-side technologies and libraries just made sense.
JS Architecture
Alright. Now down to the nitty gritty. I arranged the code into three broad groups:
- The “Core”. This contained all JS code necessary for loading any page on the site. It contained things like the initial app-start JS file (first load point for any page), the Backbone router instance, global constants, the requireJS config file, a “Page Manager” which acts as the brain of the entire script, a mediator script that allowed indirect communication between arbitrary components and a few smaller things. It also initialized certain components (see below) that we knew were needed all over the place: the nav menus, the header, the music player.
- “Pages”. Pages were JS files expressly designed to load the content of a single page; they mapped to one or more URLs and were loaded by the Page Manager based on the URL. The public API they returned to the Page Manager was a formally structured object, containing various details about the page (title, share information, nav section, load and unload function, etc).
- “Components”. Components could be handlebar templates or JS files. The idea of a component was a sharable, re-usable block that could be used anywhere.
Ideally I wanted for each page to be 100% self-contained and specify their own URLs themselves, but in practice I had to drop it. In order for the core to know what JS and Handlebars template files on disk to load, there needed to a be a central map of all URLs (regexps in our case), which was included in the Core. I guess this is really kind of a minor detail, but it’s always bugged me… [In retrospect, this would have been actually possible to do via a separate grunt task to extract the URL regexps from each component during the build, and make them all available to the Core code. But I kinda figure it's perhaps more fuss than it's worth, and I'm just being overly anal.]
I chose *not* to wrap all libs and frameworks (jQuery, moment, Backbone, hammer and others) into AMD format. We talked about it, but we decided against it. I have yet to regret it – on this project or previous ones where I did the same.
Grunt
Grunt was the hinge that made the refactor possible.
First, in order to simulate how the site would be built to the different environments, I used the grunt-template plugin to generate certain key files as needed. I used it instead of the grunt-usemin plugin because it was more granular and allowed me to do all sort of things. Running grunt for the different environments regenerated certain key files: the main index.html page (the only HTML file on the site), the require.config.js file containing the various AMD resource mappings (locally would just point to the files, in stage + prod would point to the minimized, bundled, MD5-renamed versions), the core constants file, and a Sass variable file.
Secondly, grunt replaced most of the C# functionality, and added a lot more features than we previous had:
- Specifying environment specific values to build the site for local, dev, stage, release, canary and prod. By just running “grunt local”, “grunt stage”, etc. we were able to generate the exact deployment configurations that would appear in any environment on our own dev boxes.
- Sass file precompilation.
- Image file renaming and magnification (tied to Sass to update the references to the images via sass vars)
- Handlebar template precompilation.
- JS, CSS minification and bundling.
- JSHint
- requireJS module bundling, automated renaming of the files to include their file hash (md5).
This last point was key. In the production environment, all (well, most!) JS and CSS files containing their file hashes in their filenames had two benefits: it caused browsers to only download new files after deployments that contained actual changes (i.e. browsers only ever downloaded new resources when it needed to), and secondly, that we could set extremely high expiry cache headers on those files, again preventing unnecessary downloads.
Summing-up
There are many, many more details of the project that this post doesn’t cover – this was just from a very high level. Since releasing the site we’ve continued making improvements: image lazy loading, caching of resources and API calls on both the server and Varnish (the load balancer) level, improved image spriting and more. And no doubt there will be more improvements to come.
It was an extremely fun project, and nice to tackle the problem of the refactor from many angles – build process, tools, technologies and code architecture. The nice thing about this job is that it allows me to wear different hats. This gave me exposure to the full range of problems involved with a code refactor of this size.
Lastly, what was nice – and what inspired me write this post – was that the NYTimes recently blogged about their own redesign. They picked a remarkably similar set of technologies and libs. I won’t deny this was pretty encouraging to hear about. Embarking on a grand code refactor is always a little daunting, so learning that others are solving the problems in similar ways sure helps. :)
Recent Comments