Asynchronous CommonJS Modules for the Browser (and Introducing Transporter)

By on July 16, 2010 12:15 am

Modules are an integral architectural piece of robust application development since they allow individual components to be developed with proper dependency management. Modules can specify dependencies and these can be automatically resolved and loaded to bring various pieces together automatically. In application development this is vastly more scalable and easier than having to track all the different dependencies and manually load modules or insert script tags.

The CommonJS module format is increasingly ubiquitous as the de facto module format for JavaScript. However, if CommonJS modules, by themselves, are directly executed, they require synchronous loading of modules. Synchronous loading is known to be very problematic in the browser since it locks the browser user interface, requires eval-based compilation of scripts which confuses debuggers, and is less efficient than using standard script tags.

The CommonJS group has developed a specification for callback based module loading called the module transport format. The CommonJS module transport format is specifically designed to make it easy to wrap CommonJS modules (which also use synchronous require calls) with a callback wrapper to allow for asynchronous loading on the client. For example, if one had a CommonJS module:

// complex-numbers/plus-two.js:
var sum = require("./math").sum;
exports.plusTwo = function(a){
  return sum(a, 2);

This can be wrapped with the transport format to be loaded asynchronously and used on the client:

// complex-numbers/plus-two.js received by browser:
require.define({"complex-numbers/plus-two": function(require, exports){
// define body in callback
  var sum = require("./complex-number").sum;
  exports.plusTwo = function(a){
  return sum(a, 2);
},["complex-numbers/math"]); // indicate the dependencies

There are a couple of excellent client-side libraries available for loading and handling modules in the CommonJS modules format including James Burke’s RequireJS and Yabble. RequireJS also provides a number of functions for easing construction of hand-crafted static modules with wrappers (rather than using server-side generation). These are well documented on the RequireJS API page, so I won’t go into those here, but RequireJS makes it very easy to hand-craft modules for asynchronous loading. Yabble is designed to support direct loading of CommonJS modules (with or without wrapping), and supports other CommonJS APIs including require.ensure for explicit dynamic asynchronous loading of modules.

Automated Module Wrapping

A client-side library like RequireJS can then easily load modules and dependencies (like complex-numbers/math.js) asynchronously and execute the module body when they are available. Of course wrapping modules like this requires writing a lot of extra boilerplate code, especially if you are originally writing your code in CommonJS format, so naturally it is helpful to automate this. Fortunately, there are actually several projects that can automate this wrapping, allowing you to write plain CommonJS modules, and automatically wrap the modules so that they can be asynchronously loaded in the browser. FlyScript is a CommonJS module wrapper written in PHP, and modulr is a Ruby-based module wrapper. Each of these come with their own client-side module loader, but with proper CommonJS transport format compliance they could be used with RequireJS as well.

Another module wrapper is my Transporter project that is actually written as a CommonJS module for JSGI servers (like Jack, and Node with an adapter). Transporter is particularly interesting since it is actually designed to run on a CommonJS server, and therefore paves the way for easily reusing the same modules on the server and client effortlessly. In fact, Transporter will default to loading modules using the same lookup path as the server-side CommonJS environment, to make it easy to share modules. Transporter has been primarily tested with RequireJS and Yabble.

Transporter includes the (configurable) ability to automatically resolve all dependencies on the server. This means that when a module is requested, all the dependencies are automatically included, rather than having to wait for the browser to individually request each dependency, which is much slower due to the latency and overhead of multiple HTTP request-response round trips. Transporter includes a number of other useful options including the ability to explicitly include and exclude dependencies from URLs (within script src), automatic running of the modules (in Yabble, modules aren’t auto-executed), specifying module source paths or loaders, and support for alternate converters (so that different formats besides CommonJS can be used). These are described in more detail in the Transporter documentation.

Dojo, RequireJS, CommonJS

These new asynchronous loading capabilities are not just for CommonJS modules. James Burke has been hard at work in integrating RequireJS with Dojo. He has built a tool for a converting Dojo modules to RequireJS-compatible transport format, integrated RequireJS with the Dojo bootstrap process, and created a full Dojo build that incorporates RequireJS with a fully converted copy of Dojo (including DojoX and Dijit). With this build you can use Dojo in fully asynchronous mode, getting the performance, usability, and debugging benefits of RequireJS’s loader with Dojo. After Dojo 1.5 is complete, we will be looking at various possibilities for using this type of technology and integrating asynchronous loading capabilities in core Dojo and possibly supporting CommonJS modules in Dojo as well. However, this is still a work in progress, and there are couple different options we are exploring, so I will save more details on this for a future post.