The Dojo Toolkit 1.1 introduces support to run within Adobe’s AIR environment, and I think it may have re-sparked my interest in ActionScript. It was a lot easier than I had anticipated to get started coding, but came to a halt when it occurred to me that my newly created .air installer and badge would be at best a ~5MB download simply because of the default release size. A 5MB “Hello AIR” app is less than unimpressive, so I immediately started thinking of ways to better the situation. I need a custom minified Dojo Toolkit package.

Initial Distribution

Enter “dojo-mini” – an experiment in Dojo Toolkit deployment optimization. When Alex tagged the release, I immediately downloaded it, dropped it in as the toolkit for my upcoming AIR Demo and began browsing the source making a mental list of crufty files I could drop to shave some bytes away. I was amazed with my results:

The entire dojo.* namespace, fully functioning, is ~150KB. The entire Dijit widget system, stock, including all three stock themes (Tundra, the polished Soria, and new theme Nihilo) is ~475KB, which could be trimmed further I suspect. Every piece of DojoX combined is an additional 550KB, bringing the overhead incurred down to a modest 1.1MB. I can work with a meg.

To be fair, my “Hello AIR” sample _only_ required the 75KB (27KB w/ gzip) dojo.js, and I was able to remove the entire dojo tree with great success. The base dojo.js is so feature-rich you really don’t need much else to do really great things, though having the rest of the toolkit available would be ideal. And, like any other 27KB solution to web development, you will immediately find a need for something else, and add another piece/plugin/module/whathaveyou. It adds up, and quickly.

Two of the Dojo Toolkit’s strong points are a) It really is everything but the kitchen sink, and enough tools to build a sink should you want one and 2) part of the aforementioned “everything” includes a really great build system to optimize which parts you want, while transparently allowing you to load parts and pieces. Unfortunately, the Build system’s magic is mostly targeted at optimizing the files being sent on the wire. Drive space is plentiful on servers so the size impact there is nominal, but I want to ship my .air installer as small as it can possibly be, while retaining the _full_ functionality of what the Dojo Toolkit has to offer. The kitchen sink, too.

I’ve created a shell script that will eventually make its way into the Dojo Toolkit repository for easier use, and act as a starting point for some Dojo Build System optimizations regarding test-removal and cleanup. It takes the 16MB source release and trims it down to a 1.1MB archive plus three additional archives (one for each of the namespaces). If you don’t need anything from DojoX, leave it out. Just want Dojo? Leave Dijit off. The policy is additive: Dojo Base (dojo.js) is this one thing. It’s all you need to make pages rock. Dojo Core is code that’s stable and great, but optional (additive). Dojo Core makes no assumptions except the availability of Dojo Core and Base. Dijit builds on that by adding UI to Dojo Core. Depending on what you actually need, the range of file size to make a Dojo AIR application is ~100KB to ~1.1MB.

Anyone complaining about a meg for a download obviously isn’t familiar with the potential of tools available. To reiterate, 1.1MB covers the entire functional Dojo Toolkit, including Dijit, and the alpha and experimental DojoX components: Charting, Offline, DTL, Cometd … all of it.

I’m not a shell expert, and only really spent about and hour piecing this together, so hold off your optimization comments until the end. I am really just interested in showing you what you can safely remove, and what all is going on during the extended build process I’ve created.

ROOTDIR=`pwd`;
DOJOVERSION='dojo-release-1.1.0';

# if we don't have a -src build in this folder, get one
if [ ! -d $DOJOVERSION-src ]; then
  if [ ! -e $DOJOVERSION-src.tar.gz ]; then
    echo "minify: downloading source from dojotoolkit.org (hope you have wget)"
    wget http://download.dojotoolkit.org/release-1.1.0/$DOJOVERSION-src.tar.gz
  fi
  echo "minify: extracting source release";
  tar -xzvf $DOJOVERSION-src.tar.gz 1> /dev/null 2> /dev/null
fi

cd $DOJOVERSION-src/
WD=`pwd`;

First, some lightweight automated code to ‘wget’ a 16MB source release and extract it. This step is skipped if you already have a -src/ tree in the folder with the script.

We can delete a lot of files in bulk (via folders). All of these really have no place in a client download:

# pre-clean
echo "minify: removing tests/ and demos/ and cruft"
rm -rf dojo/tests/
rm -rf dijit/demos/
rm -rf dijit/bench/
rm -rf util/resources/
rm -rf util/docscripts/
rm -rf util/doh/
rm -rf util/jsdoc/

We can also bulk remove dijit/tests/, though for the sake of this script I’m going to leave two files, cleaning out the rest. The themeTester.html in dijit/themes/ will act as a single simple test to verify the toolkit is working, and it needs these two files to demo:

# remove dijit tests, leaving a single json file for combobox, 
# and _testCommon for styles so themeTester will run
cp dijit/tests/_data/countries.json .
cp dijit/tests/_testCommon.js . 
rm -rf dijit/tests/
mkdir dijit/tests/
mkdir dijit/tests/_data/
cp countries.json dijit/tests/_data/
cp _testCommon.js dijit/tests/
rm -f countries.json
rm -f _testCommon.js

We can also safely remove all the dojox/*/tests and /demos/ folders. Each DojoX project maintains its own resources/ (required), tests/ (optional), and occasionally demos/ (optional) … We can easily go through and erase the optional folders en-masse:

# clean out dojox tests and demos
cd dojox/
for i in * 
do
  if [ -d $i ]; then
    rm -rf $i/tests/
    rm -rf $i/demos/ 
  fi
done
cd $WD

We’re off to a good start. Our code is more or less free of non-essential files at this point. One further thing here: if you know you aren’t using DojoX at all, you can simply remove each folder in the dojox/ tree. You should leave the root dojox/ folder because the build system explicitly searches for content in it, but if nothing is found, it becomes a no-op.

At this point, however, our code still has comments, and our CSS is still broken into many tiny files. Running a build will solve this. First, cleanup our workarea:

# remove any old build that might be left over
if [ -d $WD/release ]; then
  rm -rf $WD/release
fi

then run the build:

# run an optimized build
echo "minify: running build"
cd util/buildscripts
# we keepLines because otherwise debugging is really hard
# , and it's a minimal size diff
./build.sh profile=standard optimize=shrinksafe.keepLines version=1.1.0mini
  cssOptimize=comments.keepLines cssImportIgnore=../dijit.css action=release

This line is taken more or less directly from Dojo’s build_release.sh script, with the addition of ShrinkSafe optimization. ShrinkSafe is Dojo’s ubercool Code packer (like Packer, or YUI Compressor), and is directly integrated into the build system. optimize=shrinksafe.keepLines tells the build to remove all comments and whitespace from every .js it encounters, preserving newlines to give us some hope of debugging should anything turn up.

The cssOptimize command removes unnecessary whitespace and comments from CSS files, as well as does automatic @import inline injecting. The Dijit themes are setup as such that you don’t need the entire theme.css file, just the components you want, and this is a way to further optimize that. The default nature, however, is to inline all the individual .css files in a theme/ folder into the single theme.css, though it leaves all the other [processed] .css files. Ignoring dijit.css allows us to reuse a common CSS file once each within each of the themes.

/* before */
@import "../dijit.css";
@import "Common.css";
@import "form/Button.css";

and after:

@import "../dijit.css";
.commonStyleA {}
.buttonStyle {}

The build actually generates additional files, so we’re not entirely done cleaning up. All the rest of this code is wrapped within a conditional, checking to see if the build ran successfully:

# discard whatever leftovers we can find
if [ -d $WD/release/dojo ]; then
  echo "minify: build success. removing uncompressed files, and optimizing themes"
  cd $WD/release/dojo
  # do more stuff
fi

So if we have a release/dojo folder, we can move on to removing any new cruft introduced by the build process. ShrinkSafe, in an attempt to be fault-tolerant, creates copies of any files it compresses, and names it uncompressed.js preceded by the original filename. Likewise, cssOptimize creates .commented.css files, retaining the inline comments. This is great for debugging after a build, but I don’t need duplicated code in this environment. Two simple commands to remove all these files get issued:

  find . -name *.uncompressed.js -exec rm '{}' ';'
  find . -name *.commented.css -exec rm '{}' ';'

The build system automatically does another cool thing: Intern Strings. Neil Roberts covered this step and how to utilize it in your own code, but we’ll mention it again for the sake of completeness. Each of the dijit’s using a templatePath to point to some static .html file has now been “interned” and converted to a string in the actual .js file referencing it. This next step is questionable, as it assumes every dijit is using templatePath: dojo.moduleUrl, as opposed to some lazy loading mechanism meant to trick the build. Turns out in the case of Dijit, it’s safe. This may affect your custom code, so omit this next step if you are doing a lot of subclassing and template replacements:

  # remove the templatePath files, which are unused internally now:
  rm -rf dijit/templates
  rm -rf dijit/form/templates
  rm -rf dijit/layout/templates

The final step in optimization: the Themes. The 1.1MB archive includes all three official Dijit themes: Tundra, Nihilo, and Soria. Each has its own images, and a vast collection of small .css files. We noted earlier about @import injection, so now after the build is complete, our themeName.css is the mashup of all the smaller files. Let’s erase all the .css files we don’t need, leaving the single .css rollup, and the images:

# we can also [in theory] remove all but the themeName.css file in themes/*.css
for t in 'tundra' 'soria' 'nihilo'
do
  cd $WD/release/dojo/dijit/themes/$t/
  mv $t.css $t.tmp
  find `pwd` -name *.cs\? -exec rm '{}' ';'
  mv $t.tmp $t.css
done

All we’ve done is renamed the rollup .css file, removed *.css recursively, and restored the file to its original filename. We’ve left some empty directories potentially, but it’s not a major concern at this point. We’ve hit the 1.1MB mark. Smallest. Dojo. Ever.

The last thing we need to do is package up the resulting filesystem, and ship it somewhere. This next segment of code does that, though can be changed to suit your needs. I have it currently creating a single mashup archive, and then an individual archive for each of the namespaces:

cd $WD/release
echo "minify: creating archive(s)"
TARBALL='dojo-release-1.1.0-mini';
mv dojo $TARBALL
tar -czvf $TARBALL.tar.gz $TARBALL/ 1> /dev/null 2> /dev/null
mv $TARBALL.tar.gz $ROOTDIR
	
# create dojo- dijit- and dojox- tarballs
mkdir $ROOTDIR/$TARBALL-parts/
for n in 'dojo' 'dijit' 'dojox'
do
  tar -czvf $TARBALL-$n.tar.gz $TARBALL/$n/ 1> /dev/null 2> /dev/null
  mv $TARBALL-$n.tar.gz $ROOTDIR/$TARBALL-parts/
done

The md5sums are generated by issuing a simple command after the archives have been placed somewhere:

dante@turtle:~> for i in *.tar.gz; do md5sum $i > $i.md5; done

Two footnotes: For backwards compatibility, I’ve left a very large file in the archive. dijit/dijit-all.js is a rollup covering most of the Dijit widget system. It is entirely unnecessary to have, though anything of yours relying on its presence will break without you adjusting your dojo.require() calls. dijit-all.js is ~275KB before gzip, so I suspect replacing the “rolled” version of dijit-all.js with the [original] version consisting entirely of dojo.require() calls to other (now optimized) files would shave even more bytes from our archive.

The other: Your own custom code can benefit from this too. Further optimization can be done, as well. Utilizing layered builds, you could (in theory) ship dojo/ (145KB) + a single layer.js of whatever size it becomes eventually, and eliminate even more ‘cruft’, which is ideal for deployment.

Final Distribution

The results of this experiment can be downloaded from my Dojo sandbox. I encourage everyone to give it a try. Download, extract, and point your 1.1-enabled application at the new dojo.js. The performance increase should be immediately apparent, and I’m very interested in how some of these techniques could be rolled into the build system directly, making optimizations like this even easier.

Progression

Final Comparison