Dojo-Mini: Optimization Tricks with the Dojo Toolkit

By on April 2, 2008 2:02 pm

The Dojo Toolkit 1.1 introduces support to run within Adobe’s AIR environment, and I think it may have re-sparked my interest in ActionScript. It was a lot easier than I had anticipated to get started coding, but came to a halt when it occurred to me that my newly created .air installer and badge would be at best a ~5MB download simply because of the default release size. A 5MB “Hello AIR” app is less than unimpressive, so I immediately started thinking of ways to better the situation. I need a custom minified Dojo Toolkit package.

Initial Distribution

Enter “dojo-mini” – an experiment in Dojo Toolkit deployment optimization. When Alex tagged the release, I immediately downloaded it, dropped it in as the toolkit for my upcoming AIR Demo and began browsing the source making a mental list of crufty files I could drop to shave some bytes away. I was amazed with my results:

The entire dojo.* namespace, fully functioning, is ~150KB. The entire Dijit widget system, stock, including all three stock themes (Tundra, the polished Soria, and new theme Nihilo) is ~475KB, which could be trimmed further I suspect. Every piece of DojoX combined is an additional 550KB, bringing the overhead incurred down to a modest 1.1MB. I can work with a meg.

To be fair, my “Hello AIR” sample _only_ required the 75KB (27KB w/ gzip) dojo.js, and I was able to remove the entire dojo tree with great success. The base dojo.js is so feature-rich you really don’t need much else to do really great things, though having the rest of the toolkit available would be ideal. And, like any other 27KB solution to web development, you will immediately find a need for something else, and add another piece/plugin/module/whathaveyou. It adds up, and quickly.

Two of the Dojo Toolkit’s strong points are a) It really is everything but the kitchen sink, and enough tools to build a sink should you want one and 2) part of the aforementioned “everything” includes a really great build system to optimize which parts you want, while transparently allowing you to load parts and pieces. Unfortunately, the Build system’s magic is mostly targeted at optimizing the files being sent on the wire. Drive space is plentiful on servers so the size impact there is nominal, but I want to ship my .air installer as small as it can possibly be, while retaining the _full_ functionality of what the Dojo Toolkit has to offer. The kitchen sink, too.

I’ve created a shell script that will eventually make its way into the Dojo Toolkit repository for easier use, and act as a starting point for some Dojo Build System optimizations regarding test-removal and cleanup. It takes the 16MB source release and trims it down to a 1.1MB archive plus three additional archives (one for each of the namespaces). If you don’t need anything from DojoX, leave it out. Just want Dojo? Leave Dijit off. The policy is additive: Dojo Base (dojo.js) is this one thing. It’s all you need to make pages rock. Dojo Core is code that’s stable and great, but optional (additive). Dojo Core makes no assumptions except the availability of Dojo Core and Base. Dijit builds on that by adding UI to Dojo Core. Depending on what you actually need, the range of file size to make a Dojo AIR application is ~100KB to ~1.1MB.

Anyone complaining about a meg for a download obviously isn’t familiar with the potential of tools available. To reiterate, 1.1MB covers the entire functional Dojo Toolkit, including Dijit, and the alpha and experimental DojoX components: Charting, Offline, DTL, Cometd … all of it.

I’m not a shell expert, and only really spent about and hour piecing this together, so hold off your optimization comments until the end. I am really just interested in showing you what you can safely remove, and what all is going on during the extended build process I’ve created.

ROOTDIR=`pwd`;
DOJOVERSION='dojo-release-1.1.0';

# if we don't have a -src build in this folder, get one
if [ ! -d $DOJOVERSION-src ]; then
  if [ ! -e $DOJOVERSION-src.tar.gz ]; then
    echo "minify: downloading source from dojotoolkit.org (hope you have wget)"
    wget http://download.dojotoolkit.org/release-1.1.0/$DOJOVERSION-src.tar.gz
  fi
  echo "minify: extracting source release";
  tar -xzvf $DOJOVERSION-src.tar.gz 1> /dev/null 2> /dev/null
fi

cd $DOJOVERSION-src/
WD=`pwd`;

First, some lightweight automated code to ‘wget’ a 16MB source release and extract it. This step is skipped if you already have a -src/ tree in the folder with the script.

We can delete a lot of files in bulk (via folders). All of these really have no place in a client download:

# pre-clean
echo "minify: removing tests/ and demos/ and cruft"
rm -rf dojo/tests/
rm -rf dijit/demos/
rm -rf dijit/bench/
rm -rf util/resources/
rm -rf util/docscripts/
rm -rf util/doh/
rm -rf util/jsdoc/

We can also bulk remove dijit/tests/, though for the sake of this script I’m going to leave two files, cleaning out the rest. The themeTester.html in dijit/themes/ will act as a single simple test to verify the toolkit is working, and it needs these two files to demo:

# remove dijit tests, leaving a single json file for combobox, 
# and _testCommon for styles so themeTester will run
cp dijit/tests/_data/countries.json .
cp dijit/tests/_testCommon.js . 
rm -rf dijit/tests/
mkdir dijit/tests/
mkdir dijit/tests/_data/
cp countries.json dijit/tests/_data/
cp _testCommon.js dijit/tests/
rm -f countries.json
rm -f _testCommon.js

We can also safely remove all the dojox/*/tests and /demos/ folders. Each DojoX project maintains its own resources/ (required), tests/ (optional), and occasionally demos/ (optional) … We can easily go through and erase the optional folders en-masse:

# clean out dojox tests and demos
cd dojox/
for i in * 
do
  if [ -d $i ]; then
    rm -rf $i/tests/
    rm -rf $i/demos/ 
  fi
done
cd $WD

We’re off to a good start. Our code is more or less free of non-essential files at this point. One further thing here: if you know you aren’t using DojoX at all, you can simply remove each folder in the dojox/ tree. You should leave the root dojox/ folder because the build system explicitly searches for content in it, but if nothing is found, it becomes a no-op.

At this point, however, our code still has comments, and our CSS is still broken into many tiny files. Running a build will solve this. First, cleanup our workarea:

# remove any old build that might be left over
if [ -d $WD/release ]; then
  rm -rf $WD/release
fi

then run the build:

# run an optimized build
echo "minify: running build"
cd util/buildscripts
# we keepLines because otherwise debugging is really hard
# , and it's a minimal size diff
./build.sh profile=standard optimize=shrinksafe.keepLines version=1.1.0mini
  cssOptimize=comments.keepLines cssImportIgnore=../dijit.css action=release

This line is taken more or less directly from Dojo’s build_release.sh script, with the addition of ShrinkSafe optimization. ShrinkSafe is Dojo’s ubercool Code packer (like Packer, or YUI Compressor), and is directly integrated into the build system. optimize=shrinksafe.keepLines tells the build to remove all comments and whitespace from every .js it encounters, preserving newlines to give us some hope of debugging should anything turn up.

The cssOptimize command removes unnecessary whitespace and comments from CSS files, as well as does automatic @import inline injecting. The Dijit themes are setup as such that you don’t need the entire theme.css file, just the components you want, and this is a way to further optimize that. The default nature, however, is to inline all the individual .css files in a theme/ folder into the single theme.css, though it leaves all the other [processed] .css files. Ignoring dijit.css allows us to reuse a common CSS file once each within each of the themes.

/* before */
@import "../dijit.css";
@import "Common.css";
@import "form/Button.css";

and after:

@import "../dijit.css";
.commonStyleA {}
.buttonStyle {}

The build actually generates additional files, so we’re not entirely done cleaning up. All the rest of this code is wrapped within a conditional, checking to see if the build ran successfully:

# discard whatever leftovers we can find
if [ -d $WD/release/dojo ]; then
  echo "minify: build success. removing uncompressed files, and optimizing themes"
  cd $WD/release/dojo
  # do more stuff
fi

So if we have a release/dojo folder, we can move on to removing any new cruft introduced by the build process. ShrinkSafe, in an attempt to be fault-tolerant, creates copies of any files it compresses, and names it uncompressed.js preceded by the original filename. Likewise, cssOptimize creates .commented.css files, retaining the inline comments. This is great for debugging after a build, but I don’t need duplicated code in this environment. Two simple commands to remove all these files get issued:

  find . -name *.uncompressed.js -exec rm '{}' ';'
  find . -name *.commented.css -exec rm '{}' ';'

The build system automatically does another cool thing: Intern Strings. Neil Roberts covered this step and how to utilize it in your own code, but we’ll mention it again for the sake of completeness. Each of the dijit’s using a templatePath to point to some static .html file has now been “interned” and converted to a string in the actual .js file referencing it. This next step is questionable, as it assumes every dijit is using templatePath: dojo.moduleUrl, as opposed to some lazy loading mechanism meant to trick the build. Turns out in the case of Dijit, it’s safe. This may affect your custom code, so omit this next step if you are doing a lot of subclassing and template replacements:

  # remove the templatePath files, which are unused internally now:
  rm -rf dijit/templates
  rm -rf dijit/form/templates
  rm -rf dijit/layout/templates

The final step in optimization: the Themes. The 1.1MB archive includes all three official Dijit themes: Tundra, Nihilo, and Soria. Each has its own images, and a vast collection of small .css files. We noted earlier about @import injection, so now after the build is complete, our themeName.css is the mashup of all the smaller files. Let’s erase all the .css files we don’t need, leaving the single .css rollup, and the images:

# we can also [in theory] remove all but the themeName.css file in themes/*.css
for t in 'tundra' 'soria' 'nihilo'
do
  cd $WD/release/dojo/dijit/themes/$t/
  mv $t.css $t.tmp
  find `pwd` -name *.cs\? -exec rm '{}' ';'
  mv $t.tmp $t.css
done

All we’ve done is renamed the rollup .css file, removed *.css recursively, and restored the file to its original filename. We’ve left some empty directories potentially, but it’s not a major concern at this point. We’ve hit the 1.1MB mark. Smallest. Dojo. Ever.

The last thing we need to do is package up the resulting filesystem, and ship it somewhere. This next segment of code does that, though can be changed to suit your needs. I have it currently creating a single mashup archive, and then an individual archive for each of the namespaces:

cd $WD/release
echo "minify: creating archive(s)"
TARBALL='dojo-release-1.1.0-mini';
mv dojo $TARBALL
tar -czvf $TARBALL.tar.gz $TARBALL/ 1> /dev/null 2> /dev/null
mv $TARBALL.tar.gz $ROOTDIR
	
# create dojo- dijit- and dojox- tarballs
mkdir $ROOTDIR/$TARBALL-parts/
for n in 'dojo' 'dijit' 'dojox'
do
  tar -czvf $TARBALL-$n.tar.gz $TARBALL/$n/ 1> /dev/null 2> /dev/null
  mv $TARBALL-$n.tar.gz $ROOTDIR/$TARBALL-parts/
done

The md5sums are generated by issuing a simple command after the archives have been placed somewhere:

dante@turtle:~> for i in *.tar.gz; do md5sum $i > $i.md5; done

Two footnotes: For backwards compatibility, I’ve left a very large file in the archive. dijit/dijit-all.js is a rollup covering most of the Dijit widget system. It is entirely unnecessary to have, though anything of yours relying on its presence will break without you adjusting your dojo.require() calls. dijit-all.js is ~275KB before gzip, so I suspect replacing the “rolled” version of dijit-all.js with the [original] version consisting entirely of dojo.require() calls to other (now optimized) files would shave even more bytes from our archive.

The other: Your own custom code can benefit from this too. Further optimization can be done, as well. Utilizing layered builds, you could (in theory) ship dojo/ (145KB) + a single layer.js of whatever size it becomes eventually, and eliminate even more ‘cruft’, which is ideal for deployment.

Final Distribution

The results of this experiment can be downloaded from my Dojo sandbox. I encourage everyone to give it a try. Download, extract, and point your 1.1-enabled application at the new dojo.js. The performance increase should be immediately apparent, and I’m very interested in how some of these techniques could be rolled into the build system directly, making optimizations like this even easier.

Progression

Final Comparison

Comments

  • Pingback: DojoCampus » Blog Archive » Dojo and Air, a fancy file uploader()

  • Pingback: Ajaxian » Dojo-Mini and the Feature Explorer()

  • Offtopic, but I can’t suppress my curiosity… how did you created these great looking charts? :D This are one of the best-looking pie-charts I’ve seen so far!

  • @Denis: We used Apple’s Numbers app, part of iWork 2008.

  • Thank you! Seems, like I’ll need a Mac! :D

  • James Vickers

    Interesting work you are doing.

    The graphs look nice, but are harder to read than 2D graphs, particularly with the small values on the last bar chart.

  • Pingback: SitePen Blog » I’m Not Flash()

  • Just as a note, the work from this optimization has been included in Dojo’s trunk as an optional script to run a build. It still needs a little work, but will pave the way for this kind of technique to be automated. You can safely use this script with any Dojo release >= 0.9, as it doesn’t touch anything it shouldn’t (though with older releases it may miss things it could otherwise erase):

    http://trac.dojotoolkit.org/browser/util/trunk/buildscripts/build_mini.sh

  • wiwi

    Really like the fancy diagram, can I know how you draw that? thanks!

  • @wiwi: We used Apple’s Numbers app, part of iWork 2008.

  • Pingback: DojoCampus » Blog Archive » dojo.cast() - the Dojo podcast - episode 3()

  • Pingback: DojoCampus » Blog Archive » Dojo Build 301: Compacting your build()

  • Pingback: SitePen Blog » Inside the Dojo Toolbox()

  • Pingback: Dojo 1.2 and Django 1.0 on Google App Engine 1.1.3 « Route 183()

  • Pingback: SitePen Blog » Debunking Dojo Toolkit Myths()

  • Thanks for the tutor Higgins. The link to the script file is broken. Can you fix it so that we can download the shell script?

  • No problem. I downloaded the script from http://dante.dojotoolkit.org/mini/release.sh

  • It is worth noting the code in this blog has been migrated both into Dojo’s build system (a param mini=”true”) and what mini=true does not do has been adjusted to work on a build after it has been run (util/buildscripts/clean_release.sh)

  • pollux

    Hi,i’m building dojo1.3 on windows. I am runing into troubles.
    I used the command below:
    build profile=standard mini=”true” copyTests=”false” action=release

    Oh, there are still 2 thousand files left. (GAE’s 1000 file limit per application)

    please help me! how to cut my dojo down??? how to do it on windows and linux?
    thanks in advance!

  • pollux: Quotes should not be in the build option arguments. Does this work better:

    build profile=standard mini=true copyTests=false action=release

  • pollux

    hi James Burke:
    I there are sill 2022 files after building(I just want a dojo source tree less that 1000 files). I think I should custom profile for my app. But I have no idea to do this?
    how to cut dojo down? Oh my god!

  • I believe you can gzip it and use it that way on App Engine and then it counts as one file…

  • pollux

    hi Dylan
    Thanks for your help!
    I got it!
    This is another question, Can I get rid of the files I don’t need by hands? The file size limitation of the GAE(It seems 1M)

    BTW,I think dojo should provide an smart and portable build tools for us to custom dojo package freely. :)

  • pollux: if you are just using the standard distribution of Dojo, I suggest you just use the version of Dojo on the CDN: http://code.google.com/apis/ajaxlibs/documentation/#dojo

    That way you do not have to incur the costs of hosting/serving Dojo yourself.

    If you plan on making your own custom build profile, you can aggregate all the modules you need into a build layer, and you may be able to just take that build layer and leave the rest of the built files.

    However, the build layer may depend on other files. For instance, if you use any of the i18n bundles in some of the widgets, if you use dojox.gfx, it loads some files on demand, based on the runtime environment. So if you decide to manually trim out stuff you are not using, be careful and use something like Firebug to track what files are being requested to make sure you get what you need.

  • pollux

    hi James:
    Thanks very much!
    There are some limitations if I use the CDN dojo verion.I take Dylan’s approach now and it works well.

    GAE now supports 10M uploading file. That’s good news for us. hehe :)

    Thanks again all kind guys. You rock!

  • Is it possible to deploy the gzipped file on a Websphere application server instead of unpacking it? Checking in a single file is great, 2000 files is painful

  • roofrack

    I realise this is a question probably to do with the standard build system supplied with dojo. Hope you don’t mind if I ask it – maybe someone can point me in the right direction! I only need dojox DataGrid from the distribution, but can’t find a way of ONLY including the grid package when I run a build against my profile – I get the whole of dojox, which is pointless. Can this be specified in the profile, or do I just have to delete the unwanted dojox packages myself ?