At SitePen, we work very hard to to provide our customers and fellow developers with useful web app development services. We’re currently deconstructing the development of Queued, our Netflix Queue management application built with Dojo and Adobe AIR. Having presented on dojo.deferred and related topics at the Ajax Experience, and having heard that the Queued team ran into a few problems with them, I thought it would be interesting to review problems that are sometimes encountered when using Deferreds. What is the source of these problems, are these problems related to bugs or because there are misunderstandings, and how should we address this going forward?

After chatting with the team for a while and looking at their use of Deferreds, I noticed a few references and grumblings to Deferred’s silent swallowing of errors. There was even a commented out section of code containing a hacked Deferred to avoid the issue. I gathered from discussion that an error occurring NOT in the result from an XHR, but instead in its callback handler was being squelched. After discussing this issue and realizing that there was a misunderstanding of how the callback chain worked, I decided to focus on describing the callback chain and identifying any potential bugs.

Callback chains created with dojo.Deferred objects provide a standardized interface for wiring an arbitrary set of actions occurring over an unknown period of time. While they are very simple to use in most common cases, the callback and errback chain is often misunderstood, and the chaining together of Deferreds themselves even more so. ¬†¬†A decade or so ago, I remember wanting to implement something with pthreads to learn about the challenges I had read in using them. ¬†I purchased a book, Programming with POSIX Threads by Larry Butenhof, and sat down to read it. ¬†Unfortunately, I was on vacation, with no computer, and it was before I had a laptop surgically attached to my body. ¬†¬†So as I read, I didn’t actually do any of the examples or really understand how it all worked. ¬†A while passed before I got to work on that again, and so I decided to read the book again this time. ¬†For some reason, I couldn’t grasp some of the aspects around locking. ¬†I thought and thought and thought and then I thought a couple more times for good measure. ¬†All the authorities I had read on the subject described it as challenging. ¬†I couldn’t figure it out just from reading this book…it truly must be hard. ¬†¬†As I’ve found to be the case with most things I once thought to be complicated in software development, they are often actually quite easy.

I sat down and wrote a couple of small programs, and within a very short period of time it just clicked. ¬†¬†¬†It now seems obvious how they work and I can’t even seem to comprehend or describe why I had problems understanding it previously. ¬†¬†For new Dojo developers, Deferreds are a simple callback mechanism and it is pretty easy to comprehend. ¬†¬†When the applications become a little bit more complex or when developers try to figure out how to do a more complex behavior with their Deferreds, they then realize that there is more to understand. ¬†¬†Often they end up walking in my pthreads shoes or bypassing the situation altogether.

The common perception of dojo.Deferreds is that there are two simple chains, one for callbacks and one for errbacks, and these all get called in succession upon being triggered, much the same way that you might add a function for page launch in dojo.addOnLoad. ¬†This in turn leads people to believe that dojo.addCallback(cb); dojo.addErrback(eb); is the same thing as dojo.addCallbacks(cb,eb). ¬†This is NOT the case as it would be if the chain were simply two queues as is the general impression. Deferred objects in Dojo, which are directly modeled on Twisted Python’s Deferred object, contain a single queue (chain) and items are added on to the chain in tuples containing a callback and an errback. ¬†When the deferred is triggered, it will proceed down the chain, possibly changing from “callback” to “errback” as necessary.

Deferred Chain Diagram

If callbacks are added on to the chain individually, their counterpart is filled in with null.  For example:

var def = new dojo.Deferred();
def.addCallback(cb);
def.addErrback(eb);

will result in [cb, null], [null, eb] ] being placed onto the callback chain.   On the other hand if we add to the callback chain using addCallbacks():

var def = new dojo.Deferred();
def.addCallbacks(cb, eb)

The our resultant chain is a single item added to the stack, [cb, eb]. ¬†As you’ll notice in the diagram above, the red dotted lines moving from a callback to an errback go to the next item in the callback chain, not the peered item in the callback chain . ¬†If a callback throws or returns an error, it will go to the next errback handler in the chain. Likewise, if an error handler returns something other than an Error object, that output will move back to the callback side of the chain. Notably, ¬†you can continue to add onto the chain even after it has fired. ¬†This means that when you get to the end of the chain, it just holds. ¬†If the last callback in the chain throws an error, the Deferred will hold onto this error waiting for another errback handler to be added to the chain. ¬†It is this last item that makes it appear to “squelch” errors that the Queued team was seeing. ¬†However, inspection of the Deferred will show it to be sitting in an error state. Deferreds are a one-time event, but the callback chain can be added onto indefinitely.

Let’s take a look at some examples.

var cb = function(msg, d){
       console.log(msg, ": ", d);
};

var def = dojo.xhrGet({
         url: "test.txt",
         load: dojo.partial(cb, "inline callback"),
         error: dojo.partial(cb, "inline errback")
});

This is a ‘normal’ use case for the most part. ¬†An error in the request (404 for example) will trigger the error callback. Run the example.

In the following example, instead of getting an error from the IO request, we’ll use a callback with an intentional error in it, which generates an error.

var badCb = function(msg, d){
     console.log(msg, ": ", d);
     badvar.append("foo");
};

var def = dojo.xhrGet({
     url: "test.txt",
     load: dojo.partial(badCb, "BadCallback")
});

Note there is no “error” handler. ¬†The ‘badCb’ callback function will run and generate an error but it will be seemingly squelched. ¬†As mentioned above, it is not actually squelched per se, but instead just waiting to be passed on to anything added onto the chain at a later point in time. Run the example.

Taking this same code, but including an error handler, our error handler will catch this error to do with it as necessary (Example). When using the inline callbacks, they are appended to the Deferred chain in the form [loadFunc, null], [null, errorFunction].  That is, if an error occurs either as a result of the I/O operation or in the load callback, the error function will be called.   This is fine in many cases and can be exactly what you want.  With a long callback chain, however, and no error handlers or poor planning of how those error handlers are attached, it can seem as if the error was squelched.  More control can be had by adding the callbacks after creating the Deferred:

var cb = function(msg, d){
        console.log(msg, ": ", d);
};

var def = dojo.xhrGet({
         url: "/badUrl"
});

def.addCallbacks(dojo.partial(cb, "[CB,eb]"), dojo.partial(cb, "[cb,EB]"));

Our chain ends up with only a single tuple: [cb, eb]. If an error is returned from the request itself, it will get handled by the errback handler.   On the other hand if the request is successful, but there is an error generated in the callback, it will again be seemingly (but not really!) squelched.

 var cb = function(msg, d){
       console.log(msg, ": ", d);
 };

 var badCb = function(msg, d){
       console.log(msg, ": ", d);
       badvar.append("foo");
 };

 var def =dojo.xhrGet({
      url: "test.txt"
 });

 def.addCallbacks(dojo.partial(badCb, "[CB,eb]"), dojo.partial(cb, "[cb,EB]"));

The error is waiting for another errback handler to be added to the chain. Run the example.

As you can see, this provides insertion points so you can trigger errors and handle them in a variety of different ways. ¬†Remember that Deferreds aren’t necessarily associated with an I/O operation. ¬†They are simply a promise that something will happen in the future. ¬†Just because that error happens, it doesn’t necessarily mean it is the end of the road as not all errors are fatal. ¬†¬†All of this can be summed up by simply stating that you have to think about how you add items on to the chain so you can insert at the desired point. ¬†¬†You are either adding [cb, null], [null, eb], or [cb, eb] .

What about a simple example of how switching states and all of this chaining can be useful. Let’s say you are going to request a resource that may or may not exist, and if it doesn’t you want to continue using a default value. If the errback handler is added with addErrback before the callback, this can easily be achieved. The following example ends up with [null, eb], [cb, null] as its callback stack, so that when the ‘eb’ function returns an onError, the deferred will then send that value to the normal callback function.

var eb = function(err){
    return /* default object */  {}
}

var cb = function(data){
    /* do something with data */ 
}

var def = dojo.xhrGet({url: "/fubar"});

// add the errorback function first, [null, eb], 
// so that if the xhr has an error it can catch it 
// and still be able to continue to the normal chain
def.addErrback(eb);

// this chain will process either a successful return 
// from xhrGet OR the result of the errback handler
def.addCallback(cb);

Now you may be thinking that this implies you have to always end with an errback or the deferred will seem to squelch the error.  This makes it difficult to add an arbitrary number of callback items and ensure an errback is added last to catch any problems.   It is the understanding of how Deferreds themselves can be chained together that puts everything into perspective.

Dojo developers often think of Deferreds at the level of I/O, but they are much more useful than that when well thought out. ¬†Let’s create an example. ¬†¬†On our page we want to have a simple startup system ¬†that does a number of different startup tasks. ¬†For our init method, we want to initialize the communication system, and then show the user interface. ¬†However, the initComms() section can be broken down into getting the data, processing the data, and then sending off a message. ¬†By chaining deferred objects together (as opposed to adding a function onto a particular Deferred’s callback chain), complex procedures can be broken down into simple tasks, and error handling can be handled at any level or bubble up to the outermost Deferred. ¬†To chain two Deferreds together, simply add a function that will return a Deferred to another Deferred’s callback chain! ¬†For example:

var  def = new dojo.Deferred();
def.addCallback(function(){
     return new dojo.Deferred();
});

In this example, we are adding a function that returns a Deferred object. ¬†The outer Deferred will trigger the launch of the inner Deferred’s process, and will block any more callbacks until the inner has exhausted is entire callback chain. Diving right into the details, here is an example.

// Init comms function.  This will call getData() which
// returns a deferred representing the process of retrieving data
// and processing it.  After getData() is complete, 
// send a message to the server, and then log completed.

initComms = function(){
       //getData returns a deferred
       var initDeferred = getData();

       // add sendMessage to the chain.
       initDeferred.addCallback(sendMessage);

       // log this as completed
       initDeferred.addCallback(function(){
               console.log("initComms Completed");
       });

       return initDeferred;
 }

 getData = function() {
        // Request data from the server
        console.log("getData()");
        var def = dojo.xhrGet({
               url: "test.txt"
        });

        // process it in some fashion, or if there was an xhr error
        // throw the error handler
        def.addCallback(processData);

        // log that we are finished
        def.addCallback(function(){
              console.log("Get Data Complete");
        });

        //return the deferred from the initial xhr
        return def;
}

// dummy process data func
processData = function(d){
        /* do something */
        console.log("Process Data: ", d);
        return d; /* return processed data */
}

// dummy send message func
sendMessage = function(msg){
        dojo.publish("/msg", [msg]);
        console.log("Sent Message");
        return msg;
}

// showUi creates a new dojo.deferred, adds a log 
// message callback, and then returns
// The setTimeout() call is used to represent the 
// work that might go on in this function
showUi = function(){
        console.log("showUi");
        var def = new dojo.Deferred();
        def.addCallback(function(){
               console.log("showUI completed");
        });

        // for demo, trigger the showUi work with a trigger
        setTimeout(function(){def.callback(true);}, 50);
        return def;
}

// "Main"

// call initComms and get a deferred back for the whole process
var initDef = initComms();

// add showUI to the callback chain.  showUI returns a deferred object, whose
// process will be kicked off here

initDef.addCallback(showUi);

// this will not be executed until the Deferred returned from showUi 
// and all its callbacks have been completed.  We didn't add any 
// error handling in the individual functions, though we could have,
// and instead let the error bubble up to the outermost Deferred.

initDef.addCallbacks(
        function(){
              console.log("READY");
        },
        function(err){
              console.warn("ERROR Starting APP: ", err);
        }    
);

Deferreds can be both powerful and simple.  After having gone through the exercise of testing each of these scenarios and making sure that they fit the original Twisted specification, I do not believe that there is a bug here, only a misunderstanding of the errback chain.  I believe that Deferreds provide a powerful and elegant way to work with asynchronous events, but at the same time they can be difficult to follow.  

While I personally prefer using the Deferreds, many people don’t use this power and in some cases are befuddled by its behavior. Some are of the opinion that they are too complicated and shouldn’t be used for things like XHRs in Dojo going forward. What does the Dojo developer community think about the future of Deferreds for Dojo 2.x? Should we be looking at getting rid of Deferreds in Dojo? Should we simplify this into two simple queues? ¬†What’s your opinion?