Extending Dojo 0.9: Multipart Transfers

By on September 11, 2007 8:55 pm

The release of Dojo 0.9 on August 20th marks a return to our roots: fast and lean. As Alex mentioned in a previous post, the surface area of the API has been reduced. With this API refactor, things were bound to be left out, and rightly so. The “core” of Dojo needed to be lean so we cut out the fat. But cutting out the fat sometimes removes some useful functions that people use. Fortunately, Dojo 0.9 makes it easy to add features on to it with little hassle.

One feature that wasn’t ported to 0.9’s AJAX IO was multipart transfers. I stumbled across this casualty of the refactor while working on a side-project: a port of Eugene Lazutkin’s webui for OpenWRT from Dojo 0.3 to 0.9. webui is a configuration interface for OpenWRT that takes the approach of trying to do most CPU and RAM intensive operations in the browser rather than use the limited CPU and RAM of the router. For example, when you edit the rules for the router’s firewall, the firewall rules file is downloaded to the browser and dissected in JavaScript. This makes it very easy to add rules in the right places and delete the correct rule without tasking the router with this work. After you’ve modified the firewall rules to suit your needs, the file is re-assembled and uploaded to the router using a multipart transfer. Because of this functionality, multipart transfers are vital to this application.

Even though it was left out of Dojo 0.9, this feature can easily be added into this, and any, project (because of 0.9’s well thought-out AJAX methods). Let’s take a look at the 0.4.3 implementation (I’m not going to paste the entire bind function here. If you see a “// …”, it means I “snipped” some code out for brevity’s sake. Also, some code is wrapped to the next line for formatting clarity):

this.multipartBoundary = "45309FFF-BD65-4d50-99C9-36986896A96F";
// unique guid as a boundary value for multipart posts

this.bind = function(/*dojo.io.Request*/kwArgs){
  // ...

  // guess the multipart value
  if(kwArgs.method.toLowerCase() == "get"){
    // GET cannot use multipart
    kwArgs.multipart = false;
  }else{
    if(kwArgs["file"]){
      // enforce multipart when sending files
      kwArgs.multipart = true;
    }else if(!kwArgs["multipart"]){
      // default 
      kwArgs.multipart = false;
    }
  }
  
  do { // break-block
    // ...

    if(kwArgs.file){
      if(dojo.lang.isArray(kwArgs.file)){
        for(var i = 0; i < kwArgs.file.length; ++i){
          var o = kwArgs.file[i];
          t.push(  "--" + this.multipartBoundary,
              "Content-Disposition: form-data; 
                name="" + o.name + "";
              filename="" + ("fileName" in o ? o.fileName : 
                o.name) + """,
              "Content-Type: " + ("contentType" in o ? 
                o.contentType : 
                "application/octet-stream"),
              "",
              o.content);
        }
      }else{
        var o = kwArgs.file;
        t.push(  "--" + this.multipartBoundary,
          "Content-Disposition: form-data;
             name="" + o.name + "";
           filename="" + ("fileName" in o ? 
            o.fileName : o.name) + """,
          "Content-Type: " + ("contentType" in o ? 
            o.contentType : "application/octet-stream"),
          "",
          o.content);
      }
    }

    if(t.length){
      t.push("--"+this.multipartBoundary+"--", "");
      query = t.join("\r\n");
    }
  }while(false);

  // ...

  var http = dojo.hostenv.getXmlhttpObject(kwArgs);

  // ...

  if(kwArgs.method.toLowerCase() == "post"){
    // ...

    http.setRequestHeader("Content-Type", kwArgs.multipart ? 
      ("multipart/form-data; boundary=" + this.multipartBoundary) : 
      (kwArgs.contentType || "application/x-www-form-urlencoded"));
    try{
      http.send(query);
    }catch(e){
    // ...
  }
// ...
}

Let’s take a look at this code. The first thing you’ll notice (after the boundary variable) is multipart transfers must be POSTs. This makes sense because you wouldn’t want to transfer the contents of a file in the URL of a GET request :). Dojo 0.9 has two POST methods: dojo.xhrPost and dojo.rawXhrPost. The former takes the content property of the arguments and turns it into a POST query. The second takes the postData property of the arguments and passes it directly to the POST query (no transformations). Obviously, we’ll be using dojo.rawXhrPost.

Moving on to inside the “do-while”, the file property starts to get parsed. Note that the file property is either an array containing objects or an object. The objects (or object) require a name and content property; you can optionally include fileName and contentType properties.

After all of the file objects are parsed, it checks to make sure t has something in it, pushes the final boundary onto t, and joins the strings with “\r\n”. If we were sending a file with content set to “We’re sending a multipart file” and name set to “foo.txt”, query would contain:

--45309FFF-BD65-4d50-99C9-36986896A96F
Content-Disposition: form-data; name="foo.txt"; filename="foo.txt"
Content-Type: application/octet-stream

We're sending a multipart file
--45309FFF-BD65-4d50-99C9-36986896A96F--

At the very end, the XMLHttpObject is set up with a request header that tells the server that it’s going to be getting a multipart transfer and what the boundary will be. Once that is done, it is sent off.

As you can see, there isn’t a lot to do: take a property from an object passed in, wrap its contents in a multipart boundary wrapper and send it using a raw POST. We also want this function to act like the other Dojo 0.9 IO methods. This should be fairly simple. The first thing we’ll do is define the function:

xhrMultiPart = function(args){
...
}

Next, let’s make sure we check for the file property of the args object since it’s required and let’s define the boundary string as a variable in this function:

  if(!args["file"]){
    throw new Error("file must be provided to xhrMultiPart's arguments");
  }

  // unique guid as a boundary value for multipart posts
  var boundary = "45309FFF-BD65-4d50-99C9-36986896A96F";

So far, so good! You’ll notice that we skipped the tests to make set the multipart flag on the args object. Since we know that this will always be a multipart raw POST, we don’t have to set any special flags to tell Dojo’s AJAX functions to handle it differently. It saves a lot of code in the long run!

The next task is to take that required file property and turn it into multipart data. We’re going to do something a little different so we don’t have to duplicate any code:

  var d = (dojo.isArray(args.file) ? args.file : [args.file]);

This bit of code transforms args.file into an array if need be. This will allow us to use the same code to generate the multipart content whether args.file is an array of objects or one single object. Let’s port that over too:

  var tmp = [];

  for(var i=0; i < d.length; i++){
    var o = d[i];
    var fileName = (typeof o["fileName"] != "undefined" ? 
      o.fileName : o.name);
    var contentType = (typeof o["contentType"] != "undefined" ? 
      o.contentType : "application/octet-stream");

    tmp.push("--" + boundary,
         "Content-Disposition: form-data; name="" + 
          o.name + ""; filename="" + fileName + """,
         "Content-Type: " + contentType,
         "",
         o.content);
  }

  var out = "";
  if(d.length){
    tmp.push("--"+boundary+"--", "");
    out = tmp.join("\r\n");
  }

Notice the changes we’ve made: instead of using “member” in object notation for the fileName and contentType, we’ve used typeof object[“member”] != “undefined”. Other than that, it’s pretty much the same.

All we need to do is to send this data off to the server:

  return dojo.rawXhrPost(dojo.mixin(args, {
    contentType: "multipart/form-data; boundary=" + boundary,
    postData: out
  }));

Here, we mix the contentType and postData into the args variable. We don’t want to do it the other way around because that would allow those two arguments to be overridden when this function is called.

And there you have it! Here it is in all its glory:

xhrMultiPart = function(args){
  if(!args["file"]){
    throw new Error("file must be provided to xhrMultiPart's arguments");
  }

  // unique guid as a boundary value for multipart posts
  var boundary = "45309FFF-BD65-4d50-99C9-36986896A96F";

  var d = (dojo.isArray(args.file) ? args.file : [args.file]);
  var tmp = [];

  for(var i=0; i < d.length; i++){
    var o = d[i];
    var fileName = (typeof o["fileName"] != "undefined" ? 
      o.fileName : o.name);
    var contentType = (typeof o["contentType"] != "undefined" ? 
      o.contentType : "application/octet-stream");

    tmp.push("--" + boundary,
         "Content-Disposition: form-data; name="" + 
          o.name + ""; filename="" + fileName + """,
         "Content-Type: " + contentType,
         "",
         o.content);
  }

  var out = "";
  if(d.length){
    tmp.push("--"+boundary+"--", "");
    out = tmp.join("\r\n");
  }

  return dojo.rawXhrPost(dojo.mixin(args, {
    contentType: "multipart/form-data; boundary=" + boundary,
    postData: out
  }));
}

To use this new function, you’d call it much like you would any of Dojo 0.9’s AJAX IO functions:

xhrMultiPart({
  url: "/path/on/server/",
  file: {
    name: "upload",
    content: "I'm sending a multipart transfer!"
  }
}).addCallback(function(data){
  // do something once the transfer is done
  return data;
});

Or, for multiple files:

xhrMultiPart({
  url: "/path/on/server/",
  file: [
    {
      name: "upload1",
      content: "I'm sending a multipart file!"
    },
    {
      name: "upload2",
      content: "This is another multipart file."
    }
  ]
}).addCallback(function(data){
  // do something once the transfer is done
  return data;
});

You may be asking me, “But Bryan, I’d really like a more random boundary… is that possible?” That’s a good question and a problem worth solving. We need a way to produce a fairly random string of characters. One solution that was presented to me was to use the MD5 sum of a date string appended with a random number. This would be great, except the MD5 routines in DojoX add about 5K to a build using this function, and this should really be a lean routine. Instead, we’re going to use dojox.uuid.generateRandomUuid():

dojo.require("dojox.uuid.generateRandomUuid");

xhrMultiPart = function(args){
  if(!args["file"]){
    throw new Error("file must be provided to xhrMultiPart's arguments");
  }

  // unique guid as a boundary value for multipart posts
  var boundary = dojox.uuid.generateRandomUuid();

  var d = (dojo.isArray(args.file) ? args.file : [args.file]);
  var tmp = [];

  for(var i=0; i < d.length; i++){
    var o = d[i];
    var fileName = (typeof o["fileName"] != "undefined" ? 
      o.fileName : o.name);
    var contentType = (typeof o["contentType"] != "undefined" ? 
      o.contentType : "application/octet-stream");

    tmp.push("--" + boundary,
         "Content-Disposition: form-data; name="" + 
          o.name + ""; filename="" + fileName + """,
         "Content-Type: " + contentType,
         "",
         o.content);
  }

  var out = "";
  if(d.length){
    tmp.push("--"+boundary+"--", "");
    out = tmp.join("\r\n");
  }

  return dojo.rawXhrPost(dojo.mixin(args, {
    contentType: "multipart/form-data; boundary=" + boundary,
    postData: out
  }));
}

And there you have it. xhrMultiPart can now be found in DojoX. Next time, I’ll build on top of this example to create a Dijit to improve the user experience with file uploads.