Web Service to dojo.data Store in 4 Easy Steps

A very useful feature of Dojo is the dojox.data.ServiceStore data store. It allows you to layer a dojo.data API on top of any web service, opening up a world of uses from your own client-side components. Kris Zyp briefly mentioned the topic in his recent article on JsonRestStore, and the past couple of weeks have seen a bunch of refinements to the component to get it ready for next month’s Dojo 1.2 release. Let’s take a quick look at how to make it work with the web service of your choice.

Building the WikipediaStore

Dojo recently received a new data store that demonstrates exactly what we want: dojox.data.WikipediaStore. It does just what it sounds like, turning Wikipedia into a simple object you can query from your code. Building it with Dojo’s handy dojox.rpc package makes for a simple, compact, DRY implementation.

In only four steps:

  1. Create the web service object
  2. Declare the new data store, inheriting from ServiceStore
  3. Give it a fetch method
  4. Give it a _processResults method

That’s all there is to it.

Step 1: Create the web service object

Probably the simplest way to build an object that can talk to a web service is to create a Service Mapping Description (SMD) for the web service, and let dojox.rpc.Service do the heavy lifting. Since Wikipedia’s API can speak JSONP, it’s a natural fit. The SMD doesn’t need to be thorough, since it’s not necessarily meant for human consumption; the wikipedia.smd only defines a single API endpoint and lets the data store worry about how to talk to it:

    "SMDVersion": "2.0",
    "id": "http://en.wikipedia.org/w/api.php",
    "description": "Wikipedia API",

    transport: "JSONP",
    envelope: "URL",
    additionalParameters: true,
    target: "http://en.wikipedia.org/w/api.php",
    parameters: [
        { name: "format", optional: false, "default": "json" }

    services: {
        query: {
            parameters: [
                { name: "action", type: "string", "default": "parse" }

This simple specification is enough to let you do something like the following, which stuffs the content of Wikipedia’s Dojo Toolkit page into the current page:

dojo.require("dojo.io.script"); // for cross domain JSONP

    var mu = dojo.moduleUrl("dojox.rpc.SMDLibrary", "wikipedia.smd");
    var wikipedia = new dojox.rpc.Service(mu);

        action: "parse",
        page: "Main Page"
    }).addCallback(this, function(article){
        dojo.body().innerHTML = article.parse.text["*"];

Once we have this, we just need to wrap it in a data store.

Step 2: Declare the new data store

Out of the box, the ServiceStore does just about everything you likely need; however, it doesn’t automatically understand how to format queries for your service, nor does it automatically know the format of the incoming data. It simply acts as a bus, passing through whatever it is given. That’s acceptable if you don’t mind enforcing the underlying web service’s structure on people who use the data store, but for Wikipedia, it doesn’t take much to clean things up a little.

The declaration for the new store is pretty basic. It’s just the common dojo.declare approach (the real code has a tiny bit more than this, but this is the idea):



dojo.declare("dojox.data.WikipediaStore", dojox.data.ServiceStore,{
    constructor: function(){
        var mu = dojo.moduleUrl("dojox.rpc.SMDLibrary", "wikipedia.smd");
        var svc = new dojox.rpc.Service(mu);
        this.service = svc.query;

        // this lets ServiceStore's getLabel(), fetchItemByIdentity(),
        // etc. all work correctly
        this.idAttribute = this.labelAttribute = "title";
    fetch: function(/* object */ request){
        // to come
    _processResults: function(results, def){
        // to come

We haven’t done much yet, but we’re already halfway done! All that’s left is to shuttle data back and forth.

Step 3: Give it a fetch method

The dojo.data API uses the fetch method to query data stores, but it leaves it up to the store to determine the actual format of the query. The ServiceStore can pass query data right through to the underlying web API, but since we don’t want our data store’s users to have to know the Wikipedia API itself, we can define our own convention for creating queries and map that to the real API.

WikipediaStore defines two basic operations, query (to do full text searches) and parse (to load page data).

  • The query operation will be called thusly:
        query: {
            action: "query",
            text: "dojo"
        // onItem, onComplete, etc...
  • For parse, we want to be able to do the following:
        query: {
            // action: "parse", -- this is the default, so we can leave it out
            title: "Dojo Toolkit"
        // onItem, onComplete, etc...

So we define our own fetch that takes the request object passed in and massages it a little bit to transform our custom query structure into something that will work for Wikipedia’s API (by virtue of the dojox.rpc.Service). After doing this in-place modification, we call the ServiceStore’s fetch to do the actual RPC call:

fetch: function(/* object */ request){
    var rq = dojo.mixin({}, request.query);
    if(rq && (!rq.action || rq.action === "parse")){
        // api.php?action=parse&page=PAGE_TITLE

        // default to a single page fetch
        rq.action = "parse";
        rq.page = rq.title;
        delete rq.title;

    }else if(rq.action === "query"){
        // api.php?action=query&list=search&srwhat=text&srsearch=SEARCH_TEXT

        // perform a full text search on page content
        rq.list = "search";
        rq.srwhat = "text";
        rq.srsearch = rq.text;
            rq.sroffset = request.start-1;
            rq.srlimit = request.count >= 500 ? 500 : request.count;
        delete rq.text;
    request.query = rq;
    return this.inherited(arguments);

Now we can use our simplified dojo.data query format above to retrieve data.

Step 4: Give it a _processResults method

Once we’ve requested data, the store needs to know how to process it. That’s where the _processResults function is useful.

_processResults: function(results, def){
        // loading a complete page
        results.parse.title = dojo.queryToObject(def.ioArgs.url.split("?")[1]).page;
        results = [results.parse];

    }else if(results.query && results.query.search){
        // loading some search results; all we have here is page titles,
        // so we mark our items as incomplete
        results = results.query.search;
        var _thisStore = this;
        for(i in results){
            results[i]._loadObject = function(callback){
                    query: { action:"parse", title:this.title },
                    onItem: callback
                delete this._loadObject;
    return this.inherited(arguments);

The theory here is simple; it’s the same process as with fetch, just in reverse. We receive JSON data from Wikipedia, and we need to restructure it such that when we call the ServiceStore’s _processResults at the end, it’ll set up the internal object correctly.

The most interesting thing about our _processResults function is thection with the _loadObject callback. When we do a search on Wikipedia, all we get back is a list of page titles in request.query.search; we can use these to create stub objects, but there obviously isn’t enough data here to really use. We’d like to be able to fill in the stub object with something like store.loadItem({ item: someStubItem }), and that’s exactly what _loadObject allows: it tells ServiceStore how to load individual items (ServiceStore even supports lazy loading of objects, but only with transports that can be made synchronous; since JSONP always runs asynchronously, WikipediaStore can’t take advantage of it).

That’s really all there is to it; WikipediaStore seamlessly integrates Wikipedia into our apps. The Wikipedia store demo shows how to use it.

Integrating with Dijit

Of course, one of the big advantages of using the dojo.data API is that quite a few Dijit components can already grok it. For example, one of the demos for the Grid is built around pulling in data from Yahoo. Using the same principles as the WikipediaStore, this demo creates a YahooStore that wraps the Yahoo webSearch API endpoint. By implementing custom fetch and _processResults functions (see the component source), we get a beautifully Yahoo-ish grid.

Yahoo search ServiceStore grid demo screenshot

(Image shamelessly borrowed from Kris Zyp’s article on JsonRestStore)

So put together an SMD, a dojo.declare, and a couple of functions for your web service and create your own Dojo data store. When you can easily turn web services into data stores, the web is your oyster!

Learn more about how SitePen can be your partner

SitePen is a strategic consultancy committed to achieving technical objectives, solving critical business problems and helping our customers build web applications the right way, the first time.
Share This