Dojo Object Stores

By on February 15, 2011 12:02 am

Dojo 1.6 introduces a new data store API called Dojo Object Store. This new store API is based on the HTML5 IndexedDB object store API and is designed to greatly simplify and ease the interaction and construction of Dojo stores. Based on HTML5 IndexedDB

Update: dstore is developing as a successor to Dojo object stores, so if you are starting a new application, you may want to look into this project. However, this blog post still describes an important API that is widely used, and is foundational for dstore, sharing much of the same philosophy and even much of the interface.

Dojo 1.10 also includes new stores in dojox/store for local database support. Check the latest blog posts for more information about this.

There are several key philosophies behind the Dojo object store API:

  • Separate UI from data concerns – This has long been the motivation for our data API. This API helps us to separate concerns and permits independent evolution of widgets and data providers.
  • Keep it simple – You can literally create a usable store by simply creating an object instance with query() and get() methods that return objects. As you add functionality like creating new objects, you add an add() method, and for updating objects you include a put() method.
  • Plain JavaScript Objects – Rather than using opaque items, the new object store API uses plain JavaScript objects. Once a get() method has returned an object, or a query() method has returned an array of objects, you can access and enumerate properties using standard JavaScript properties and for-in loops. Saving objects is a matter of passing a plain object to a put() method.
  • Promise-based – The interface is the same for sync and async except sync methods directly return values and async methods return promises. Any method can choose to be sync or async (promise-returning). This greatly simplifies interaction with known synchronous stores, since callbacks aren’t needed. This also greatly simplifies the API since asynchronous concerns are separated from the interface.
  • Simple feature discovery – No extra methods are needed to determine the feature set of a store. If you want to know if you can add new objects to a store, just check if it has an add() method; if you want to see if it is queryable, check if it has a query() method. If you want to create a read-only store, only implement the read functions (get, query, getIdentity).
  • Layered functionality – By layering functionality, it is possible to start with a lightweight, simple store and then add functionality as needed. Dojo’s core comes with a caching wrapper (dojo/store/Cache) for improving performance and a wrapper for notification of data changes (dojo/store/Observable). These store wrapper/layer modules are optional and can be added to any other store. This makes it possible for us to keep our core stores — dojo/store/Memory and dojo/store/JsonRest — very simple and small, and makes it possible for you to easily create new stores.

Dojo Object Store API is an interface between different data consumers to different data producers. This interface can have any implementation, and Dojo core comes with two key implementations that are commonly needed: dojo/store/Memory and dojo/store/JsonRest.

Simple: dojo/store/Memory

This is a very simple in-memory store. This is highly useful for quickly creating a store, particularly for smaller datasets. A Memory store can be created by simply providing a plain array of objects as the data source for this store, and then you can start querying and interacting with the store (see the Memory object store documentation for more usage details).

The Memory store is a synchronous store, which means it directly returns values, making it very easy to use. For example, to retrieve an object by id:

var product = productStore.get("slinky");

And again, the new object store returns plain objects, so we can get properties with simple property access:

var name = product.name;

Using plain objects makes updates easy as well:

product.name = "New name";
productStore.put(product);

One of the exciting new features in Dojo object stores is improved querying. Queries are made using the query() method and the returned result set provides convenient iterative methods — much like dojo.query — that can be consistently used regardless of whether the store is sync or async. Therefore, we can rely on using forEach, map, or filter methods. The memory store supports several forms of querying. First, we can query by name-value matches (like Dojo Data’s ItemFileReadStore). Here we query by category:

store.query({category:"shoe"}).forEach(function(shoe){
  // called for each match
});

Name-value matching provides a simple, easy query mechanism, but sometimes more complex queries are needed. The Memory store also accepts functions for filtering, thus allowing arbitrarily complex queries. For example, to query by products with a price less than 10:

store.query(function(product){
  return product.price < 10;
}).forEach(function(shoe){
  // called for each match
});

We can also refer to functions by name, and they will be looked up as a method on the store:

store.lessThanTen = function(product){
  return product.price < 10;
});
store.query("lessThanTen").forEach(function(shoe){
  // called for each match
});

JSON and REST: dojo/store/JsonRest

JsonRest assumes the existence of a server-side API, and is designed for interaction with a store. This implements a solid, standards-compliant HTTP/REST client interface to a server. The dojo/store/JsonRest follows REST principles for high scalability and is well-suited for large datasets. JsonRest is an asynchronous store, and returns promises from all the operational methods that are asynchronous (the exception: getIdentity should always be synchronous).

The JsonRest object store follows much the same approach to HTTP-compliant server interaction as dojox.data.JsonRestStore. However, JsonRest has been greatly simplified in the redesign for the new store API. A JsonRest store can be created by simply providing a URL to connect to a server (see the JsonRest object store documentation for more usage details):

require(['dojo/store/JsonRest'], function (JsonRest) {
  store = new JsonRest({target:"/Data/"});
  ...

Store methods intuitively map to HTTP methods. Calling store.get("some-id") would result in a GET request sent to /Data/some-id and return a promise/Deferred for the result. For example:

store.get("some-id").then(function(someObject){
  // use someObject
});

Calling store.remove(id) will similarly result in a DELETE request. The add(object) and put(object) methods will trigger requests too. If the object passed to put(object) (or add(object)) includes the identity property, a PUT request will be used. If the object does not include an id or the second argument (options) includes an incremental property set to true, a POST request will be used.

The request will include an If-Match: * header if options.overwrite is true or an If-None-Match: * header if options.overwrite is false or if add(object) is used. This communicates to the server when object creation or modification is expected.

Normalizing sync and async

If you are writing a widget that may be used with either a sync or an async store, we recommend leveraging dojo/when. Rather than relying on a then() method, which is only available if the store method is async, we can use dojo/when and any return value will be properly handled. For example if we were doing a get() with an unknown store, we could write:

require(['dojo/when'], function (when) {
  when(store.get(id), function(object){
    // this function will be called with object once get() is complete, 
    // regardless of whether it returns a promise, or directly and 
    // immediately returns a value
  });
  ...

The dojo.when() function can be used with any store method.

Client-Side Data Caching: dojo/store/Cache

In addition to the core store implementations, Dojo comes with two store wrappers. The first wrapper is dojo/store/Cache. This wrapper is called with two stores: a caching store and a master store. A typical scenario for using the Cache wrapper is using JsonRest as a master store and a Memory store as a client-side caching store. This allows you to leverage the JsonRest store for server communication of changes, while using the Memory store for caching and avoiding unnecessary HTTP requests for data. Here is an example of how we could set this up:

require(['dojo/store/Memory', 'dojo/store/JsonRest', 'dojo/store/Cache'],
    function (Memory, JsonRest, Cache) {
  memoryStore = new Memory({});
  restStore = new JsonRest({target:"/Data/"});
  store = new Cache(restStore, memoryStore);
  ...

Now we could perform a query with our aggregate store. Here we will just query for all the objects (we can query for everything by omitting a query):

var results = store.query();

This will result in the response being cached in the memory store. Later we can get() an object without incurring an HTTP request:

object = store.get("some-id");

Changes to data through put(), add(), and remove() are all reflected in the cached data. Querying generally requires finer grained application control over exactly what should be cached and what should not. Consequently, the Cache store does not attempt to automagically query against the cache. However, if you choose to query using the cache, it is trivial. Simply query the caching store, the memoryStore in our example:

memoryStore.query({category:"shoe"}).forEach(...);

Observing Data Updates: dojo/store/Observable

Dojo also comes with a store wrapper/layer for adding support for notification events of data changes. The Dojo object store API takes a very different approach to notifications than the legacy Dojo Data API. The old API was problematic in that notifications were store-wide, and it was impossible to determine exactly how an event should really affect a rendered result set.

The Dojo object store solves this problem by connecting the observation of notification events to query result sets instead of the store. With the dojo/store/Observable module, you can wrap a store and the resulting store will always have "observable" result sets returned from queries. That is, the object/array/promise returned from the query() method will have an observe() method that can be called to monitor the result set for changes. See the Observable store wrapper documentation for the exact signature of the observe() method and callback.

The Observable module makes it extremely easy to render a result set and then respond to any changes in the underlying data with real-time UI updates. Let's look at an example. We will create an unordered list (<ul>) of items corresponding to objects in storage. We will first create the list, and then respond to changes:

require(['dojo/store/Observable', 'dojo/dom'],
    function (Observable, dom) {
  // first add observable functionality to our store
  store = new Observable(store); 

  var listNode = dom.byId("list");
  var itemNodes = [];
  // now we will do the query for data
  var shoes = store.query({category:"shoe"});

  // and then render each item returned
  shoes.forEach(function(shoe){
    // and we render each node
    insertRow(shoe, itemNodes.length);
  });

  // now we monitor the result set for any changes
  shoes.observe(function(object, removedFrom, insertedInto){
    if(removedFrom > -1){ // existing object removed
      dojo.destroy(itemNodes[removedFrom]);
      itemNodes.splice(removedFrom, 1);
    }
    if(insertedInto > -1){ // new or updated object inserted
      insertRow(object, insertedInto);
    }
  });
  function insertRow(product, index){
    return itemNodes.splice(index, 0, dojo.create("li",
      {innerHTML: product.name + ": " + product.price}, listNode));
  }

By setting up an observing function that can remove rows and insert rows, we are properly set to respond to any changes, including additions, deletions, and object updates. The Observable module even sends index updates such that updated objects will properly be moved to a new index, if there are sort order changes within the result set. Also note that by observing the shoes result set in the example, we will only get updates that match the criteria for this particular result set. If an object that is updated, deleted, or added, and it does not have a category of “shoe”, no notification events will be posted for this listener. If an object was not a "shoe" before and is updated to be a "shoe", this will trigger a notification of an addition to the result set. If an object was a "shoe" before and is updated to not be a "shoe", this will trigger a notification of a removal from the result set.

The Observable module also adds a notify() method to stores. This is extremely useful for Comet-driven real-time applications that asynchronously receive updates from a server and wish to notify the store (and all the store's result set listeners) of the change. Be sure to check out Dojo Socket for more details on Comet and real-time apps with Dojo 1.6.

Working with Existing Widgets and Stores

Most of the Dijit widgets are still based on the legacy Dojo Data API. However, Dojo comes with an adapter for using a new object store with a Dojo Data-based widget. The dojo/data/ObjectStore module is an adapter that accepts an object store and returns a data store. Dojo also comes with an adapter for using a legacy data store with widgets that expect an object store. The dojo/store/DataStore module is an adapter that accepts a data store and returns an object store.

Hierarchy

The object store API defines a method for hierarchy, created by providing a getChildren(object, options) method. getChildren should be called with a parent object, and returns a set of children. The implementation of getChildren is generally application-specific, but there are a couple of common ways to implement hierarchy:

  • Objects include a "children" property with an array of children - With this approach, each object defines its children in an array, thus specifically preserving the order of children. This works well for smaller ordered data sets.
  • Objects include a parent reference - With this approach, each object defines its parent, and we retrieve children by querying for all objects with a given parent id. This works well for larger data sets that may involve additional handling like paging or sorting.

Adapting to Other Stores

Update: the dojox.storage are very old, legacy modules, and should probably be avoided. This section is included for historic/informative purposes. Because the store API is a common pattern, there are a number of libraries that can easily be adapted to the store API. The dojox.storage providers are a close match, except the put() signature is slightly different. This can easily be adapted:

var storage = dojo.delegate(dojox.storage);
var storage.put = function(object, options){
  var deferred = dojo.Deferred();
  dojox.storage.put(options.id || object.id, object, function(status){
    if(status == dojox.storage.FAILED){
      deferred.reject(status);
    }else if(status == dojox.storage.SUCCESS){
      deferred.resolve(status);
    }
  });
  return deferred;
};

Or we could adapt to Jens Arps StorageJS library, which uses set() instead of put():

var store = dojo.delegate(storage);
var store.put = function(object, options){
  return storage.set(options.id || object.id, object);
};

The StorageJS API also has an allKeys() method that could be converted to a query() method.

Object Stores

The new Dojo object store infrastructure has been redesigned from the ground up to incorporate the best ideas from dojo.data, standardize around HTML5 IndexedDB, simplify usage, and layer functionality. We are excited to start building applications with this new approach, and we are looking forward to the Dojo community's feedback on this design.

Comments

  • Excellent redesign: hits all of the things I found counter-intuitive about the original dojo.data APIs (having to get fields via the fiddly pass-item-back-to-store method, sync support) while retaining all the powerful features. Great job!

  • Doug Holton

    Just curious if you can dojo.store.cache using a persistent client-side db (web storage, web sql, indexeddb) instead of just an in-memory store. For example say I have a (relatively small) dictionary with about 10,000 words or whatever. I’d rather not have them have to download the entire dictionary up front, but I’d rather not have them do server fetches each and every time, also, or have to re-download items every time they re-run the app.

  • @Kris

    Another great writeup. Just a few nits:

    “If the object does not include an id or the second argument (options) includes an incremental property set to true, a POST request will be used.”

    Calling this property “incremental” seems a bit too specific when what you really want is the store to generate your identity for you, which could just as well be a uuid. How the store generates your identity is up to the store itself, right?

    Also, the dojox.storage example looks like it’s missing a `return deffered.promise;` in the dojox.storage.put function.

    @Doug

    You could absolutely use dojo.store.cache with a store that uses client-side persistence — you just need to write the store first :)

    Fortunately authoring stores is fairly easy. It probably wouldn’t take too much work to refactor perstore’s sql store for the web sql database API. The IDB API ought to be pretty straitforward too (considering it’s the inspiration). Of course, queries are cursor driven and it’s built on an event model instead of returning promises, but both of these differences are pretty easy to paper over.

  • @Dean, thanks for code correction. The “incremental” property isn’t intended for indicating a request for auto-generated id, rather it is intended for partial updates (incremental instead of a entire object PUT). Since a partial updates doesn’t match the semantics of PUT, it is best done with a POST.

    Indeed, the Perstore SQL module should work well for web SQL databases, since that is what is intended to wrap: https://github.com/kriszyp/perstore/blob/master/lib/store/sql.js

  • Pascal Robert

    I’m trying this:

    memoryStore = new dojo.store.Memory({});
    restStore = new dojo.store.JsonRest({target: ‘/cgi-bin/WebObjects/WOWODC.woa/-5667/ra/conferenceSessions.json’});
    store = new dojo.store.Cache(restStore, memoryStore);
    var results = store.query();
    console.log(store.get(“28”));

    store.query() did return all objects, so that’s fine. But store.get(“28”) still try to make a network request instead of looking at the memory store.

  • @Pascal: It is little hard to tell without looking at the HTTP response. Are the returned objects being populated into the memoryStore (you can look at the index property of the memoryStore to check).

  • Mark

    Hope you don’t mind comments on these older blog entries but having been trying to implement store based on this article and have a question.
    I am using the new dojox.mobile.EdgeToEdgeDataList to display data from a store. If I just use an ItemFileWriteStore then when I add a new item to the store the list is automatically updated.
    If I instead use a Memory store and a dojo.data.ObjectStore as mentioned above then I need to call refresh on the list to see any new items.
    Is calling refresh the right way to do this or should I be setting up an ‘Observable’ to do this for me somehow.
    Just trying to figure out the ‘right’ way to manage all this

    Thanks

  • @Mark: In 1.6 you do need to refresh. In 1.7, the ObjectStore will work with Observable wrapped stores so that changes are propagated through the dojo.data store.

  • Pingback: Code Design and Approach for the Next Grid()

  • Superbird

    One missing features in dojo/json/rest is the support for json reference or lazy loading as in the jsonRestStore. Is there any plan to add it ?

  • @Superbird: We would like to add this functionality as separate wrapper module that would add this functionality as needed.

  • richso

    am I right that the current dojo.store.Memory is not yet making use of HTML5 localStorage (IndexedDB) ?
    if so then when will there be a dojo class for modeling the use of localStorage (IndexedDB) as dojo object store ?

  • Correct. My expectation is that we will work to integrate StorageJS ( https://github.com/jensarps/StorageJS ) into a future update.

  • DannyLi

    @richso, I just made a dojo object store,based on indexedDB. you may find it helpful.
    https://github.com/ChangdongLi/Dojo_IndexedDB_Store

  • Ken

    I need the functionality of the EnhancedGrid to persist to local storage, working in offline mode.
    Is there a LocalStorageStore available that works like the ItemFileWriteStore for the EnhancedGrid?
    Also, what methods are available to detect changes in localStorage that then update the EnhancedGrid values?

  • In order to get my apps ready for dojo 2.0 I’m wondering will dojo/data (ItemFileReadStore) completely disappear or will it still be available.

  • @Peter, you should look to migrate to either dojo/store/Memory, or perhaps one of the new extensions at https://github.com/kfranqueiro/dojo-smore

  • @Dylan, Thanks for your response however, let me be a little more specific. I provide the generic CheckBox Tree at https://github.com/pjekel/cbtree which currently supports the dojo/data stores. In addition, the 1.9 branch adds support for the dojo/store however, going forward (dojo 2.0?) I would like to be able to support just one and drop all legacy support.
    The dojo 2.0 migration note (http://livedocs.dojotoolkit.org/releasenotes/migration-2.0) state that the dojo.data API has been replaced with the new dojo/store API, again does that mean dojo/data will no longer be available at 2.0?
    I do understand that providing support for any legacy stores is a personal choice but I’m looking for a more “official” statement from the dojo team, is dojo/data in or out :)

    Finally, a note on the dojo/store API. The dojo/store API states, and I quote, “Every method may return a promise for the specified return value if the execution of the operation is asynchronous” however, the API does not specify the expected behavior of such promises. For example, if a store.get() retruns a promise and the requested object does NOT exists does the promise resolve with “undefined” or is the promise rejected? The fact that an object doesn’t exist does not constitute an error. This is vital information for store implementers or anybody who builds on top of the dojo/store API.

    Thanks in advance.

  • Oz

    Promises are callbacks! And it has always been possible to handle sync operations with callbacks if you want to hide the differences between sync and async.

    All promises do is provide a rather nice way of dealing with callbacks.

  • The query is async, then you want to do results.then(console.log(store.get(“28”)));

  • @Oz While true, the ability to cancel or resolve them, and have them work with values that have already been fulfilled, etc., gives you more control over their state than a traditional callback. Also, new features in the DOM that are async have a promises API, as well as them being a new native feature in EcmaScript 6.

  • Pingback: Introducing the Next Grid: dgrid | Blog | SitePen()