More Query and Caching Power for Data Stores: ClientFilter, JsonQuery, and JsonQueryRestStore

By on December 18, 2008 1:02 am

Dojo includes several new modules which open up new querying and caching capabilities for Dojo data stores. dojox.data.ClientFilter is available in Dojo 1.2, and provides the ability to cache queries, update the query results and cached queries based on local modifications, and performs sub-queries and sorting on the client-side. The JsonQuery is a new mixin class for Dojo 1.3 that converts Dojo query objects and sort objects to JSONQuery expressions; JSONQuery can be used for delivering complex queries to the server. JsonQueryRestStore is a new module (for Dojo 1.3) that extends the JsonRestStore and combines the ClientFilter and JSONQuery such that any query can be executed on the client or the server automatically based on what data has been locally cached on the client-side, utilizing dojox.json.query to execute queries on the client-side when appropriate.

ClientFilter

ClientFilter is a base class that can be extended by other stores, to add client-side querying and sorting functionality. ClientFilter uses these capabilities for two ends. First, queries can be cached and the cached queries can later be used for other queries and sorting operations as long as the query contains the same or superset of the data required by a subsequent query. In order to use the query cache, a ClientFilter subclass should call ClientFilter’s cachingFetch method from its fetch method. Also, to cause queries to be cached, the argument passed to the fetch method should have a queryOptions property set to {cache: true}. For example,

dojo.declare("MyStore",dojox.data.ClientFilter,{
   fetch: function(args){
      var deferred = this.cachingFetch(args);
      ...
   },
   ...
});
myStore = new MyStore();
myStore.fetch({query:{name:"value"},queryOptions:{cache:true}});

The ServiceStore and its subclasses (which includes JsonRestStore and it’s subclasses) will automatically use ClientFilter (and cachingFetch) for you if it has been loaded.

dojo.require("dojox.data.ClientFilter");
dojo.require("dojox.data.JsonRestStore");
myStore = new dojox.data.JsonRestStore({target:"/data"});
myStore.fetch({query:{name:"value"},queryOptions:{cache:true}});

Widgets, like the grid, can take queryOptions for passing to the store:

table dojoType="dojox.grid.DataGrid" queryOptions="{cache:true}" ...

Alternately, you can set the cacheByDefault to true on the store object, and all queries will be cached:

myStore.cacheByDefault = true;

When the fetch is executed the results will be placed in the query cache. If the request is repeated the cached version can be retrieved rather than requiring another request to the server. Also, if the the same request is made with different sort parameters (the result of the clicking on a column in the grid), the cached result set can be used and sorted on the client-side to satisfy the request.

The second goal of the ClientFilter is to update result sets after local modifications have taken place. If a new item was added or an item was updated, the display of the data should usually be updated, but a store user, like a widget, may not know how to update the collection that is being displayed. Often widgets simply add new items to the end of a list and if an item is modified, no re-sorting is done. However, the ClientFilter can actually update previous query results based on object modifications; new items and modified items being added, moved, or removed to the correct place in a result set based on the query and sorting parameters. For example, suppose we performed a query:

var requestArgs = {sort:[{attribute:"firstName"}], onComplete:function(results){
   fetchResults = results;
};
myStore.fetch(requestArgs);

And then we modified one of the items:

myStore.setValue(fetchResults[1], "firstName":"John");

The fetchResults was originally sorted by firstName, but after this modification, the sorting may now be incorrect. With a ClientFilter powered store, we can update this result set:

myStore.updateResultSet(fetchResults, requestArgs);

This will iterate through all the recent modifications and update the fetchResults array, moving, adding, and removing items as necessary to ensure the result set is correct. Furthermore, this can be very valuable for new queries while items are dirty. With the item modification we made above, if we haven’t yet called save(), the server will not be aware of this change. If we did another fetch that actually made a server request, the server may return the wrong results based on the sorting or query parameters due to its lack of knowledge of the current state of client-side data. ClientFilter can solve this problem as well: it will automatically apply the updateResultSet method to all queries, updating the results based on any local modifications (dirty items) that haven’t been sent to the server.

The ClientFilter can perform client-side queries based on name-value pair query objects, and sort parameters. However, out of the box, the ClientFilter does not understand string-based queries; the Dojo Data API specifies that string queries are store specific so it is impossible for ClientFilter to know how to interpret all string-based queries. However, you can implement two simple methods to enable caching and client-side querying of a string-based queries. clientfilter.png isUpdateable(request) should be implemented, taking the same argument as the fetch method and returning a boolean indicating whether or not the query can be updated (handled on the client-side). matchesQuery(item, request) should also be implemented, taking an item and returning a boolean indicating whether or not the item should be included in the query providing by the second argument. By implementing these functions, ClientFilter can be made to work with string-based queries. Or alternately, you can use the JsonQuery module to implement these functions for JSONQuery based queries.

JsonQuery

dojox.data.util.JsonQuery is a module that can convert Dojo Data query objects and sort attributes to JSONQuery expressions for the server and also implements isUpdateable(request) and matchesQuery(item, request) for JSONQuery. JSONQuery can be used to describe substantially more complex queries than simple URL query parameters. JSONQuery supports a full range of operators (<, >, !, &, |, =, etc.), sorting, recursive search, and more. The JsonQuery module is an abstract mixin class, so you can add the JSONQuery capabilities to a store:

dojo.declare("JsonQueryEnabledSomeStore", [SomeStore, dojox.data.util.JsonQuery]);

JsonQueryRestStore

dojox.data.JsonQueryRestStore extends JsonRestStore and combines the power of ClientFilter and JsonQuery. With JsonQueryRestStore, JSONQuery can be used to formulate queries that can be executed either on the client or the server. JsonQueryRestStore utilizes the JsonQuery module to transform name-value object queries to JSONQuery format and integrates with the ClientFilter module such that queries and sorting are executed on the client-side when cached results are available, and sent to the server as needed. Note that the queryOptions.cache must still be set to true to cache queries with the JsonQueryRestStore.

Conclusion

ClientFilter provides a foundation for optimizing data stores, using cached data as appropriate to avoid server-side requests for lower latency updates. JSONQuery is a powerful query language which can easily be used via the new Dojo Data JsonQuery module (in Dojo 1.3). JsonQueryRestStore is a new JsonRestStore subclass which provides a complete solution for consistent client and server-side querying, automatically handling the use of query caches for optimum performance.

Comments

  • Andrea

    Can we use JsonQueryRestStore with OfflineRest ?

  • @Andrea:
    Yes, did I forget to mention that in this post? They are indeed built to work together (although you may need to ensure that you do an initial cached query on locally persisted query for the JsonQueryRestStore to properly do the in-memory querying).

  • Christer

    Hi,

    Can the ClientFilter be used with AndOrReadStore/AndOrWriteStore?

  • Vili

    Kris,

    I find ClientFilter extremly useful. However, I run in the following issue using JsonQueryRestStore:

    1. do an initial fetch with the empty string query (“”)
    2. add new item to the store and save it
    3. fetch again: this is coming from the cache as expected and the new item is there
    4. add another item to the store and save it
    5. fetch again: coming from the cache as expected, however, item added in #3 is not returned while item added in #4 is around.

    Did some debugging and turned out that the serverVersion is always incremented that much that only the most recent changes are considered by updateResultSet. I would have expected the cached results to be extended per store save but this is not the case. That way we could ignore “old” changes from _updates when the store is no dirty.

    The scenario above is pretty deterministic. The store “forgets” everything between last server fetch and the very last added item in case the changes were saved.

    When I add items and do not save the behavior is what we expect: all new dirty items are returned.

    Thanks

  • @Vili: Thanks for letting me, just checked in a fix.

  • …for letting me know…

  • David Starke

    I just pulled the Dojo source specifically to get the fix to the issue Vili reported above, and it still doesn’t work correctly.

    It looks like you are testing the version of the cached results for null using “||” so you can default to the current server version. Unfortunately, the incoming version (when I debug it) is always 0, so it gets overwritten with the server version, resulting in the same behavior as before.

    Also it doesn’t look like it records either the updated results or version in the cache.

    Thanks

  • @David: Arg, sorry about that. Should be fixed correctly now.

  • Michael

    What is the right way to refresh an already loaded query from the server?

    for example,

    I have ClientFilter and JsonQueryRestStore. I load a dataset by calling Fetch with no arguments. The set loads.

    Some time later I want to check the server to see if a server-side change has added something to the set. I would like to fire appropriate notifications only if there are changes. What is the right way to do that?

    It seems that if I call Fetch again it does not query the server, even with queryOptions: {cache:false}

    It occurs to me that I could poll the server using a homemade function and then call myStore.newItem to add the new server-side items to the cache, but that would cause the Store to attempt to post these new items to the server thinking they were newly added client-side, wouldn’t it?

    thanks

  • @Michael: In trunk/1.4 you can call:
    store.clearCache()
    In 1.3 you will have to do:
    store._fetchCache = [];

  • Michael

    Thanks for the response, that has been working out very well.

    I have two stores that share a many-to-many relationship. When I make one of my many-to-many connections is the right thing to do to call setValue on both stores?

    for example, let ‘foo’ and ‘bar’ be dojo.data items of type Foo and Bar. I want to make a connection between them. Does this look right?

    var new_bars = foo.bars.concat([bar])
    var new_foos = bar.foos.concat([foo])

    fooStore.setValue(foo, ‘bars’, new_bars)
    barStore.setValue(bar, ‘foos’, new_foos)

    fooStore.save()
    barStore.save()

    considering that these stores will both call a ‘PUT’, which is idempotent, this should work, but I’m wondering if this is a best practice. Thanks!

  • @Michael: You certainly can create relationships in this manner, although it is prone to difficulties in maintaining the synchronization of the bi-directional references. If foos are always accessed from bars, or vice versa, you might only need to define references in one direction. But if you need bi-directional references, your approach looks correct.

  • vlad

    It is not quite clear if it is possible to perform
    ‘client-side-only’ queries using the mechanisms described above.

    For example using Perserver rest store (or JsonRestStore)

    mystore.fetch(query:{myid:’*’},queryOptions:{cache:true})

    is supposed first look in the cache for the
    same (or subset) query, and start calling onItem
    if it could not find anything — it will go
    to the server and execute the query there.

    Whatever the query returns will be ‘cached’ because
    cache:true is specified.

    That is cache:true is NOT for looking up the data
    in the memory, it is to tell the store to cache
    the resulting result

    So the above query will go to the server at least if nothing was found in cache, is the assumption correct?

    On the other hand, if I want to *only* search
    the store in memory

    should I use

    mystore.fetch(clientFetch:{myid:’*’},queryOptions:{cache:true}) ?

    and will this case guarantee that no server side
    requests will be issues? (this is not the case for me now)

    And if not, is there a way to make the store
    only search in memory.

    And finally, when searching in memory, can I ask
    the store to search for comitted as well as
    uncomitted items ? what’s the mechanism?

    thank you

  • @vlad: The decision to determine whether to query the client or server is made by ClientFilter calling isUpdateable and querySuperSet on the store. If isUpdateable returns true querySuperSet returns an object (the filter object to apply to the cached query result set), then it will utilize the client side cache. You can certainly override these functions to customize the decision about whether to go to the server or not.

    ClientFilter will always include uncommitted items in the query results (that is it will update the result sets based on the current set of dirty items).

  • Tod

    I’m really interested in making the JSONQuery capabilies work with an ItemFileWriteStore. It is probably pretty easy but I’m a bit of a hacker.

    The sample code from above:
    dojo.declare(“JsonQueryEnabledSomeStore”, [SomeStore, dojox.data.util.JsonQuery]);
    … is a start.

    I’ve got dojox.json.query working on a JSON dataset from the server, and then create an ItemFileWriteStore from the resulting data and display it in a grid. But I need to modify (create/upate/delete) items in the store and then change the query parameters and have the grid update correctly. Running the JSONQuery on the store itself would be ideal.

    What would the code look like to run the query on the ItemFileWriteStore?

  • I don’t think you can run JSONQuery on ItemFileWriteStore because it does not store the data in the same format as JSON document that it accepts. The format is hard to understand, running JSONQuery on it would not work as expected.

  • Iram

    Hi Kris,

    Is it possible to publish an example which uses the JsonQueryRestStore with dojox.grid.DataGrid?
    What is especially relevant for me is the client-side filtering and sorting.
    Thanks a lot,
    Iram

  • Sri

    Hi Kris,

    Could you please post us an example for implementing lazyloading in Json rest store using dojox data grid. For eg. Lets assume that i need to hit the db and get values from a table. How do i cache the data and show it dynamically while the table is being scrolled?

    Thanks in advance.