[Note: This blog post is out of date. For up to date information on Dojo Offline please see the official web page.]
The last few weeks we’ve been putting together our API for the Dojo Offline Toolkit (DOT). How will a programmer use this toolkit in their work? How will it be integrated into their applications?
Last week we reported on addressing usability for offline access, with offline mockups of popular web apps. Usability is just as important for programmers as it is for end-users, it just takes a different form: the API, or Application Programmer Interface. Getting the API right is just as important as the UI; programmers need the love too.
Look Mah, No Proxy!
Before we dive down into the API, I want to share a nice surprise: the Dojo Offline Toolkit API has been designed to not necessarily need a web proxy. For example, if your browser has native support for offline access, then we don’t need to download the small web proxy — the browser will simply cache these offline resources. The plan for Firefox 3 is to natively support such an API — in this scenario, Dojo Offline could simply use the browser’s offline cache rather than requiring you to download the web proxy.
Here’s an even nicer thing: as soon as we finish the JavaScript API you see in this blog post, you can start using Dojo Offline even without the full, local web proxy being done. How is this possible? Well, if you use the trick discovered by Julien Couvreur and set your web server headers to correctly send HTTP/1.1 caching info (like my Moxie application does), then the browser cache will store your files for offline use. This is appropriate for prototyping and getting beginning applications out there; the web proxy is an optimization meant to improve the reliability of offline cached files. Without the web proxy browser cache misses are possible, which would translate into UI resources not being available when offline, but this might be good enough for prototyping and for some applications (in fact, Scrybe works this way if you look carefully at their screencast).
And here’s the really nice thing: when the full web proxy becomes available, your application will require very little change (in fact, one change, as you will see below) to gain the improved UI reliability the web proxy offers.
Let’s quickly see how DOT achieves this generality. DOT refers to the ability to truly store web files offline as a durable cache, rather than a web proxy; a durable cache is a more generic term, since it could be backed in any number of ways: by the web browser with native support or by the DOT local proxy. By default, DOT requires a durable cache; this can be changed with offline.requireDurableCache = false. During startup, the default UI queries if a durable cache is required; if so, we see if one is available; if it is not, we prompt the user to install one. When a durable cache becomes available (such as the DOT local proxy), the toolkit can simply prompt the user to download it.
KISS – Keep It Simple
Let’s start with the simplest example of programming against Dojo Offline. The philosophy behind the Dojo Offline API is to have sensible defaults for everything (thanks DHH!), while providing framework configuration hooks that can be overridden if you want to take things over and go past the defaults.
Our example application will be a kind of web-based Outlook, with emails and tasks. Users can read and create emails offline, and also update their task list when away from the network. Everything below will be pseudo-code to illustrate how Dojo Offline can be used. Please keep in mind that this is just the planned API and is not available yet — this is all design work in preparation for coding this week; also note that the API is sure to change both during development as well as after the first version is released. Expect it to evolve in mid-air as we slam into reality.
Let’s look at the full pseudo-source code for our example application, which we will break down and explain in pieces. Don’t worry — all of this will be explained in detail below.
Our HTML file will look something like the following:
Parts of my application's UI here
More of my application's UI here
While our JavaScript file will look something like this:
// bring in Dojo Offline dojo.require("offline.*");
// indicate what files we want offline, such
// as our application's images, HTML, CSS,
// JavaScript, etc.
offline.files.put(["/images/toolbar.gif","index.html","/css/global.css","dojo.js"]);
// define our application's name; this will be used to
// customize the default Offline Toolkit UI offline.ui.appName = "Web Outlook";
// define what elements we would like disabled when we go offline;
// these will be re-enabled when we go online
offline.ui.disableElementsOffline(["configurationLink","searchField"]);
var emails, contacts, tasks;
// called when the page is finished loading and the offline toolkit
// is ready to be used offline.onLoad = function(){
// define where to put our offline user data; when
// we sync with the server, each kind of data will have
// an itemType, such as "emails", "tasks", etc. These
// data structures will automatically be kept in sync
// so we can use them in our application, and will
// automatically be persisted in local client-side
// storage
emails = offline.getDataStore("email");
contacts = offline.getDataStore("contact");
tasks = offline.getDataStore("task"); }
// default implementation of offline.onOffline and offline.onOnline
// will simply use the disableElementsOffline values
// to enable and disable these elements, and do an automatic
// synchronization inside of onOnline
function addContact(contact){ contacts.newItem(contact); }
function displayEmails(){
var displayMe = null;
if(offline.isOnline() == false){
displayMe = emails.find().items;
}else{
displayMe =
// ...remotely fetch emails
}
// do something with array of emails
}
function deleteTask(task){ tasks.deleteItem(task); }
function sendEmail(email){
if(offline.isOnline == true){
// send this email to the server
}else{
// else we are offline; just queue this email up
// store it's value in our offline cache
emails.newItem(email);
// now create a custom sync log entry for a 'send' command
var c = new offline.sync.Command();
c.name = "send";
c.itemType = "email";
c.item = email;
offline.sync.log.add(c);
// later, when we sync, this command will be replayed to the
// server for the email to be sent
}
}
User Interface
Our first task is to integrate an offline UI into our application. Creating a good, usable offline UI that gives the user effective feedback is difficult, so DOT will come bundled with the Offline Widget shown in the Dojo Offline mockups last week. This is simply a bundle of HTML and CSS, inside of offline.html and offline.css. You can either simply cut and paste this code into the HTML of your application, or can just drop a script tag inline in your HTML, which will cause the Offline Widget to simply appear in your page as a JavaScript Include:
Parts of my application's UI here
More of my application's UI here
Now, when the page loads, the offline widget will simply appear there, ready to be used in your application. This widget comes bundled with the logic necessary to also interact with the user, without needing customization by the developer.
If you don't want the UI and simply want to take things over yourself, you can set a simple flag:
offline.ui.enableUI = false;
Now you are free to create your own custom UI. If you like the UI functionality but don't like it's look or layout, you can break it's HTML apart and spread it across your application; the offline.ui infrastructure simply looks for the correct HTML ID names for elements that are inside the Offline Widget HTML; if they are not present it will not fail, it will simply not use those UI components. Further, you can style things as you wish in your own CSS.
As you will see throughout the API design that follows, the philosophy is to make the DOT framework almost ready to run out of the box inside your apps, but powerful and configurable enough for future growth.
Offline Files
Let's jump into some actual pseudocode for our example DOT-enabled application.
Our first source code task is to define what UI resources we need offline, such as our JavaScript, CSS, HTML, etc:
// bring in Dojo Offline
dojo.require("offline.*");
// indicate what files we want offline, such
// as our application's images, HTML, CSS,
// JavaScript, etc.
offline.files.put(["/images/toolbar.gif","index.html","/css/global.css","dojo.js"]);
On the first line we pull in the Dojo Offline libraries; all of the DOT libraries live under offline, such as offline.files, offline.sync, offline.ui, etc.
Second, we pass an array of the resources we want to have available offline; note that this is different than user data. These are the files that make up our user interface, such as HTML, JavaScript, etc. We pass an array of paths, relative to the current location, to offlines.files.put.
Application Name
We also define our application's name if using the default bundled Dojo Offline UI:
// define our application's name; this will be used to
// customize the default Dojo Offline Toolkit UI offline.ui.appName = "Web Outlook";
The default UI will plug this in in a number of places; for example, in the Dojo Offline mockups you may have seen text such as "Learn More About How to Use 'AppName' Offline" for example; the word 'AppName' would simply be replaced by your application's name when displayed.
On and Offline Events
DOT provides two standard event listeners that tell us when the user has gone online or offline, called onOnline and onOffline. These simply take functions that are called when this occurs:
offline.onOnline = function(){
alert("We're online!");
}
offline.onOffline = function(){
alert("We're offline!");
}
Here's the great thing: DOT provides standard implementations of both of these methods -- you don't have to override them yourself. When a user goes offline, onOffline will disable elements on the screen that are not available when offline, such as search; when the user goes back online, onOnline will renable these elements and do an automatic synchronization with the server to upload any locally changed data and download any new ones for the local data cache.
Most applications will follow this pattern, and don't need to override these methods. It makes sense that when you go offline, many apps will disable certain UI elements that can't work offline, such as a search box or things that need the network like a scrolling stock ticker. It also makes sense that when you go back online to re-enable these elements and to synchronize data automatically. Again, as you will see, you can easily configure your apps to have different behavior, but you don't need to care if it doesn't matter to you.
Disabling Elements Offline
How do you indicate which elements to automatically disable when the user is offline? Simply use the offline.ui.disableElementsOffline method, which you pass an array of ID names:
// define what elements we would like disabled when we go offline;
// these will be re-enabled when we go online
offline.ui.disableElementsOffline( ["configurationLink", "searchField" ]);
In this case we pass two IDs, configurationLink and searchField, which we want to be disabled when we lose the network. DOT is smart enough to know how to disable different kinds of elements, so it will correctly disable form elements, links, buttons, etc. When an element is disabled, it will also gain the CSS class offline-disabled so you can style it to your hearts content.
offline.onLoad
How do you know when the offline toolkit is ready to go and be used? Simply use the offline.onLoad method; this is used instead of the page's standard window.onLoad method, since the underlying Dojo Storage data cache might not be ready exactly when the standard page's onLoad is:
offline.onLoad = function(){ alert("Dojo Offline is ready to be used!"); }
offline.getDataStore
So how do we actually get data that we can play with, the actual stuff that our application uses to get its task done, such as emails, tasks, etc? Dojo Offline splits the data it deals with into different item types, such as emails, tasks, etc. As you will see below when we cover synchronization, when the client and server communicate to sync, syncing is broken across the different kinds of item types for an application; this makes it easy for us to back these into simple to use data objects for the application programmer, automatically persisting these to disk:
var emails, contacts, tasks;
// called when the page is finished loading and the offline toolkit
// is ready to be used
offline.onLoad = function(){
// define where to put our offline user data; when
// we sync with the server, each kind of data will have
// an itemType, such as "emails", "tasks", etc. These
// data structures will automatically be kept in sync
// so we can use them in our application, and will
// automatically be persisted in local client-side
// storage
emails = offline.getDataStore("email");
contacts = offline.getDataStore("contact");
tasks = offline.getDataStore("task");
}
Once onLoad is called, we now have three nice data structures we can use in our application: emails, contacts, and tasks.
Application Logic and offline.DataStore
At this point you have seen all of the boiler plate for Dojo Offline. Here is it so far; it's very small for what you are getting:
// bring in Dojo Offline
dojo.require("offline.*");
// indicate what files we want offline, such
// as our application's images, HTML, CSS,
// JavaScript, etc.
offline.files.put(["/images/toolbar.gif","index.html","/css/global.css","dojo.js"]);
// define our application's name; this will be used to
// customize the default Offline Toolkit UI
offline.ui.appName = "Web Outlook";
// define what elements we would like disabled when we go offline;
// these will be re-enabled when we go online
offline.ui.disableElementsOffline(["configurationLink","searchField"]);
var emails, contacts, tasks;
// called when the page is finished loading and the offline toolkit
// is ready to be used
offline.onLoad = function(){
// define where to put our offline user data; when
// we sync with the server, each kind of data will have
// an itemType, such as "emails", "tasks", etc. These
// data structures will automatically be kept in sync
// so we can use them in our application, and will
// automatically be persisted in local client-side
// storage
emails = offline.getDataStore("email");
contacts = offline.getDataStore("contact");
tasks = offline.getDataStore("task");
}
// default implementation of offline.onOffline and
// offline.onOnline will simply use the
// disableElementsOffline values to enable and
// disable these elements, and do an automatic
// synchronization inside of onOnline
Okay, so how do we actually start getting work done? Let's look at some example application-level methods for our example application:
function addContact(contact){ contacts.newItem(contact); }
function deleteTask(task){ tasks.deleteItem(task); }
function displayEmails(){
var displayMe = null;
if(offline.isOnline() == false){
displayMe = emails.find().items;
}else{
displayMe = // ...remotely fetch emails
}
// do something with array of emails
}
These are pseudocode methods for adding a contact (addContact); deleting a task (deleteTask); and displaying emails (displayEmails). You can imagine fleshing them out with greater UI logic and application control.
These three methods all use our three magic data structures we initialized in onLoad: contacts, tasks, and emails, which are instances of offline.DataStore.
offline.DataStore variables do a lot of the hard work of syncing and persistence for you, so you don't have to manually craft your own solution. They automatically maintain their internal state when syncing with a server, so you don't have to craft a difficult custom solution, and also persist themselves into the local data cache and load themselves on page load. They make your life easy. You can crack them open and break the abstraction if you want, but the important thing is you don't need to in the most common case and during prototyping. They will automatically contain whatever local data you have created locally, as well as remote data that was synced from the server. You'll see the magic they do inside later, but for now lets see how you use them externally.
For each of your application item types that you want to have available offline, such as emails, contacts, etc., whenever you create an item, delete an item, retrieve an item, or update an item, you should do it through these data structures:
// adding a contact
var contact = {first: "Brad", last: "Neuberg", email: "foobar@foobar.com", phone: "510-555-1212"};
contacts.newItem(contact);
// deleting a contact contacts.deleteItem(contact);
We see these two methods being used in our example application methods:
function addContact(contact){ contacts.newItem(contact); }
function deleteTask(task){ tasks.deleteItem(task); }
In our displayEmails method, we get a bit more sophisticated. First, we see if we are online:
if(offline.isOnline() == false){
This call will see if the user has manually gone on or off-line. If we are offline, then we get our full list of emails to display:
function displayEmails(){
var displayMe = null;
if(offline.isOnline() == false){
displayMe = emails.find().items;
To get our full list, we first call find(), which is a placeholder method for now. It can take a query string, to query over its data set. Right now querying is a no-op, and it simply returns an array of all of this data store's items, exposed through items. This method is an extension point for a future world where more sophisticated querying or searching is supported.
If we are online, then our code simply toggles and does would it would normally do online, which might be to fetch our full list of emails remotely:
}else{
displayMe = // ...remotely fetch emails
}
Finally, now that displayMe has our list of emails, whether we are on- or offline, we can work with them:
// do something with array of emails
Here's the full displayEmails method:
function displayEmails(){
var displayMe = null;
if(offline.isOnline() == false){
displayMe = emails.find().items;
}else{
displayMe = // ...remotely fetch emails
}
// do something with array of emails
}
What's going on inside these magic offline.DataStore instances? Before I can crack these open and show you the juju, let's address synchronization.
Synchronization
Up to now you may have noticed that we haven't really mentioned synchronization much yet. There isn't even a call to a synchronize method of some kind in the pseudo-code above. What's going on?
By default, Dojo Offline will auto-synchronize when the page is first loaded and when we go back online after being offline (onOnline); as a programmer you don't have to hook anything up for this to simply happen when you drop DOT into your environment. This happens automatically unless you have turned it off with offline.autoSync = false. Here's some example pseudo-code of what the default implementation of offline.onOnline might look like as bundled with DOT:
offline.onOnline = function(){
// enable disabled elements
offline.ui.enableOfflineElements();
// do an autosynchronize
if(offline.autoSync == true && offline.enabled == true){
offline.sync.synchronize();
}
}
Like everything else, offline.sync.synchronize() has a default implementation. What does it do? It does three things: it refreshes any UI resources we want available offline, which we defined by our earlier call to dojo.files.put; it uploads any local data changes that were made when we were away from the network; and it downloads any new data changes necessary from the server. Here is some extremely pseudo-code possibilities of what this method might look like internally to aid understanding; note that the actual code will be asychronous, we just show it synchronously here to highlight the default sync process:
offline.sync.synchronize(){
this.onStart();
// refresh our UI resources
this.onRefreshUI();
offline.files.refresh();
// send our client log to the server
this.onUpload();
this._sendLog();
// get our server log from the server
this.onDownload();
this._retrieveLog();
this.log.replay();
this.onFinished();
}
A few things to point out. First, you will see a number of event hooks you can override for your application to receive progress on the status of synchronization: offline.sync.onStart, offline.sync.onRefreshUI, offline.sync.onUpload, offline.sync.onDownload, and offline.sync.onFinished. If you are using the default DOT UI, offline.ui overrides these methods so that it can provide friendly UI feedback to the user on the state of synchronization.
Syncing does three major things: it refreshes our resources files (HTML, JavaScript, etc.); it uploads local changes to the server; and it downloads new data from the server.
Refreshing our UI is as simple as calling offline.files.refresh(), which offline.sync.synchronize() does above. Internally, this will cause DOT to simply iterate over the array of offline files we passed into offline.files.put, and make an XHR call on each one; this will cause the browser, and therefore any offline web proxy that might be available, to follow HTTP/1.1 semantics in refreshing any files, or adding new ones, that might be needed offline.
Uploading and downloading data is a bit more sophisticated; lets dive down to the wire to see what is actually going on during synchronization.
Are You Wearing a Wire?
Synchronization can be hard; entire startups and companies have found themselves dragged into the deep rabbit hole of synchronization, never to be heard from again as they struggle to put a 1.0 out and understand the SyncML spec. Dojo Offline's goal is to stay nimble, identifying the most common and most useful kind of synchronization.
While syncing can be hard, providing a powerful, default synchronization is a major service that Dojo Offline can provide, since doing it right is hard on many levels, and if we do the hard thinking and the hard implementation many app developers lives will be simplified.
The key golden thread making sync a tractable problem was solved by Alex Russell. Alex suggested that we use a transaction oriented API for communication between the client and server. Whenever a client-side action happened while offline, we would simply add an entry to our log about this action. When the network reappeared, we would push this log to the server, which would replay it, and respond with its own log. The client would then replay this server log, which would update our local data cache.
This is exactly what Dojo Offline will do, though the application developer will be shielded from the raw application log unless they need custom access. Now we can understand the offline.DataStore instances we saw earlier from the inside, such as tasks and emails. Whenever we were calling things like tasks.newItem(someTask), internally the DOT framework would simply be adding another entry to our offline log, represented as instances of the simple JavaScript object offline.Command:
tasks.newItem = function(item){
// represent this action as an offline.sync.Command object
var c = new offline.sync.Command();
c.name = "update";
c.itemType = "tasks";
c.item = item;
// add it to our offline log
offline.sync.log.add(c);
}
The programmer doesn't have to know any of this, but it is internally overridable for unusual synchronization scenarios.
When we go back online, we can simply call offline.sync.log.toString(), which returns a nice JSON string that we can send to the server that has all of of our offline.sync.Command's. This is POSTed to the server, at the default address /sync, and the server responds with its own log. The entire exchange is a single request-response; authentication happens outside of this, as standard cookies like other web services.
Let's look at an example client-side command log ready to send to the server. Here's what is POSTed to the server (view client sync request in new window here).
Don't let all the curly braces scary you; this is standard simple standard JavaScript (JSON). The client simply tells the server what time it thinks it is, 2007-01-15 12:22:12Z; when the server responds, it will also tell the client what time it thinks it is, which the client will use for future communication to prevent time skew between the client and server.
Following this, we simply see our log as an array of commands. Each command has a name, which is the kind of action that was stored; a status, which is optional and not present above, but which can carry extra information as you will see when the server responds; an itemType, such as email, task, contact, etc.; and an item entry. Every item entry is it's own serialized JavaScript object; these are the application's specific kind of data objects, which can have any kinds of fields. Only two attributes are required of every item object: an id, which is a unique ID for this item; and a timestamp, which is the time in milliseconds from the top-level timestamp when this action occurred.
Most actions in data-oriented things can be boiled down to what is known as CRUD: Create, Retrieve, Update, and Delete. For Dojo Offline, we boil this down to just three: Create, Update, and Delete. A server can take these three kinds of actions, held in the name field, and simply replay them to update it's own view of the data. The server will use the id element on an item to know what to update or delete, and use the timestamp element to do automatic conflict resolution and merging; if a client action happened before something that happened on the server, for example, it can choose to have the client action be the winner.
A quick note on the id value. This must be a unique ID across each set of itemTypes. For example, if I have an email with the ID 33, there can't be any other email with this ID, but there could be a task with it. This ID could be the ID of an Email row in a relational database table, for example. When a new item is created while offline, we have no way to create a unique ID, so we simply give new items negative IDs, starting at -1 and counting downwards (-2, -3, -4, etc.). This ID is unique enough for us, so we can later reference it in our client log (for example, we might create a new contact while offline, then update it while offline -- we must be able to reference it in our command log).
You might see one strange action name above, send. This is a custom application action, which I will touch on in a bit.
Let's see what the server responds with in this example (view server sync response in new window here).
The first thing the server does is give it's time, just like the client. Next, we see our log commands, again just like the client. Our first set of log commands give us feedback on earlier events, such as created, deleted, and updated. The status field also provides any automatic conflict resolution that may have occurred, such as status: "conflict, client won".
Next we see a bunch of commands for updating our local cache; first, we see a large number of deletes. This is basically the server clearing out any old emails in our cache, to make room for a large set of new ones. The deletes are therefore followed by a large number of creates. When the client replays this log (offline.sync.log.replay()), after getting it from the server, it will simply execute these commands locally, deleting, updating, or creating whatever new information was sent down from the server.
That's it for the wire protocol. For your app, all you will need to do is have your server output a list of commands in this simple format; the client will automatically speak the correct format to the server. If you don't like the default syncing implementation, it can be configured on a number of levels. For the simplest, you can change the sync URL in offline.sync.syncURL. For more sophisticated custom syncing you can simply over-ride offline.sync.synchronize() yourself and throw everything away, doing your own custom syncing. The most common scenario for different kinds of syncing is to have different network endpoints for different kinds of items that speak a custom sync protocol (i.e. there might be an XML service you already have for emails, a JSON one for tasks, etc.). To aid this, offline.sync.log.slice() can be called with an itemType, such as offline.sync.log.slice("email"), which would return a subset of the command log with just email actions. You could iterate over this to build up the actions and data necessary to talk to your custom service, followed by actually invoking the service.
A few final notes. DOT requires an important simplification for our first release: the server must do automatic, defacto merges and conflicts, without user input. No UI is provided for this initial phase to allow the user to see diffs of two different versions, for example. Instead, the server must make these choices automatically, providing messages that DOT can use to show users a sync log if they are interested.
Finally, the default action types might not be sufficient for your application. For example, in the client response above, we have a name: "send" for an email; rather than being a CRUD style action, this is a verb. On the client side, you can create a custom action and place it into the log manually:
function sendEmail(email){
if(offline.isOnline == true){
// send this email to the server
}else{
// else we are offline; just queue this email up
// store it's value in our offline cache
emails.newItem(email);
// now create a custom sync log entry for a 'send' command
var c = new offline.sync.Command();
c.name = "send";
c.itemType = "email";
c.item = email;
offline.sync.log.add(c);
// later, when we sync, this command will be replayed to the
// server for the email to be sent
}
}
The server can now handle this action in whatever way is appropriate. On the client side, if a custom server action is sent in the command log, DOT provides a standard hook to see these when they are invoked: offline.sync.onCustomCommand, which will be passed the custom offline.sync.Command object. The default implementation of this method is empty; fill it out to override for your own custom actions.
Danger, Will Robinson!
It's useful to point out one other framework hook that is useful to application developers. Remember that Dojo Offline lives in a data storage environment where data saves might be denied by the user or the underlying data storage infrastructure. You might be trying to save information that is simply too large, such as a huge document, or the user might have completely turned off data storage for your domain.
DOT provides a function you can override to know if one of your offline.DataStore requests failed to save, such as persisting some emails, or even more seriously, if some data core to Dojo Offline itself fails to save. This event function is named offline.onFailedSave. The default implementation of this method for data is to give an alert box informing the user that saving failed, informing the user they should increase the total size allowed by this application. The storage settings for the Dojo Storage provider in this user's browser is then shown.
If core data necessary for Dojo Offline itself fails to save, such as the list of files inside our web cache, then DOT's philosophy is to 'fail fast' rather than enter an inconsistent state. If any core data fails to save, the flag offline.coreSaveFailed is set to true, and offline access is disabled with offline.enabled = false. If the default DOT UI is being used, the user is informed that they must increase the size of available storage for offline access to work.
Odds And Ends
This is a space for important little bits that should be captured here, but might not be generally interesting to everyone.
Class diagrams in Omnigraffle format are available here.
Earlier, we saw some calls to offline.isOnline():
if(offline.isOnline() == false){
This call will see if the user has manually gone on or off-line; programmatically, this can be initiated by calling offline.goOnline() and offline.goOffline(). Dojo Offline will not automatically provide automatic network on- or off-line notification, since this can get difficult quickly. Instead, you can manually check to see if your site is available to the browser by calling offline.isSiteAvailable(callback); this method will cause the browser to attempt to contact your web site by doing a network call. Based on it's response, we set offline.isOnline() and call callback asychronously.
By default, DOT will simply use your site's root URL as the 'availability URL'. For example, if your web application is served from http://foo.com/index.html, then offline.isSiteAvailable will call http://foo.com in the background. This might not work for your site, so this URL can be configured by calling offline.siteAvailableURL:
offline.siteAvailableURL = "/site-available";
For the Dojo geeks out there, here's an interesting fact: our offline.DataStore is actually a Dojo Data implementation, implementing the dojo.data.core.Read, dojo.data.core.Write, and dojo.data.core.Identity interfaces. See the Dojo Data page for more info on how these APIs can be used.
For Dojo Data's dojo.data.core.Write interface, I'd like to add the method updateItem, which can pass a full updated item into the Data Store, versus having to call a bunch of dojo.data.core.Write.setValue(), set(), or unsetAttributes() methods; I might not support those three methods, since they might not play well with our sync log. Would love comments from Bryan on this.
Here are some important facts about offline.onLoad. If we are using the bundled UI with Dojo Offline, which is on by default (offline.ui.enableUI = true), then the Offline UI Widget will automatically update itself at this point. Further, by default, Dojo Offline does an automatic synchronization on page load, so that a user does not have to initiate this. This can be turned off by setting offline.autoSync = false. This means that the user must manually initiate all synchronization, or the application developer must do synchronization themselves.
What's Next?
The next step is to start coding the API! Expect to see coding kick off this week, with all code done in the public through Subversion and frequent updates posted on this blog and the SitePen Labs blog. If you have comments about specific sections of this post and API, please use the HyperScope version so I can have direct links to the paragraphs in question. Please note that this API might be reduced in scope; the final deliverable might not have the full API based on schedule pressure.
Shout Outs
A huge number of folks helped out with this API -- if I forgot someone send me an angry email and I'll add you ;)
- Alex Russell
- Julien Couvreur
- Owen Williams
- Dustin Machi
- Brendan Eich
- Dave Camp
Brad, nice API. I used to push on the edge of the Single-Page-Application model for offline access, until other priorities and projects popped up. I started a task tracking app called Next Action: http://trimpath.com/project/wiki/NextAction that I hope to one day upgrade to Dojo Offline Toolkit.
Like you, I used the negative-number trick to handle records created while offline. It’s ok, but doesn’t fully solve the conflict/merge hair-pulling.
One API thought: there could end up being lots of dual code paths, with much “if (offline.isOnline())-then-else” testing. How about something like (handwave)…
emails = offline.getDataStore(“email”,
{ ajaxy-CRUD-request-info-and-handlers-when-online });
function displayEmails() {
var displayMe = emails.find().items;
// do something with array of emails
}
That might help to reduce the online/offline if-then-else across the codebase?
Brad, great stuff.
I noticed your code lives in a top-level package. I might have missed it, but I gather you’re trying to keep your APIs (and perhaps your implementation?) separate from the Dojo Toolkit. Is that a stated design goal? Being able to use it against other toolkits makes a lot of sense. I wonder, though, about the namespace issues. Are you a top-level namespace because you plan to have this code in a top-level Foundation project? It’s a nit-pick, but is there a convention for scoping top-level names which end up in the global namespace?
Steve: Good to hear that the negative number trick has been used before. In our case when the client has sent a new item over to the server, the server creates that item; in the response log, the server first directs the client to delete the old negative item, then create a new entry that has the new correct ID (you will notice that this ‘create’ request has a small, new field in the item: ‘origID’).
About making more convenience methods like you suggest; these are good, but at this point I think I want to just get the API out there and see what sticks against the wall. We can imagine all sorts of cool ways to make this kind of coding more transparent, but these can come in later iterations.
Adam: I originally adopted ‘offline.*’ because my paths were getting too long, such as “dojo.offline.file.put”. I might actually just go back to “dojo.offline” when I start coding tomorrow.
Oh Steve, just realized you did NextAction; thats a great app! I used that for a long time as my task tracker.
As an experiment, are you interested in working with me to see what it would look like to hook a server-side component + Dojo Offline into NextAction? As I roll the API out in the comming weeks, it would be useful to see how it interacts with real apps.
Adam: We both went to the same college, though I was a few years after you. Do you still live in NYC?
Sure, would love to collaborate!
btw, I’ve been recently looking at the new REST API’s in ruby on rails 1.2 on a current project, so, a warning that that stuff might infect my thinking on how a server conversation might look. Laziness reigns.
More feedback on sync…
- With the sync message handshaking, I worry whether the server would need to remember a lot of unique offline state per client. (“I, the server, last sent Alice’s client this set of snapshot information. And, I also last sent to Bob this other snapshot.” etc) The server, now thinking it knows what Alice and Bob know, can now correctly generate sync reply messages — until something gets messed up. One idea to help us create optionally simpler and more braindead servers (with admittedly less efficient messaging): in the synch reply message from the server, perhaps have some way for the server to tell the client to instruct ‘wipe out everything you know because here’s a brand new complete set of data’.
- On negative id numbers and joined information… Take, for example, a sales-force CRM app. A field sales rep is visiting new prospects and is offline. He enters in some new order quotes, and also creates some new customer records, too. Assume that the new price quote records are ‘linked’ or have some foreign-key like id ‘pointers’ to navigate to the new customer records. During the sync, you not only have to fix up the primary negative number ids of the records, but also those negative linked foreign-keys ids. One solution idea: make the negative numbers space unique across the entire offline system. So instead of id’s like: new_order_ids = [-1, -2] and new_customer_ids = [-1, -2]. Make it look like: new_order_ids = [-1, -2]; new_customer_ids = [-3, -4]. In Next Action, we use a rails-like naming scheme where any ‘field’ that ends with an ‘_id’ suffix gets examined during sync if it has a negative value. So, order[-1]['customer_id'] would get corrected during a sync.
- Finally, an example where negative numbers aren’t enough: in the same sales-force CRM app, two field sales reps sync up to get their morning snapshot of inventory data (say quantity 10 widgets) and each hit the road. Separately, they take orders for the widgets for different customers, say quantity 8 and 6. When they sync later, someone loses. You’d definitely would want some UI to let the loser know. And, you can’t just delete the losing order. Easiest solution: punt — this problem is left as an exercise to each app developer, ha ha. :)
[...] Dojo won me over with the slick interfaces, abundance of functionality, documentation for integrating with symfony, strong contributors and community, and potential of Dojo Offline Storage. Symfony brings … [...]
Brad,
After looking at your sample client sync request, I thought that it might be useful for you guys to keep in mind that PHP (for example, I’m not sure about other server side stuff) has a JSON decoder, but it expects keys in quotes. Read more (particularly in developer comments) here: http://us2.php.net/manual/en/function.json-decode.php .
I’m really looking forward to using this!
-Matthew Foley
Matthew: Thanks for telling me that; I did not know that.
I have found a way to significantly simplify the syncing model, resulting in much less server work and no actual custom sync protocol using JSON. More tomorrow (Monday).
Ugh:
if(offline.isOnline() == false){
if(offline.isOnline == true){ // looks like there is a typo in this line
How about a more pragmatic:
if(!offline.isOnline()){
if(offline.isOnline()){
[...] To start using Dojo Offline now in conjunction with the browser cache, you must have the following in your JavaScript code: dojo.off.requireOfflineCache = false. You must also turn on HTTP caching headers on your web server; how to turn on these HTTP headers and which ones to turn on are explained in the Dojo Storage FAQ. See the Moxie code links below for more examples of how to use the API. Note that the Dojo Offline JavaScript API has changed, especially for syncing, since our API blog post about a month ago and has become much simpler — see the Moxie source for details. The demo of Moxie shown in the screencast above can also be played with right in your browser. Please note that both Moxie and Dojo Offline are still alpha; you are literally seeing Dojo Offline being developed in front of your eyes, and glitches remain in both of them. Please debug and provide test cases for errors you find to help development. [...]
Simon: Hi; the API has changed quite a bit since this blog post, becoming much simpler and straightforward. It is now dojo.off.isOnline, so you could have if(dojo.off.isOnline == true){}
[...] There has been many ideas about a offline storage mechanism for web applications. Dojo implemented this in its Library. Firefox 3 promises this. Now we have a new arrival in this area – Google Gears. Unlike Dojo’s implementation, Gears require an extension for it to work. Google Gears is an open source browser extension that lets developers create web applications that can run offline. [...]
[...] headers within HTTP to store objects within the browsers cache. Javascript libraries such as Dojo implemented support for offline web applications using the same principals, but applications were very limited in scope as there was no easy way to [...]
[...] headers within HTTP to store objects within the browsers cache. Javascript libraries such as Dojo implemented support for offline web applications using the same principals, but applications were very limited in scope as there was no easy way to [...]
I wondered why there isn’t a off line API for dojo??