Effortless Offline with OfflineRest

By on September 23, 2008 12:01 am

offlinerest.png There is growing support in browsers for offline capabilities with the HTML 5 specification for local storage, offline notifications, and offline application cache, but adapting an application to store changes locally and do synchronization when connectivity is restored remains a major challenge for developers. Dojo 1.2’s new dojox.rpc.OfflineRest module automates the local storage of data and synchronization by leveraging the Dojo Data and REST abstractions. The OfflineRest module augments the JsonRest service in Dojo such that requests are cached in local storage for offline access, and modification requests (put, post, and delete) modify the cache and are recorded for delivery to the server; immediately if online, otherwise when connectivity is restored. Furthermore, JsonRest is the core engine used by JsonRestStore. Consequently, you can simply use the standard Dojo Data API with the JsonRestStore and effortlessly add offline capability with no modifications to your data interaction code. The underlying Rest service automates the handling of caching, storing data locally, and syncing changes. In addition the new OfflineRest module has no dependency on plugins, but rather progressively utilizes offline features that are available, while still operating properly on legacy browsers without offline support.

RESTful JSON + Dojo Data

By on June 13, 2008 12:03 am


NOTE: This post is very out of date. For more up to date information about RESTful JSON in Dojo applications there are a number of more recent tutorials are available, such as the Dojo Store introduction, as well as tutorials demonstrating store-driven grids and trees, among others. You should also take a look at dstore, the next generation Dojo store architecture, for an even more modern take on RESTful JSON. We have a series of tutorials introducing the concepts.

Dojo 1.2 now includes an infrastructure for interacting with JSON REST services and data storage systems. JsonRestStore is a new Dojo Data store for interacting with the RESTful JSON data sources. This new data store allows you to communicate with server side database/persistent data storage using the Dojo Data API with JavaScript and efficiently handles create, read, update, and delete (CRUD) operations. This can greatly simplify client server communication, interacting can be done simply using straightforward JavaScript instead of having to build your communication for CRUD actions. In addition, Dojo data stores can plugin to many Dojo widgets (Dijits).

Complex database driven widgets can be utilized with minimal coding effort. RESTful JSON is an increasingly popular database interface, and in later posts we will look at how JsonRestStore can be used with Amazon S3, CouchDB, Persevere, and Ruby on Rails. The JsonRestStore fully implements the Dojo Data read, write, notification, and identity interfaces. Therefore it can be used with any widget that can utilize these data stores including widgets with editing and live auto-updating features.

Dojo 1.1 in the News

By on April 4, 2008 3:56 pm

Since Dojo 1.1 was released a week ago, several outlets have published articles:

Mobile Gears could help network latency

By on March 5, 2008 4:54 pm

Mobile application development has many challenges. The announcement of Google Gears on Mobile Devices will help solve the problems of network connectivity, network latency, and limited bandwidth. On the desktop, a lot of the focus on Gears was its ability to allow applications to function when your computer was not connected to the network. In addition to building mobile web apps here at SitePen, we released Dojo Offline, which integrates the Dojo Toolkit with Gears to make building disconnected web applications even easier. As we all know, mobile devices lose connectivity with the network on a regular basis. Gears’ ability to keep the data that users need stored on the phone will be key to keeping mobile applications running even when networks fail to keep phones connected.

Dojo Offline Toolkit Status Report for Week Ending April 7th, 2007

By on April 9, 2007 12:15 am

This is it folks! This is the last week of dev for Dojo Offline until we pop the Dojo Offline beta out the door, either later this week or on Monday, April 16th.

Last Week

Last week we finished the Windows installer for Dojo Offline. It comes out to about only 300K! No multi-megabyte download runtimes here. We also continued to fix QA bugs with an eye to the beta launch. The Mac OS X installer was also worked on.

This Week

This is the big week! We will finish the Mac OS X installer, fix important QA bugs that remain and get things into a shippable state, write up good docs, and get a Dojo Offline beta out the door.

The following week, on Tuesday, April 17th, SitePen and Brad Neuberg will be giving the first public presentation on Dojo Offline and creating offline enabled web apps on the open web. The session is at 4:50 PM at the Web 2.0 Expo.

Dojo Offline Toolkit Status Report for Week Ending April 1st, 2007

By on April 2, 2007 12:14 am

Last Week

We finished a bunch of big tasks last week, mostly having to do with fit and finish and our installers:

* The Moxie demo for Dojo Offline used to take too long to load — it was loading about 27 resources on page load — we optimized this to about 3 resources on page load drastically improving page load time.
* We created about 80% of a Windows installer using an open source toolkit from Microsoft called WiX. Unfortunately, WiX
turns out to make easy things hard and hard things close to impossible, trying to turn XML into a programming language. We abandoned using WiX.
* Instead of WiX, we started back over with NSIS, the Nullsoft Scriptable Install System. NSIS turns out to be far better for our tasks, and the installer for Windows is almost over. This installer does alot on installation, such as autoconfiguring the available browsers to use our PAC file, keeping a reference to older proxy settings so they can be reset on uninstallation, starting the Dojo Offline proxy up on system startup, etc.
* Dojo Offline development was being done on the head of Subversion, but the head is changing too drastically for a planned deep overhaul of Dojo. Instead, we branched from the 0.4.2 branch to a new Dojo Offline branch and migrated everything over. All Dojo Offline development is now happening on this branch.

This Week

Expect to see the Windows installer done this week, automatic restarts done this week (if the proxy crashes for some reason), and the Mac installer done. Also, anyone who would love to volunteer with QA by testing the installers, uninstallers, and Moxie please email me: bkn3@columbia.edu

Dojo Offline Toolkit Status Report for Week Ending March 25th, 2007

By on March 26, 2007 12:13 am

Last Week

We’re almost there! Here’s a laundry list of some of the code checkins from the last week:

* We now provide a way to give a ‘magic’ domain name that will resolve to the localhost to help in testing in scenarios where the developer is running both the client and the server on the same machine. Polipo, our local proxy, doesn’t read and parse your hosts.conf file for local domain names or use the platform’s standard gethostbyname function; instead, it rolls its own DNS communication for various reliability and performance reasons, which means that it bypasses the hosts.conf file. This can make it difficult to do rapid testing on your own box, which we now provide a workaround for.
* Did much more testing of offline ‘replays’ on Windows, fixing bugs that were found. Offline replays are when we attempt a communication on the network through the local proxy, have it fail becuause the network is down or malfunctioning, and then ‘replay’ this request against the local proxy’s offline cache in order to provide cached results.
* Fixed a regression where we would always print a ‘details’ message in the Dojo Offline default UI even if there were no messages or errors to report.
* Polipo 1.0 came out in the interim; we were at something like 0.9.99. Diffed our codebase against the 1.0 release to make sure we picked up any code changes made in the jump from 0.9.99 to 1.0.
* Fixed bug found in QA: On Firefox/Mac, sometimes if we are offline and start loading moxie, the whole page doesn’t load
* Made sure UI tells user to restart their browser when a web app has been added to the offline list for the first time. When we add a web app we update our PAC (Proxy AutoConfig) file; some browsers don’t see changes to a PAC file made after startup, and will only pickup changes when restarted. Created a way to have JavaScript layer ‘query’ to see if the PAC file has been updated yet — if not, we tell the user to restart their browser. Tested and QAed on Windows and Mac across Safari, IE, and Firefox.
* QA test done with bug fixes: do repeated on off line cyclings with local proxy to make sure DNS names aren’t cached in a wierd way for offline detection
* Created install.bat file for quickly ‘installing’ and refreshing an installation on Windows to aid rapid testing on that platform
* By default, Polipo saves cached files from memory to disk on a long timeout, sometimes several minutes — found source of issue — Polipo now caches new objects from memory to disk within 2 seconds. Before this a bug would occur on Windows where we would hit a web app for the first time, cache it’s offline resources during the syncing process, and then drop off the network and restart the browser to use the app — you would get a cache miss and errors. Fixed.
* Figured out how to have Windows running in Parallels access Mac OS X without having a network connection, to aid in debugging and more rapid development on Windows. I run the ‘server’ portion of the Moxie demo on Mac OS X, and access it as a ‘client’ on Windows to test the whole system on that platform. In my old configuration I had to be on the network for the virtualized Windows to see the Mac OS X system, since it looks like a different machine and needed an IP address to resolve.
* Did some important project housekeeping I had been putting off: based on usability testing from about 1 1/2 months with Moxie, the Dojo Offline UI was modified to be much simpler, with less options — this was done as a custom UI inside of editor.html — I’ve now rolled these changes back into the widget.html file to make all DOT based apps have this simpler UI, and have removed a bunch of UI featuritis that was related to being too close to the problem, such as the configuration UI (users don’t want to configure offline options — they just want it to magically work)

This Week

Our dev schedule now includes two extra weeks in April to aid in QA of the system. I will be sending out a big call for QA volunteers to help with testing in the first week of April. I’ll blog more about this on Tuesday or Wednesday of this week.

The goal this week continues to be about reliability and performance. Improving the stability of the local proxy, bug fixing, QA testing, getting the Moxie demo to load quickly because that will be the benchmark that folks will judge the system on, etc. I also expect to start putting together the installers and install scripts, which will be tricky since they need to automodify any browser’s that are on the system’s PAC file settings. Any info from folks on how to do this for Safari would be appreciated — I’ve found some solutions but I’m not happy with them.

Dojo Offline Toolkit Status Report for Week Ending March 18th, 2007

By on March 19, 2007 12:12 am

Last Week

Last week we did lots and lots of coding on the local proxy for Dojo Offline. Here’s a laundry list of some of the code checkins and QA fixes:

* Moxie had an encoding bug related to new lines being incorrectly serialized during the Dojo Offline syncing process — fixed
* Polipo, our local proxy, was modified so that if it is online and a network error occurs, either from DNS, talking to a server, etc., we automatically ‘fault’, move offline, and attempt to replay the request against our local cache.
* Sometimes, while loading the Moxie page using the local proxy while online with the browser cache cleared, it can take a LONG time or the browser can hang — Fixed, turned out to be related to how we were handling network timeouts
* Create automated build system for Mac OS X and Windows — now, we can quickly push out Mac OS X and Windows builds with a single build command, issued on the Mac
* On Safari, we don’t consistently get Moxie from the offline cache even if it is there – Fixed
* Windows/Firefox: Go off the network, start up DOT, open browser, go to Moxie — “Falling back to using system resolver” – Fixed
* Got the GNU Debugger (gdb) working on Windows, so we can quickly create Windows builds on the Mac and then debug Windows-specific bugs on Windows
* On Safari: “Safari: Connect to codinginparadise.org:80 failed: Host is down” – Fixed with more generic solution to how we handle network errors, noted above
* “Couldn’t send DNS query: Socket is not connected” – Fixed with more generic solution to how we handle network errors, noted above
* Firefox/Mac OS X — go on network, start proxy, go to moxie, clear browser cache, close browser, drop off network, double click moxie link on desktop — browser just sits and spins – Fixed
* On Safari, off the network, get the following when double clicking link from desktop — only sometimes: DNS: couldn’t open /etc/resolv.conf: No such file or directory – Fixed

This Week

The big task this week is to keep driving QA against the local proxy to improve reliability. Lots of the bug fixes above were basically of this type; we are testing the local proxy against Safari, Firefox, and IE on Windows and Mac OS X.

One example is to fix a caching edge case between the browser and the local proxy. Basically, sometimes we want to ‘bust’ the browser cache to really talk to the local proxy, without busting the local proxy cache. This fixes a bug that can occur under some conditions.

Along side this, we want to get ‘automatic restart’ code going for the local proxy. The local proxy will start on system startup — if it crashes for some reason, we want it to automatically restart.

This week I’d like to also hit other fit and finish issues, as well as potentially start putting together installers.

Dojo Offline Toolkit Status Report for Week Ending March 11th, 2007

By on March 12, 2007 12:11 am

Last Week

Development was a bit slow last week since I was at Microsoft for a few days at their research laboraties and TechFest, and I gave a keynote at Yahoo on Thursday titled “Inventing the Future”. The video of the keynote should be available online this week.

This Week

The big task is the same as last weeks — to continue to refine the local client side proxy. This includes improving linkage between the JavaScript layer and the local proxy to make sure we work correctly when the network is present or gone; pulling and adding resources from the local proxy cache even without HTTP/1.1 headers sent from the server if we are offline; and more. We might also begin to create installers/uninstallers for Windows, as well as code to register the local web proxy for startup when the system starts.