The abandoned ES4 effort made a valiant attempt to add typing (without losing dynamism) via gradual typing; however this still demanded the inclusion of intrusive typing annotations—thus adding type constraints to code polluted the otherwise simple direct JavaScript programming style necessitating more complex, difficult-to-follow code. Other efforts include libraries to define types on functions without extending syntax.

It is now possible to unobtrusively define type constraints on properties and methods in JavaScript with portable JSON schema based definitions called JSON schema interfaces (JSI). JSI is a typing system for JavaScript that does not force any new syntax on JavaScript. Schema structures can be used to define the type constraints in a way that works with existing JavaScript classes and objects.

The new typing module can be used both as a standalone bundle (with the JavaScript JSON schema library) which can be downloaded now. It also can be used as a Dojo module, available as dojox.lang.typed. Adding type constraints to JavaScript classes is very simple; the type constraints follow the JSON schema structure, where the class acts as the root of the schema. One can define property constraints in the properties property. The JSI extension to standard JSON schemas also allows method signatures to be defined in the methods property, with the ability to define the types for parameters and return values.

Using Typed Classes

To demonstrate, first we will create a normal JavaScript class using the standard prototype pattern:

Balloon = function(diameter){
  this.diameter = diameter;
};
Balloon.prototype = {
  getVolume: function(){
     return Math.pow(this.diameter, 3) * Math.PI / 6;
  },
  setVolume: function(volume){
     this.diameter = Math.pow(volume * 6 / Math.PI, 1/3);
  }
};

This class should work fine on its own, but if we want to define constraints to ensure that its interface is properly exposed and used as expected, we can declare the class as typed and add type constraints. Using the standalone library, we first define the class as type-able:

Balloon = typed(Balloon);

And now we can define constraints:

Balloon.properties = {
  diameter:{
     type:"number",
     minimum: 0,
     description:"This is the diameter of the balloon"
  }
};

Now we can try out our constraints:

// this will work normally
goodBalloon = new Balloon(2);
// this will throw a TypeError because the diameter must be a number
goodBalloon.diameter = "not a number";
// this will throw a TypeError as it results in an invalid value for the diameter
badBalloon = new Balloon("not a number");
// this will throw a TypeError because the diameter must be at least zero.
goodBalloon.diameter = -10;

This is effectively JSON schema applied to JavaScript objects with live constraints. Rather than just validating the state of the object at some point in time, properties are kept valid throughout the life of the object. Notice also that the property definition we created only constrains a single property; any other properties can be added or removed from the object freely. Schemas are additive in their constraints; an empty schema does not enforce any constraints on an object, and it can take any form.

However, JSI takes us beyond just constraining properties. We can also constrain the calls to methods. To do this, we can create method definitions in a similar way that we create property definitions in a schema.

A method definition has two important attributes: parameters and returns. The parameters attribute takes an array which defines the type for each parameter value to be passed to the method by position. The returns attribute takes a schema/type definition to constrain what values can be returned by the method. We could define types on the methods that the class implements (those defined on the prototype):

Balloon.methods = {
  setVolume:{
    parameters:[
      {
         type:"number",
         minimum: 0,
         description:"The new volume for the balloon."
       }
     ],
     description: "Set the volume (changes the diameter property)"
  },
  getVolume:{
    parameters:[],
    returns:{
       type:"number",
       minimum: 0,
       description:"The volume of the balloon."
     }  
  }
};

Now we can try out the method definitions:

// create a new balloon:
goodBalloon = new Balloon(2);
// will work properly
goodBalloon.setVolume(10);
// this will throw a TypeError
goodBalloon.setVolume("big");

We can also define class inheritance with our schemas, as well as utilize composition for property and method definitions. Suppose we had defined a subclass of Balloon called ColoredBalloon:

function ColoredBalloon(color){
  this.color = color;
}
ColoredBalloon.prototype = new Balloon(0);
ColoredBalloon = typed(ColoredBalloon);

We can define the inheritance structure in our schema when we define the schema for the subclass:

ColoredBalloon["extends"] = Balloon;

We can use composition for our property definitions. If we have another class called Color:

function Color(red, green, blue){
   ...
}

We could define a color property for our ColoredBalloon class that takes Color instances:

ColoredBalloon.properties = {
  color: Color
};

Now we can try our color property:

balloon = new ColoredBalloon(new Color(33,44,55));
// assign a color, this should work
balloon.color = new Color(255, 0, 0);
// this will fail, since the value is not an instance of Color (instanceof check is used)
balloon.color = "red";
// this will fail, since we inherit the property definitions from Balloon
balloon.diameter = "small";

We can also use existing classes in method definitions as well. For example, a method definition for a java-bean style setter for color would look like:

ColoredBalloon.methods = {
  setColor: {
    parameters:[Color]
  }
};

And all the method definitions are inherited from Balloon (as well as the method implementations through standard JavaScript prototype inheritance).

Using the Typing Module in Dojo

This typing capability is available in the dojox.lang.typed module, and can be used just like the standalone “typed” function. In addition, this module integrates with Dojo’s class constructing convenience function, dojo.declare.

One feature that makes the dojox.lang.typed module convenient is that we don’t need to reassign the return value of typed to the class variable; we can simply do:

dojo.require("dojox.lang.typed");
dojox.lang.typed(dojo.declare("acme.Balloon", null, {
  setVolume: function(volume){
  ... method implementations
}));
acme.Balloon.properties = {
  ... property definitions  
};

Furthermore, we can automate the typing of all our classes by setting the Dojo configuration variable dojo.config.typeCheckAllClasses to true before loading dojox.lang.typed. Then all classes that are declared (after dojox.lang.typed is loaded) through dojo.declare will be typed.

Development/Production Distinction

The typing module is intended to be used for development, easing the construction of classes that need to work together by verifying invariant assumptions. It is highly recommended that type checking be removed for production code. Type checking adds extra bytes (to download the schemas) and has a performance hit (to perform all the type checks), which should be avoided for production. Fortunately, the unobtrusive design makes this very doable. The actual class implementations are unaffected by typing definitions, and as long as the application runs without any type errors, the application should behave exactly the same without type definitions as it does with them. With this design, type definitions can be stored separately from implementations and easily omitted for production code.

In Dojo, ensuring that typing information is omitted can easily be done with the build process. There are a couple of ways of doing this. First we can utilize the build’s excludeStart() function to remove typing information. Therefore we could define our Balloon class:

//>>excludeStart("typed", true);
dojo.config.typeCheckAllClasses = true;
dojo.require("dojox.lang.typed");
//>>excludeEnd("typed");
 
dojo.declare("acme.Balloon", null, {
  setVolume: function(volume){
  ... method implementations
});
 
//>>excludeStart("typed", true);
acme.Balloon.properties = {
  ... property definitions  
};
//>>excludeEnd("typed");

The build process will then strip out all the schema information.

While using excludeStart() is a simple way to exclude type information, it can be a little ugly. Alternately, we could have our schemas stored in separate modules, and then use build profiles to omit the typing modules. This can contribute to better implementation/interface separation.

acme/Balloon.js:
 
dojo.provide("acme.Balloon");
dojo.require("acme.Balloon-schema");
 
dojo.declare("acme.Balloon", null, {
  setVolume: function(volume){
  ... method implementations
});

acme/Balloon-schema.js:
 
dojo.provide("acme.Balloon-schema");
dojo.config.typeCheckAllClasses = true;
dojo.require("dojox.lang.typed");
acme.Balloon.properties = {
  ... property definitions  
};

Now in the build profile, we can include the schema modules as layerDependencies to completely exclude them from the build:

dependencies = {
	layers: [
		{
			name: "../acme/Balloon.js",
			layerDependencies : [
				"../acme/Balloon-schema.js"
			],
			dependencies: [
				"acme.Balloon"
			]
		},
...

Limitations and Potential

Type checking on property changes can not be performed in IE because this feature relies on JavaScript getters and setters. However (as noted before), this is a development-time tool, and full cross-browser support is not necessary to reap the benefits of type checking. Checking for invalid type assumptions—and utilizing typing information for integration of components between different developers—can be achieved during development on a subset of all browsers. Type checked applications will still work on IE; there just won’t be type checks performed on property changes. For production, when full-cross browser support is needed, it is recommended that type checks be eliminated anyway.

The JavaScript-based type checker is also limited in that it can’t intercept delete operations on properties. When a delete takes place, the setter is not fired, and the type checker is not able to reject the action (in the case of required properties). The type checker also cannot control the addition of new properties that have been declared (in the property definitions or prototype).

JSI-based type checking does not have any mechanism for defining types of local variables. This is intentional. Local variables rarely benefit from type information. Integration of components from different sources can define their interaction solely through public interfaces (which are typeable with JSI), and local type inference mechanisms can usually provide sufficient information about local variable types for IDEs and code analysis.

A more complete JSI-based type checking system can be seen in Persevere. Persevere implements JSI for type checking in the server-side JavaScript environment, and is able to reject delete and new property actions that do not conform to the schema. Typing in JavaScript has been a controversial idea due to the obtrusive nature of traditional typing annotations, but JSI allows gradual type constraints to be applied in a way that does not interfere with JavaScript’s simple syntax and object model.

This typing scheme has also been proposed to the ECMAScript working group. As noted before, the typing system proposed for edition 4 required extensive new syntax and relied significantly on namespaces, which have been rejected. A schema-based typing system is perhaps the most reasonable alternative for adding typing capabilities to ECMAScript, and provides significant benefits over the more static type systems that quash the interactive, dynamic nature of JavaScript. With the more gradual approach that is being taken to evolve the language, this typing system provides the perfect mechanism for preserving the familiar syntax and meta-programming abilities that exemplify JavaScript.

Portability and Metadata Retrieval

JSON schema interfaces are completely language-agnostic. This means that interfaces can be defined that apply to implementations in different languages. This can be particular useful for defining contracts in RPC-style situations. In fact, the SMD format, which is used extensively by Dojo for defining remote services, is essentially a remote service-oriented form of JSI.

One of the challenges with traditional class systems is finding a way to reify type information. This is a non-issue with JSI, since the type system is based completely on a runtime data structure that is readily available and already understood.

Typing in JavaScript

While there has been a lot of controversy over typing, only the most extreme would not be willing to concede that in certain situations, typing can be a valuable asset for defining contracts between components. JSI allows for more robust evolution of an application, faster detection of invariant violations that lead to hard-to-find bugs, and better information for integration of components and type-based tooling. Furthermore, leveraging JSON schema gives developers a very rich set of options for constraining values, much more than most type systems.

JSI gives us a standardized data structure for defining types—allowing for more robust large-scale application development, a better pathway for JavaScript tool evolution, clean implementation/interface separation, and an integrated, intuitive class/typing system. And it’s available in both the Dojo Toolkit and Persevere today!

Receiving Data Notifications

REST Channels is designed to unite the concept of topics with resource locators. Therefore, REST Channels can leverage Bayeux’s topic names to indicate the resource that is being targeted. We can then subscribe to Bayeux’s channels and then delegate all the data notification messages to the restListener which will delegate the proper data events through the data stores:

// first initialize cometD
dojo.require("dojox.cometd");
dojo.require("dojox.cometd.RestChannels");
 
dojox.cometd.init("/cometd");
 
dojo.subscribe("/cometd/**", function(msg){
   dojox.data.restListener({
      channel: msg.channel,
      event: msg.data.event,
      result: msg.data.result
   });
});

This could be used in conjunction with a JsonRestStore, and messages would be delegated to the store’s events:

weatherStore = new dojox.data.JsonRestStore({target:"/weather/"});

Now it is possible to publish data change notification messages such that clients will receive and interpret the messages properly. To publish a data notification message on Jetty’s cometD hub, we can call the doPublish with something like:

bayeux.doPublish("/weather/lax", null, laxWeatherUpdate, null);

Which should produce a Bayeux notification message like:

{
  "channel": "/weather/lax",
  "clientId": "23ab8a887baaa",
  "data": {
     "event":"PUT",
     "result":{
       "low": 65,
       "high": 78,
       "skies": "clear"
    }
  }
}

Now this message will be routed through the restListener and onSet events will be fired for any attributes that were updated in the local cache of the “/weather/lax” item. If the lax item was being displayed in a data-aware widget, it will automatically be updated. For example, the DataGrid supports data notification events, and if this item appears in a DataGrid, the corresponding row will automatically display the new value.

Subscribing to Resources

One of the key integration points that REST Channels provides between data access and notification routing is in auto-subscribing to resources when they are retrieved. The primary use case for REST Channels is in providing a real-time view of data, and therefore when data is retrieved from the server, REST Channels can automatically subscribe to the resource to receive all future notifications of changes to the object. With a true REST Channels implementation, subscription information is included in GET requests, so that a single request can be made to a server that both requests a resource and subscribes to it at the same time. If you are just using a cometD server, such an integration is not automatic. There are a couple of different approaches for subscribing to resources.

On the client-side, we can send subscription requests when we make GET requests. This can be done by intercepting the REST GET request handler, and creating subscriptions.

dojo.connect(dojo, "xhrGet", function(args){
  dojox.cometd.subscribe(args.url, function(){
    // nothing needed, handled by the catch-all subscription
  });
});

All GET requests will then trigger a subscription automatically. One could easily add some conditional logic that only subscribes on a subset of requests.

The biggest disadvantage of client generated subscription requests is that they generate two requests for each resource access, one GET and one subscription request (as a POST). If the REST handler for a server can be modified to understand the subscription header that REST Channels adds to GET requests, the server can subscribe the client without requiring additional requests. The key to making this work, is to be able to connect cometD’s client connections with the request handler. One way to do this is to define a Bayeux handler that will put the current client object in the HTTP session, and then retrieve it from the session in the REST handler. The REST handler can then subscribe the client to the resource locator topic.

One tricky aspect of this approach is that it is possible for multiple client connections to exist in a single session. Each page will have its own cometD client connection, but will be in the same HTTP session. Therefore, the client connections need to be stored and accessed by their client id. In order for the REST handler to determine the correct client id for a client, you can have the Client-Id header set on the requests, and then this id can be used to find the correct Bayeux client connection. RestChannels automatically sets the Client-Id header, but the correct client id from the cometD handler must be assigned to the RestChannels clientId variable:

dojox.cometd.init("/cometd").addCallback(function(){
  dojox.rpc.Client.clientId = dojox.cometd.clientId;
});

Together these techniques can be used to combine a REST architecture with a cometD server for data notifications with Dojo’s REST capabilities in JsonRestStore, and complete integration into Dojo’s data event notification system.

These statistics are even more impressive when one considers all the additional functionality that Persevere provides while outperforming these other storage systems. MySQL utilizes traditional fixed structure schemas requiring homogenous records in a table, while JavaScriptDB (as well as CouchDB) support storage of heterogeneous objects of any structure in tables. Persevere/JavaScriptDB goes further with the flexibility to evolve schemas and handle partial schemas. Persevere also provides integrated server side JavaScript (SSJS) with persistence, Comet-driven data change notifications, JSONQuery, standards based HTTP interface with content negotiation, JSON-RPC interface to SSJS, cross-domain handling, CSRF protection, and more. All of these things are additional features that one would have to add to the stack for other storage systems, making them even slower. Persevere includes this functionality out of the box, while still maintaining extremely fast performance.

Test Scenario

These tests were performed on a Mac/OS-X with a 2GHz dual-core Intel processor and 1 GB 667 MHz DDR2 memory. The PHP/Apache/MySQL setup used MAMP 1.7.2 which includes PHP 5.2.6, MySQL 5.0.41, and Apache 2.0.59. The CouchDB tests were performed with CouchDBX version 0.8 (which uses CouchDB 0.8.1). Persevere’s nightly builds from late March were used for JavaScriptDB tests.

Three different operations were performed in the tests:

• Insert/POST operation to create a new object
• Update/PUT operation to update an object
• Query/GET to search for objects by an (indexed) field/property

On the PHP/MySQL tests, all operations were handled by a very simple PHP script that first did a quick security check query against a small table (actually empty for the tests) to emulate the security capabilities of Persevere and CouchDB (although CouchDB’s security capabilities are limited, and probably often require additional logic), and then the main query was executed against MySQL, whether it be an INSERT, UPDATE, or SELECT. All the created objects/records had four properties/fields for all three systems. In MySQL, two properties were indexed, one being the primary key, the other being the property that was queried on in the query requests. In CouchDB, a simple view was created that indexed on a single property. This view was used for the requests that queried by index. Both Persevere and CouchDB tests used their standard HTTP interface for creating, updating, and querying.

Tests were carried out by a HTTP client running on 10 threads concurrently issuing a sequence of 200 of each type of request. The “full test” performed create, update, and query requests. The “write test” performed create and update requests, and the “read test” only performed query requests. The files used to perform the benchmarks are available here.

Test Conclusions

Two sets of tests were run—one that used fast commits that do not wait for committed data to be actually physically written to the disk (allowing for normal OS write-back caching), and high-integrity commits which cause the committed data to be forced to the disk. JavaScriptDB has a setting for choosing which style of commits to use. In MySQL, the MyISAM storage engine was used for fast commits, and the InnoDB storage engine was used for high-integrity commits. CouchDB always uses high-integrity commits.

While PHP is not the fastest language, the script that was used for these tests was very trivial, and it’s unlikely that PHP code execution significantly detracted from the overall performance of the PHP/MySQL combination. With this simple, streamlined PHP script, a very fast classic setup was used. Alternate languages would not be likely to improve the performance by very much. Yet, Persevere’s JavaScriptDB still beat this setup by a significant margin. With more complex request handlers that might provide more of the functionality that Persevere already provides, the margin would be likely to increase even more. Quite simply, the classic application server + MySQL database setup is hard-pressed to compete with Persevere in terms of performance for most normal database interactions.

So how does Persevere achieve this level of performance with the JavaScriptDB storage? The dynamic object-oriented nature of the data that is stored in JavaScriptDB is much different than that of a traditional relational database, so a number of innovative approaches were employed.

Direct Data-Bound Object Representation

One of the central concepts of Persevere is that all persisted data is mapped to JavaScript objects. This enables server side JavaScript functions and handlers to easily be able to interact with persisted data, and provides a convenient in-memory representation of data that allows for intuitive normal object-oriented data interaction. However, in Persevere this more than just a convenient API—it also facilitates efficient memory utilization by providing a single in-memory representation that can be reused at multiple levels.

In a traditional application stack, a record must have separate in-memory representations for each different level in the stack. A database may have an in-memory representation before serializing result sets back to the application. The application may have result set level representation, which then might be mapped to an object representation. Every one of these levels consumes more memory. These extra layers increase latency and overhead as well. In addition, most database driven applications rely on TCP/IP communication with the database, which consumes a large amount of resources as well. With the JavaScriptDB, the single in-memory object is efficiently reused at the database level for all result sets and data caching. This not only means less memory-consumption, but it also translates to more efficient CPU cache utilization for Persevere, and direct low-latency access to data.

Shared Cache of Objects with Copy-on-Write

Not only are in-memory objects shared between the application level and the database level, but Persevere also utilizes a shared cache of objects between threads to ensure that any given record/object only exists in memory at most one time. Traditional application frameworks process separate HTTP requests concurrently and each request will have its own result set and a copy of data. These can lead to significant duplication of data in memory. With Persevere, objects are always reused if they are still available in memory.

While this technique is relatively simple for read-only data, Persevere still maintains virtual memory isolation between threads to protect against concurrent access between threads and ensuing race conditions. Persevere does this by performing copy-on-write style values in objects. When a property is modified, internally its value is modified to being a “transactional” value that actually has multiple states depending on which thread is accessing the objects. Therefore an object can be modified by one thread, but another thread can access the same object without seeing the uncommitted change. Property changes are made visible when transactions are committed. This technique allows Persevere to maintain transactional isolation between concurrent request handlers, while minimizing the record/objects that must be held in memory. Persevere’s architecture combined with JavaScriptDB’s integration minimizes memory consumption, allowing internal caches to be maximized for optimal performance.

Persevere utilizes the sophisticated least recently used (LRU) caching capabilities of the Java Virtual Machines’s (JVM) soft referencing mechanism. In-memory objects, as well as JavaScriptDB’s indices, are cached via soft reference tables. This allows the JVM to utilize an integrated view of reachability and object access timestamps to determine which objects to collect and discard. This means that objects that are reachable by currently executing code will always stay in the cache (since they must stay in memory due to reachability) as long as they are reachable. Unreachable objects are then discarded according to LRU strategies. Since the JVM’s garbage collection handles object collection at a global level, it is also able to optimally select objects for collection without being constrained by module level view. This means that if the indices are not being used frequently, more memory can be allocated to the object cache and vice versa. Caches are maintained according to usage and reachability with the JVM’s global perspective for optimal discarding strategy.

Append-based Database Storage

JavaScriptDB uses an append-based database format to store data. Many traditional database will synchronously commit data to a transaction log file before committing data to the main storage table, which requires multiple writes. On the other hand, JavaScriptDB appends transactional data directly to the main storage file such that writes can be committed with a single IO operation. This also enables JavaScriptDB to efficiently maintain a version history of the database and its records. The storage file is essentially a running log of transactions, and these transactions are exposed as the Transaction table. By storing data as a sequential set of transaction, JavaScriptDB not only can persist data quickly, it also provides efficient access to the transactions that have taken place and a version history of the database and objects within it.

Adaptive On-Demand Concurrent Indexing

JavaScriptDB features a dynamic approach to indexing that minimizes the configuration and management required to create and maintain tables, and maximizes performance. By default, JavaScriptDB indexes all properties of persisted objects, so typical queries can almost always be run in fast O(log n) time. However, the indexer does not block write operations to complete index updates when objects are added, deleted, or modified. Rather, the indexer execution takes place concurrently in a background asynchronous task executing threads. As objects are indexed, the index update operations are delegated to the appropriate index nodes, which are also executed as asynchronous tasks. When an index is needed for a query, any outstanding updates along the node tree path are completed so the query can execute.

It is worth noting that this on-demand indexing does not mean that the entire index must be updated to execute a query. Often (and usually in the case of large databases) an object may be updated that affects an index node that isn’t used in a subsequent query. In this case, the query can still execute without waiting for the index node to be updated. JavaScriptDB properly orchestrates concurrent indexing such that nodes are updated through lower-priority background threads when possible, and immediately updated on-demand as necessary. This allows write operations to take place very quickly, while still allowing indexes to be ready for fast query operations as well. This also allows Persevere to utilize resources and CPU processing more evenly and smoothly. Background processes can take CPU time as needed when client requests are not demanding immediate data retrieval.

Furthermore, JavaScriptDB uses adaptive techniques with indexing. If a particular index has been unused for some time while many objects have been added to a table with the corresponding property, JavaScriptDB will stop proactively updating the index to conserve resources. When an index is no longer proactively updated, the index will only be updated on-demand, when a query is performed that requires that index. Once the index is updated, it will resume proactive updates (at least until disuse causes it to go back to a non-proactive update state). This approach allows JavaScriptDB to automatically do appropriate and efficient indexing with minimal manual configuration. JavaScriptDB does also support manual configuration of indexes, for situations where you may want explicit control of indexing.

Batched writes in integrity mode

One of the most expensive operations that a database can perform is a forced synchronous disk write operation. These operations are necessary for high-integrity commit mode where the commit does not return until the database is certain that the data has actually been written to the disk, fulfilling the durability component of ACID compliance. These operations can take around 10ms. In order to improve the performance of high-integrity commits, Persevere will detect when multiple writes are taking place concurrently and batch multiple writes together in a single synchronous disk write operation. When a number of concurrent write requests are being sent to Persevere, this can significantly reduce the number of synchronous writes that must take place and greatly improve performance.

Pluggable Storage

Persevere uses a pluggable storage system. JavaScriptDB is one of several data source plugins (the default data source) that can be used with Persevere. Persevere supports heterogeneous storage configurations. This means you can leverage the performance and flexibility of JavaScriptDB in Persevere without abandoning existing relational databases, as well as other data sources. Even custom data sources can be created for unique storage systems.

The ServerJS working group is also considering a standard API for database interaction that might possibly allow JavaScriptDB to be used as a standalone database engine for use by other Rhino-based frameworks like Helma (of course Persevere + JavaScriptDB can already be used with existing JavaScript modules, and it can be used as a database for Java applications through it’s Java API).

Future Improvements

This is the first release of JavaScriptDB, so there is still significant opportunities for continuing to improve and refine this storage engine. Currently, JavaScriptDB does not utilize indices for nested object queries (the equivalent of inner joins in relational DBs). Consequently queries of the form [?prop1=’something’] will execute in O(log n) time, but queries of the form [?prop1.prop2='something'] will only execute in O(n). Future versions will provide fast O(log n) for a much broader range of queries. A later release will also provide true ACID compliance (the current version does not fulfill the atomicity constraint). Finally, replication/clustering services will be added in the future as well, for distributing Persevere workload across multiple servers.

Real Value

As Alex Payne pointed out, the economy may be ending the era of disregard for system performance and efficiency with the excuse of buying more servers. More servers costs more money, and architectures like Persevere that can efficiently handle large numbers of users and traffic with minimal hardware resources equates to real money saved.

Persevere combines numerous advanced capabilities for web-accessible data including standards-based HTTP interface, JSONQuery, JSON-RPC, server side JavaScript, Comet-based data notifications, robust security, and more. Now these capabilities are available with speed and scalability that outperforms the most common web application systems, allowing you to build high-performance client/server Ajax web applications with unprecedented ease, efficiency, and value.

Update: Jan Lehnardt pointed out that CouchDB is now at version 0.9.0 and OS-X is not the optimal platform for CouchDB, so the latest version CouchDB can presumably improve upon the CouchDB performance shown in these tests. Hopefully we can progress towards better benchmarking tools for this new breed of databases.

In the user interface design phase, we decided that each of the lists in the “Your Queue” section would be based on HTML tables. In Queued, only the DVD queue and the Instant list can be reordered, so we’ll focus on the DVD queue.

Here’s the HTML for the DVD queue:

<table class="queueList">
	<thead>
		<tr class="listQueuedHead">
			<th class="index">Movie</th>
			<th class="top"></th>
			<th class="title"></th>
			<th class="instant">Instant</th>
			<th class="stars">Star Rating</th>
			<th class="genre">Genre</th>
			<th class="format">Disc Format</th>
			<th class="remove">Remove</th>
		</tr>
	</thead>
	<tbody id="queueTemplateNode" class="listQueuedBody"></tbody>
</table>

This table’s body corresponds to a queueList object:

qd.app.queueList = function(){
	this.list = [];
	this.domNode = null;
	this.constructor = function(options){
		dojo.mixin(this, options);
		this.domNode = dojo.byId("queueTemplateNode");
		this.dndSource = new dojo.dnd.Source(this.domNode, {
			accept: "movie",
			skipForm: true,
			withHandles: true,
			isSource: true,
			singular: true,
			creator: dojo.hitch(this, this.createItem)
		});
		this.dndSource.insertNodes(false,
			dojo.filter(this.result.items, function(i){
				// in English: for "queue" and "instant",
				// skip when position==null
				return i.position ||
					(this.type!="queue" && this.type!="instant");
			}, this)
		);
		dojo.connect(this.dndSource, "onDropInternal", this, "onDrop");
		dojo.connect(this.dndSource, "onDndCancel", this, "onDragCancel");
	};
};

When the queueList is instantiated, a dojo.dnd.Source object is created; this means the body of the table is now a source for items to be dragged out of. This particular source also functions as a container — items can be dropped on it — and will only accept items with a type of “movie” (meaning that all other items will be ignored). We then populate the queue when insertNodes is called with the filtered list of items we want to add. The insertNodes function calls the creator parameter to dojo.dnd.Source. Let’s take a look at what that function does:

this.createItem = function(item, hint){
	if(hint == "avatar"){
		var node = dojo.doc.createElement("div");
		node.className = "movieDragAvatar";
		node.innerHTML = item.item.title.title;
		this.draggingitem = item;
		return {
			node: node,
			data: item,
			type: this.type
		}
	}
	var listItem = new qd.app.queueItem({
			item: item,
			type: this.type,
			parent: this
		}, this.domNode);
	return {
		node: listItem.domNode,
		data: listItem,
		type: this.type
	}
};

As you can see, createItem gets called in a couple of situations. We mentioned insertNodes, but it is also called when you drop an item onto a source and when you start a drag. The latter situation calls the creator with the hint argument set to "avatar". In that situation, we return a node that has a simple representation of what we are dragging that follows the mouse. When insertNodes calls createItem, it will instantiate a qd.app.queueItem which handles creating the correct markup for a row in the table. Then, insertNodes adds the node property of the returned object to the node passed to the dojo.dnd.Source constructor. You might be wondering what a qd.app.queueItem does in its constructor; it simply takes the information provided in the item and constructs a table row.

The dojo.dnd.Source object also provides some events to which we can connect. The first, onDropInternal, is triggered when the user drops an item dragged from the DVD Queue into the DVD Queue — in other words, a re-order of the queue. The event handler, onDrop, looks like this:

this.onDrop = function(nodes){
	var node = nodes[0];
	this.dndSource.getAllNodes().forEach(function(n, i){
		var listItem = this.dndSource.getItem(n.id).data;
		if(n.id == node.id){
			this.reorder(listItem, i+1, false);
		}
	}, this);
	this.draggingitem = null;
};

The reorder method handles talking to Netflix’s service to move items around and it also renumbers the rows in the HTML table. The second interesting event (to us) is onDndCancel. It is triggered when a drag operation is canceled — for example, when the user hits ESC during a drag. It is also triggered at the end of any drop. Our event handler, onDragCancel, cleans up after our drag operations:

this.onDragCancel = function(){
	if(this.draggingitem){
		this.draggingitem = null;
	}
};

In the end, that’s all it took to get drag and drop working in Queued. Dojo’s drag and drop API is a robust, flexible system which can be easily tailored to your application by passing a creator function to a dojo.dnd.Source. All the creator function has to do is know how to create a representation of the item to be appended to the source (in this case, a row appended to a table) and create a representation of the item while it’s being dragged. After you’ve set that up, you just have to listen to the source’s events and make adjustments on the server-side accordingly.

Dealing with the AIR Application Sandbox

One major issue we discovered, during the development of the offline-to-online synchronization, was running head-first into the wall created by the application sandbox. This sandbox is AIR’s mechanism for preventing any kind of scripting exploits within AIR applications, and it is very robust. However, one of the major restrictions is the following:

For HTML content in the application security sandbox, there are limitations on using APIs that can dynamically transform strings into executable code after the code is loaded (after the onload event of the body element has been dispatched and the onload handler function has finished executing). This is to prevent the application from inadvertently injecting (and executing) code from non-application sources (such as potentially insecure network domains).
…
One restriction is in the use of the JavaScript eval() function. Once code in the application sandbox is loaded and after processing of the onload event handler, you can only use the eval() function in limited ways.

(For more information, you can check out the AIR documentation about security.)

While we were aware of this restriction from previous work with AIR, this threw us for a curve when implementing the transaction queue. The original design of the queue was to place the functions actually needed to accomplish a synchronization task within the database itself, so that when the user returned online, the database could simply be read and executed. But without the ability to use eval() or the new Function() constructor, we could not use this approach.

Instead we created a function map:

var methods = {
	rate: function(args){
		return qd.service.titles.rate(args);
	},
	add: function(args){
		var dfd = new dojo.Deferred();
		qd.app.queue.addMovieById(args.movieId, null, args.queue);
		setTimeout(function(){
			dfd.callback();
		}, 250);
		return dfd;
	},
	termAdd: function(args){
		var dfd = new dojo.Deferred();
		var queue = args.queue;
		args.result = function(item){
			qd.app.queue.addMovieById(item.guid, null, queue);
			dfd.callback();
		};
		delete args.queue;
		qd.service.titles.fetch(args);
		return dfd;
	},
	modify: function(args){
		return qd.service.queues.modify(args);
	},
	remove: function(args){
		return qd.service.queues.remove(args);
	},
	discs: function(){
		return qd.service.queues.discs({ max: 1 });
	},
	instant: function(){
		return qd.service.queues.instant({ max: 1 });
	}
};

View the code that handles the offline sync.

By writing out a function map in this matter, we could then store the arguments used with this map as JSON structures in the database (as well as the map key), and call it using JavaScript’s associative array syntax:

dojo.forEach(data, function(item){
    var method = item.method;
    actions.push(function(){
        if(methods[method]){
            try {
                qd.services.online.onSyncItemStart(item.prompt);
                methods[method](dojo.fromJson(item.args||"{}"))
                    .addCallback(function(){
                        qd.services.online.onSyncItemComplete();
                    }).addErrback(function(err){
                        console.warn(err);
                        qd.services.online.onSyncItemComplete();
                    });
            } catch(ex){
                console.warn("sync: ", ex);
            }
        }
    });
});

By using the associative array syntax, we were able to circumvent the sandbox restrictions put in place by AIR, and do the online sync in an easy-to-follow manner. While this works fine for us, it is not an intuitive solution for most developers.

Initial Window Placement

One of the problems with the structure of an AIR application is the way the manifest is used for compilation. For those unfamiliar with AIR application development, a manifest is the initial XML file AIR uses to compile an application into an executable format; it defines things like the version of the application, the publisher of an application, and other attributes AIR needs to make your HTML-based application into a usable application. Among these attributes are the definitions of the initial window—both the size and the initial placement.

Most desktop applications are written using the following initialization routine:

1. A splash screen is placed on the desktop
2. The application performs any initialization tasks it needs to—such as loading in parameters, defining/checking variables, and other things
3. When the initialization completes, the actual application interface is presented to the user.

For a number of reasons, we decided to develop Queued without a splash screen. We felt that we could complete all necessary initialization without seriously interfering with the user experience; splash screens have a tendency to delay user interaction to the point where it can be frustrating (as in, It is my queue and I want to see it NOW!), so our goal was to make it so you were connected to your information as absolutely fast as possible.

Unfortunately, one of the side effects of this approach is the inability to use basic features like the last x/y window coordinates of your previous usage of Queued. Don’t get me wrong—we can capture that information and store it without any difficulty. But usually that sort of thing is used during an initialization process, i.e. splash screen (from the user interface point-of-view), and because we decided not to use that approach, the main window for Queued will always appear in the same spot (with the same size) regardless of what you have done previously with the main window.

We felt (pretty strongly) that loading up the application as fast as possible was an acceptable trade-off to remembering the last settings of the window hosting the application.

The ELS, performance, and serialization

In Part I, I talked a bit about AIR’s Encrypted Local Storage, or ELS, and how we used it to set up an encrypted database without forcing user interaction. This storage mechanism (similar to the Firefox browser storage or the proposed HTML5 feature) is essentially a cookie store that allows for larger data structures and limits; and it has been designed to make sure that the store is isolated by application and user (so 2 users on the same machine, using Queued, would each have their own ELS). Thinking that using the ELS would be faster than using the encrypted database for certain things, we began using it not only to store the password of the database—but also for essential user information, such as your name, some of your preferences, and your queues.

We did not realize that this was A Bad Idea™. On my own Netflix account, I have an average of 35 titles in my disc queue, and another 35 in my Instant queue. All queue items in Queued are stored as a fairly complex JSON structure (see this post by my friend and colleague Mike Wilcox) and because of this, a user’s queue information can get fairly large. Adobe itself states (with regards to ELS size and performance):

The encrypted local store may perform more slowly if the stored data exceeds 10MB.

What we found was that it took a lot less data to make it perform slowly; saving my information (with my queues) was taking something on the order of 10 seconds to do for approximately 2.5MB of data, and was interrupting the user interface thread, providing an unacceptable user experience.

We solved this issue by pulling all of the user’s queue information and using the database instead to cache it. Now we only use the ELS to store basic data—such as the user’s information, preferences and authorization info (soon we won’t be using the ELS at all but that is another story for another time).

Serialization and the ELS

Another issue with the ELS is the way objects are serialized and marshaled for it. The ELS uses ByteArrays for storage (presumably to facilitate the encryption process). AIR’s ByteArray is pretty robust; it allows you to read and write all sorts of data types to the internal stream, taking care of the serialization process (i.e. converting a type into a concrete format) in a seamless manner.

Unfortunately, the serialization format AIR uses is the ActionScript Message Format (AMF). For Flash, Flex and AIR apps written based on ActionScript objects, this is a great way of doing things. But for native JavaScript objects, not so much—partially because AMF assumes existing class structures for marshaling (i.e. deserialization), and JavaScript object structures (such as the ones we use for defining things like titles and queue items) are essentially classless mixins.

In the end, we simply used Dojo’s JSON handling facilities to both serialize and parse, and simply pushed things into the ELS as a string.

Managing Queued’s Exit Events

One of the coolest features of Queued is the Mini-Queue:

This is a chromeless window that shows you (at a glance) what you have at home, and can do other things (such as alerting you when Netflix receives or ships other titles). It’s a neat piece of functionality that allows you to run Queued in the background without having to have the main window open all the time.

However, introducing a second window (particularly a chromeless one) into the application structure caused a number of management issues—in particular, dealing with the application’s exiting event. What we needed was a reliable way of controlling how Queued quit when showing the Mini-Queue but without showing the main application window, and we found that trying to use the exiting event seemed to be unreliable (certain operating system actions would cause the event not to fire, debugging it was a significant pain, and there were a number of other problems I won’t go into here).

In the end, we set up Queued so that if you have the Mini-Queue open, the only reliable way to exit the app was to use the Dock Icon/Taskbar tray to force the exit:

We will probably revisit this in a future release of Queued.

Tray/Dock Icon Loading and Menu Handling

Speaking of the context menu attached to the tray or dock icon…we ran across an issue where the application icon that is supposed to appear in the tray of the Windows taskbar would not load correctly. For loading the icons, we used dAIR’s Icon (from the Dojo Extensions for Adobe AIR library), which handles many of the basic AIR tasks in a convenient higher-level Dojo-esque manner.

For reasons still undetermined—though we’ve traced it to concurrent network activity—loading the icon images for the tray consistently failed when loading Queued. Without being able to know when any kind of network activity is already in progress, the only solution we found was to retry the image load until successful (or until we reached a limit of retries). This solution is less than ideal but It Works™.

Tray menu handling

Last but not least, one quirk of Adobe AIR is the separation between the DockIcon vs. the SystemTrayIcon. Both classes are designed to support OS-based functionality; however, because of the nature of the DockIcon we ran into major issues when trying to determine if someone had opened the context menu attached to it.

Simply put, the DockIcon does not support any kind of mouse click events, whereas the SystemTrayIcon does.

In the end, we simply wrote the system tray functionality to mimic the native behavior of the DockIcon. We found this solution to be less than ideal, since what we really wanted was for the user to be able to close the main window of Queued, and have the icon/app indicator disappear from both the dock and the Windows taskbar (a la uTorrent). Because of the behavior of the DockIcon, this was simply not possible to do under OS X.

A feature request improvement for Adobe AIR would be to implement an icon + menu that gets placed in the Menu Bar of OS X, and have that act the same way as the SystemTrayIcon class, something like this:

This would solve the aforementioned issue in a consistent manner.

Conclusions

Working with Adobe AIR was a bit of a treat, and we had a lot of fun creating Queued—but in many ways, we learned new lessons the hard way. I hope talking about some of the issues we ran across during Queued’s development can help you when creating your own AIR-based applications!

The foundation for the beautiful theme for Queued was laid down by colleagues Damon Dimmick and Torrey Rice, and their amazing wireframe and mockup work (respectively) provided the building blocks for laying down Queued’s skin.

Powered by WebKit

One of the advantages to building an Adobe AIR application that we can’t talk enough about is that you only have to develop for one layout engine. Since Adobe uses WebKit to power the AIR runtime, we as developers get to take advantage of all of the mature, feature-rich goodness that comprises WebKit, including some of the things in the WebKit Supported CSS Properties. Additionally, Adobe has a document that describes the extensions to CSS that AIR uses, which my colleague Sam Foster will talk about in greater detail in an upcoming article. In our development process, we found that AIR—while not being able to take advantage of all of the WebKit supported CSS properties—still provided us a good chunk of styling advantages that we were able to implement to make the theming job quicker and easier.

Keeping it Local

One of the things we were able to do with Queued is to take away some of the application’s workload in regards to box art image files. Rather than saturate the wire with this data, we use a local store to house your box art images (along with other data such as preferences, etc.) to help keep Queued’s online presence nimble.

PNG Transparency

Building applications for use with Adobe AIR’s WebKit engine also means that we don’t have to worry about IE 6’s PNG alpha transparency issues. Not having to deal with this thorn in the side allowed us to use drop shadows to our heart’s content, and allowed the movie information pop-up to easily achieve the look we were after in our mockups.

CSS 3, How We Love Thee

Every developer on the Queued project remarked how nice it was to have to support only one layout engine; theming Queued has certainly left us spoiled in this regard. When styling Queued, we were also able to take advantage of some of the CSS 3 selectors and properties that are supported by WebKit. One of the things we wanted to do was prevent the user from being able to highlight areas of the application that they really didn’t need to. By using -webkit-user-select, we were able to tailor the user-selectable content to our needs.

html, body, #layoutNode {
	width:100%;
	height:100%;
	overflow:hidden;
	-webkit-user-select:none;
}
 
#movieInfoDialogNode .info {
	position:absolute;
	top:46px;
	right:8px;
	width:560px;
	height:412px;
	padding-right:40px;
	overflow-y:auto;
	-webkit-user-select:text;
}
 
#searchResultsList .movie .info {
	display:block;
	position:absolute;
	left:150px;
	right:8px;
	padding-top:7px;
	-webkit-user-select:text;
}

We also took advantage of WebKit’s border-radius support, using it in areas like the search input background.

#searchBar input {
	width:238px;
	padding:5px 8px;
	outline:none;
	-webkit-border-radius:12px;
	border:1px solid #C60000;
	color: #AFAFAF;
	background:url("../img/searchBg.png") repeat-x left top;
}

To prevent movie titles with really long names from breaking our layout, we used text-overflow. We also used text-overflow in other areas of the application, such as login input.

#searchAutoSuggestList li {
	color:#bf0000;
	margin:0 6px;
	padding:0 3px;
	width:196px;
	white-space: nowrap;
	overflow:hidden;
	text-overflow:ellipsis;
	-webkit-text-overflow:ellipsis;
}

Another great example of a CSS 3 property that we were able to use is support for multiple backgrounds. We wanted to style the selected nav element of the application with a “glowing” type of effect coming out from the left and right sides of the element. Rather than having to use multiple divs to get this done, we were able to use multiple backgrounds to produce the effect. The following three images show the selected element with no background image, one background image (positioned top right), and two background images (one positioned top right, the other top left), ultimately providing the glowing effect we had on our mockups.

#topMoviesSubNav ul li.selected,
#queSubNav ul li.selected,
#authSubNav ul li.selected,
#prefsSubNav ul li.selected {
	background:url("../img/subMenuActiveBorder.png") repeat-y top right, 
        url("../img/subMenuActiveBorder.png") repeat-y top left;
	position:relative;
}

Conclusion

Theming Queued was a tremendous amount of work, but by taking advantage of local storage, having only one layout engine to work with, and harnessing some of the CSS 3 goodies supported by AIR’s version of WebKit, that workload was decreased. This allowed us to get the job done quicker, and we had a lot of fun along the way.

Stocker uses these technologies to emulate a stock monitoring application. We’re using made up data, but that’s actually more interesting. The Persevere server generates new stock items at certain intervals, and then pushes them to the browser with Comet. Then the Data Store updates its items and triggers an onSet notification. The DataGrid and DataChart are both connected to the same store, and are listening to that event. They then update their displays and show the stock items and their latest data.

Persevere

Persevere, which we chose to use inside the robust Jetty web server, allows us to quickly develop a data-driven web application that can directly interface with Dojo, without having to setup a relational database and create Ajax request handlers. It’s similar to Jaxer, in that it is a rich interactive server-side JavaScript environment (though Persevere uses Rhino), and is accessible via JSON-RPC, meaning Persevere is great for pure Ajax front-ends. SitePen’s Kris Zyp has a large number of articles explaining how Persevere works.

Persevere uses JSON Schema for defining the structure of class instances. Stocker uses one class schema definition that looks like this:

"name":"Stock",
"extends":"Object",
"schema":{
	"prototype":{},
	"instances":{"$ref":"../Stock/"},
	"extends":{"$ref":"../Class/Object"},
	"go":function(amount){
		startUpdateStock(amount);
	}
}

As you can see, you can even include methods in the schema, which can be used for a variety of things. The method go() shown here calls the server-side method startUpdateStock(). You can access this method from the command-line through this static class:

JS> Stock.go();

In the server-side code, we start with a set of simple objects on which we base the Stock data. An example of one such object looks like this:

{ symbol:"ANDT", name:"Anduct", price:3.13 }

Note: Stocker uses more properties than this, but it is edited for clarity.

When startUpdateStock() is fired the first time, these objects are read in, randomized, and then saved as a Stock instance. When saved, the instance is written to a table and persisted. Basically it works like this: think of Stock as your SQL table, and each instance as a row of data. The columns in the table are represented by the properties that you see in the object above, such as symbol or price. And all you have to do to write to this table is:

new Stock( { symbol:"ANDT", name:"Anduct", price:3.13 } );

And modifying the data is just as easy. We loop through all of the instances in the Stock class (or table) and change the properties of the object. Persevere even automatically commits the changes for you, so this is all you need to do:

Stock.instances.forEach(function(instance){
	var m = randomizeStockData(instance);
	for(var nm in m){
		instance[nm] = m[nm];
	}
});

Notice we were able to use forEach on the instance array. We’re able to use all the functionality of Mozilla JavaScript 1.7, like array functions or getters and setters, and don’t have to worry about cross-browser issues when writing server-side JavaScript in Persevere.

Comet

In Stocker, we created an arbitrary amount of six Stock instances. Once they are created, the client-side can now fetch them and access their properties using the dojox.data.PersevereStore. Of all the data stores available in Dojo, the PersevereStore is my favorite. You pretty much declare your store with a pointer to your class, do your fetch, and you’re done:

var store = new dojox.data.PersevereStore({target:"/Stock/"});
store.fetch({query:"[?symbol='*']", onComplete: function(_items){
	console.log("fetched:", items)
}});

PersevereStore provides you with a connection to the server via Comet. When we randomized our stock data on the Persevere server, this committed the change to the database. These changes are then pushed to the browser. You can listen for these changes in the onSet event of the PersevereStore:

dojo.connect(store, "onSet", function(data){
	console.log("onSet:", data);
});

DataGrid

Now it’s time to start wiring up the user interface, and believe it or not, things get easier here. The DataGrid is largely the work of Bryan Forbes and Nathan Toone, and is a first class user interface component. We started with the HTML structure that matched the properties we wanted to display from our Stock instances:

<table dojoType="dojox.grid.DataGrid" >
    <thead>
        <tr>
            <th field="name">Stock Name</th>
            <th field="symbol">Symbol</th>
            <th field="price">Price</th>
        </tr>
    </thead>
</table>

You could also add a store and fetch attribute, but we opted to hook this up within the code:

dojo.addOnLoad(function(){
	grid.setStore(store, "[?symbol='*']");
});

The DataGrid has a built in fetch which grabs the current stock instances from Persevere, and then updates the items when the store’s onSet event is fired.

We did some CSS styling so the Grid would match our Stocker theme, and we were done with the Grid!

DataChart

The DataChart is brand new for Dojo 1.3, was created by yours truly, and was built specifically for Stocker, as it was obvious that charting was missing the ease of implementation that DataGrid provides. DojoX has an amazing vector graphics charting system written mainly by Eugene Lazutkin. It has a rich API that allows for maximum customization. However, DojoX charts didn’t support data stores, and while heavy customization is good in some cases, it can also create a barrier of entry for new developers. DataChart provides that interface between the two, all while supporting Dojo Data. I did a full write-up of DataChart in a previous post. The intent was to make DataChart as easy to setup as the DataGrid:

<div id="chartNode" style="width: 600px; height: 400px;"></div>

chart = new dojox.charting.DataChart("lines", chartDefault.chart);
chart.setStore(store, "[?symbol='*']", "price");

There is also some custom code that places chart legends inline with their respective items in the Grid. There’s also some code that handles switching between some of the chart types (just to be fancy). But other than that, we were done with charts!

BorderContainer

To get the BorderContainer to work for you, you have to understand that it supports subsets of two different designs, headline or sidebar. If you want the sidebar pane to extend from the top of the layout to the bottom, you’d naturally use sidebar. If you want the header and footer to extend the entire width, you use headline. Within either layout you have five regions: top, bottom, left, right, and center. The center pane stretches to fit, so you set the widths or heights of all panes but that one. Each region can either be a ContentPane or another BorderContainer. Stocker is slightly more complicated than standard layouts allow, so we use a nested BorderContainer.

What follows is the structure of the Stocker layout. For readability, dojoType is type, BorderContainer is BC and ContentPane is CP. Stocker uses the headline design which is the default, so it need not be not set. Its center pane is another BorderContainer with a top and center.

<div type="BC" gutters="false">                                     
    <div type="CP" region="top" splitter="false">
        Header
    </div>
    <div type="CP" gutters="true" region="left">
        Sidebar
    </div>
    <div type="BC"  gutters="false" region="center" liveSplitters="true">
        <div id="gridPane" type="CP" region="top" splitter="true">
            Grid
        </div>
        <div type="CP" region="center">
            Chart
        </div>
    </div>
    <div type="CP" region="bottom">
        Footer
    </div>
</div>

All children of a BorderContainer have an attribute region, which is the location in the design. BorderContainer’s have two other attributes: liveSplitters and gutters. Setting gutters to true puts a margin around it. The Stocker margins are handled in the CSS so these are set to false. The param liveSplitters indicates that one or more of the children will be resizable. A resizable pane acts much like the framesets of yesteryear, that had dividers between them that you could grab and drag to resize the frames.

Therefore, child panes can have the attribute splitter=true, which allows the user to resize it. It’s not obvious which child panes should get this attribute, since the splitter is shared amongst two panes. The trick to knowing is that the center pane, since it’s always stretchy, never gets this attribute—it is always applied to the pane that shares the splitter with it.

The DataGrid and DataChart were inserted in their proper ContentPane containers, the buttons for chart switching were placed in the sidebar, and the header and footer content was implemented. We launched the Persevere server, opened Stocker in our browser and watched the updates!

Conclusion

Stocker was created to show the power of Dojo and some of it’s more advanced components, and how easily they can be implemented in your site or web application. We’ll also be talking about Stocker and Dojo in a number of upcoming conference talks this year. If you’re interested in learning more about Stocker and how to put it together, drop us a line!

SitePen has workshops planned in various locations around world, and we’d love it if you could join us. You’ll learn everything you need to build Stocker and more at the following one-day Intro to Dojo, Charts, Grids, and Comet workshops:

• Sydney, Australia, April 23, 2009
• London, England, May 13, 2009
• Stockholm, Sweden, May 26, 2009
• Milan, Italy, June 8, 2009
• Paris, France, June 10, 2009

Or, join us for a special two-day Dojo Workshop in Munich, Germany, May 7-8, 2009 with the uxebu team.

Cross-Window Communication: the OAuth Handshake

One of the most challenging issues we needed to solve was how to do the OAuth handshake with Netflix. As a quick review, the handshake is a process in which the user is directed to a web page on the application’s server—where they can enter their authentication information and receive a token in response:

The Problem

Adobe has done a great job in melding the web development paradigm with the desktop application paradigm; among the things that they’ve done is to apply some of the same sandbox security principles to any window created that applies to most browsers. Unfortunately for us, part of this sandbox applies to any window that loads an HTML page that does not originate from the same domain—which meant that when we load the authorization page from Netflix’s servers, we no longer had any kind of script or event access to that window. This includes access to things like the window.onload event…which meant that we had no real way of knowing what happens within the window.

Unfortunately, the OAuth handshake is designed such that the user is normally redirected to a web page on the originating server, with a parameter passed to it that the originating server then uses to redirect the user back with an access token.

The Solution

What we discovered was that we had access to the created window’s window.location property and the onClose event of the window object we used, so in the end my friend and colleague Mike Wilcox simply polled the open window for a change in URL. Here’s the code in question (abbreviated):

var seenOnce = false;
var v = setInterval(function(){
	var wurl = win1._window.location;
	if(wurl != url){
		if(!seenOnce && wurl=="https://api-user.netflix.com/oauth/login"){
			seenOnce = true;
			return;
		} 
		else if(wurl=="http://www.netflix.com/TermsOfUse"){
			return;
		}
		else if(wurl.indexOf("Failed")>0){
			return;
		}
		clearInterval(v);
		v = null;
		win1.close();
	}
}, 1000);
var c2 = dojo.connect(win1, "onClose", function(){
	if(v){
		dfd.errback("user");
		clearInterval(v);
		dojo.disconnect(c2);
		return;
	}
 
	//	we're good to go, so go get the access token
	dojo.xhrGet(dojox.io.OAuth.sign("GET", {
		url: "http://api.netflix.com/oauth/access_token",
		handleAs: "text",
		error: function(err, ioArgs){
			dfd.errback("auth");
		},
		load: function(response, ioArgs){
			var a = response.split("&"), o = {};
			dojo.forEach(a, function(item){
				var p = item.split("=");
				o[p[0]] = unescape(p[1]);
			});
			qd.app.authorize(o.oauth_token,
				 o.oauth_token_secret, o.user_id);
			dfd.callback(o.user_id);
		}
	}, token), false);
});

What the above code does is poll for a change in the URL of the opened window, and acts upon that change. If the boolean seenOnce ends up as true, we know that the user has authorized Queued for access to their Netflix information, and can finalize the OAuth handshake by getting the actual access token using dojo.xhrGet and dojox.io.OAuth.

dojo.connect vs. AIR’s EventListener

Another problem we ran across was the fact that all of the AIR objects made available to you during development were essentially incompatible with Dojo’s signals and slots workhorse, dojo.connect. For those who don’t know Dojo all that well, dojo.connect is an ingenious way of attaching functions to other functions. It does this by swapping out the original function with another one, that will execute a queue of connected functions that have been applied by connect.

The Problem

The issue here is that dojo.connect assumes that the object being connected to is a JavaScript Function object; however, methods on AIR objects are not true JavaScript Function objects at all. This basically means that the following code example will not compile with AIR:

dojo.connect(air.NativeApplication.nativeApplication, air.Event.USER_IDLE, 
	function(evt){
		doSomething();
});

Coming from a pure Ajax development environment, this can throw anyone for a curve.

The solution

The workaround for this was the following approach:

1. Create a private function
2. Create a function that is attached to the wrapping JavaScript object
3. Call the private function with another function, passed to the native AIR object’s addEventListener method.

Using the example from above (and assuming this is within an object), here’s the code:

var self = this;
 
// The private function
function onIdle(evt){
    self.onIdle(evt);
}
 
// The public function
this.onIdle = function(evt){
    // this is an event stub that can be connected to by
    // other objects.
};
 
// attach the private function to the AIR object
air.NativeApplication.nativeApplication.addEventListener(air.Event.USER_IDLE, 
	onIdle);

The pattern is a little verbose, but now we can do something like this, elsewhere in the application:

dojo.connect(qd.app, "onIdle", function(evt){
    // and now we can make use of the app's onIdle event
    // from anywhere else in Queued.
});

Once we figured out this pattern, we were able to take full advantage of events fired off by AIR’s native objects using the familiar dojo.connect paradigm.

AIR’s New Encrypted Database and the Asynchronous Connection

One of the exciting new features included with AIR 1.5 is the new encrypted SQL database, based on the SQLite engine. We use this database to cache all of the Netflix title information that has been viewed, as well as the user’s queue information and the transaction queue (used for synchronization when returning from offline mode). One of the neatest things about the new engine is that you can open either a synchronous or an asynchronous connection to the database.

The advantage of using an asynchronous connection is that all actions performed against the database are executed on a separate thread. This means that doing some major SQL will not interrupt the thread that the user interface is run on—resulting in a faster-performing application.

The Problem

Aside from the complexity this introduces in code (see the qd.services.data object), we discovered something else—on faster machines, throwing a large number of SQL statements at the database in a short period of time would lock the database file.

This was particularly problematic when trying to store all of the Netflix titles that might exist in someone’s queue. As an example, in my own account I have rented over 350 titles over the lifetime of my membership (I’ve been a Netflix member since 2003); when viewing my rental history with Queued for the first time, Queued attempts to store the information for all 350+ titles so that I can view that information offline if I want to.

The Solution

Since we were able to reproduce this issue consistently (some of my colleagues are avid movie watchers and have rented a lot of titles), the solution was to implement an internal queue of our own (oh, the irony!) within the qd.services.data. We could do this because we wrapped AIR’s database access with our own, to simplify the interface and to allow for other parts of the application to connect to various events fired by qd.services.data. Here’s the basic code:

var queue = [];
function exec(){
    var o = queue.shift();
    if(o){
        o.deferred.addCallback(exec);
        o.deferred.addErrback(exec);
        o.statement.execute();
    }
}

(You can see the full code here, starting at line 238.)

In Queued, every query sent to the database is prepared with a dojo.Deferred object; this deferred is the primary callback mechanism used throughout Queued to handle async requests. What the above code does is search the internal queue for the next statement—and if it finds it, attaches the private exec function to the statement’s callback chain, ensuring that it will only execute the next statement when the previous one has completed.

By adding in this internal queue, we solved the database locking issues.

Storing the Password for the Encrypted Database

The encrypted database included with AIR requires the use of a password for opening it; you set the password when you create the database. Because the database is encrypted using the AES cipher, the length of the password is important; it must be exactly 16 bytes long.

Asking a user to create this password and use it to login to Queued each time the application was run seemed a bit heavy-handed to us (especially if we needed to start padding that password to meet the 16 byte requirement), so we needed to come up with a way of dealing with this without user intervention.

The solution

To solve this issue, we generate a random password the first time Queued is run, like so:

var tab = "ABCDEFGHIJKLMNOPQRSTUVWXYZ!@#$%^&*" + 
     "~?0123456789-_abcdefghijklmnopqrstuvwxyz",
     key = "";
for(var i=0; i<16; i++){
    key += tab.charAt(Math.round(Math.random()*tab.length));
}

This is not particularly robust by crypto standards, but we felt it was more than sufficient for Queued.

From there, we took full advantage of the Encrypted Local Storage (or ELS), also included with AIR. This local storage is different from the encrypted database; it’s much smaller and cannot handle large amounts of data (I’ll talk about this in Part II).

But it is perfect for storing a password to a different data store. So we pop this password into the storage once it’s generated. Here’s the full code (run onLoad):

qd.services.init = function(){
    qd.app.splash("Getting database password");
    pwd = storage.item(dbProp);
    if(!pwd){
        qd.app.splash("Generating database password");
        var tab = "ABCDEFGHIJKLMNOPQRSTUVWXYZ!@#$%^&*" + 
            "~?0123456789-_abcdefghijklmnopqrstuvwxyz",
            key = "";
        for(var i=0; i<16; i++){
            key += tab.charAt(Math.round(Math.random()*tab.length));
        }
        pwd = storage.item(dbProp, key);
        qd.app.splash("Password generated (" + pwd.length + ")");
    }
 
    qd.app.splash("Initializing network monitor");
    qd.services.network.start();
    qd.app.splash("Initializing database services");
    qd.services.data.init(pwd, db, qd.services._forceCreate);
};

The above code looks to the ELS for an existing password, and if it doesn’t find it it will create it and force the database services to create the database. In this way, we’re able to take full advantage of AIR’s encrypted storage facilities without forcing user interaction.

Other issues

In part II of this post, I’ll talk about five other issues we ran across during the course of Queued’s development—the application sandbox, initial window placement, the ELS and performance, issues with capturing and handling the application’s exit, and the loading of icons for the tray/dock.

After chatting with the team for a while and looking at their use of Deferreds, I noticed a few references and grumblings to Deferred’s silent swallowing of errors. There was even a commented out section of code containing a hacked Deferred to avoid the issue. I gathered from discussion that an error occurring NOT in the result from an XHR, but instead in its callback handler was being squelched. After discussing this issue and realizing that there was a misunderstanding of how the callback chain worked, I decided to focus on describing the callback chain and identifying any potential bugs.

Callback chains created with dojo.Deferred objects provide a standardized interface for wiring an arbitrary set of actions occurring over an unknown period of time. While they are very simple to use in most common cases, the callback and errback chain is often misunderstood, and the chaining together of Deferreds themselves even more so.   A decade or so ago, I remember wanting to implement something with pthreads to learn about the challenges I had read in using them.  I purchased a book, Programming with POSIX Threads by Larry Butenhof, and sat down to read it.  Unfortunately, I was on vacation, with no computer, and it was before I had a laptop surgically attached to my body.   So as I read, I didn’t actually do any of the examples or really understand how it all worked.  A while passed before I got to work on that again, and so I decided to read the book again this time.  For some reason, I couldn’t grasp some of the aspects around locking.  I thought and thought and thought and then I thought a couple more times for good measure.  All the authorities I had read on the subject described it as challenging.  I couldn’t figure it out just from reading this book…it truly must be hard.   As I’ve found to be the case with most things I once thought to be complicated in software development, they are often actually quite easy.

I sat down and wrote a couple of small programs, and within a very short period of time it just clicked.    It now seems obvious how they work and I can’t even seem to comprehend or describe why I had problems understanding it previously.   For new Dojo developers, Deferreds are a simple callback mechanism and it is pretty easy to comprehend.   When the applications become a little bit more complex or when developers try to figure out how to do a more complex behavior with their Deferreds, they then realize that there is more to understand.   Often they end up walking in my pthreads shoes or bypassing the situation altogether.

The common perception of dojo.Deferreds is that there are two simple chains, one for callbacks and one for errbacks, and these all get called in succession upon being triggered, much the same way that you might add a function for page launch in dojo.addOnLoad.  This in turn leads people to believe that dojo.addCallback(cb); dojo.addErrback(eb); is the same thing as dojo.addCallbacks(cb,eb).  This is NOT the case as it would be if the chain were simply two queues as is the general impression. Deferred objects in Dojo, which are directly modeled on Twisted Python’s Deferred object, contain a single queue (chain) and items are added on to the chain in tuples containing a callback and an errback.  When the deferred is triggered, it will proceed down the chain, possibly changing from “callback” to “errback” as necessary.

If callbacks are added on to the chain individually, their counterpart is filled in with null.  For example:

var def = new dojo.Deferred();
def.addCallback(cb);
def.addErrback(eb);

will result in [cb, null], [null, eb] ] being placed onto the callback chain.   On the other hand if we add to the callback chain using addCallbacks():

var def = new dojo.Deferred();
def.addCallbacks(cb, eb)

The our resultant chain is a single item added to the stack, [cb, eb].  As you’ll notice in the diagram above, the red dotted lines moving from a callback to an errback go to the next item in the callback chain, not the peered item in the callback chain .  If a callback throws or returns an error, it will go to the next errback handler in the chain. Likewise, if an error handler returns something other than an Error object, that output will move back to the callback side of the chain. Notably,  you can continue to add onto the chain even after it has fired.  This means that when you get to the end of the chain, it just holds.  If the last callback in the chain throws an error, the Deferred will hold onto this error waiting for another errback handler to be added to the chain.  It is this last item that makes it appear to “squelch” errors that the Queued team was seeing.  However, inspection of the Deferred will show it to be sitting in an error state. Deferreds are a one-time event, but the callback chain can be added onto indefinitely.

Let’s take a look at some examples.

var cb = function(msg, d){
       console.log(msg, ": ", d);
};
 
var def = dojo.xhrGet({
         url: "test.txt",
         load: dojo.partial(cb, "inline callback"),
         error: dojo.partial(cb, "inline errback")
});

This is a ‘normal’ use case for the most part.  An error in the request (404 for example) will trigger the error callback. Run the example.

In the following example, instead of getting an error from the IO request, we’ll use a callback with an intentional error in it, which generates an error.

var badCb = function(msg, d){
     console.log(msg, ": ", d);
     badvar.append("foo");
};
 
var def = dojo.xhrGet({
     url: "test.txt",
     load: dojo.partial(badCb, "BadCallback")
});

Note there is no “error” handler.  The ‘badCb’ callback function will run and generate an error but it will be seemingly squelched.  As mentioned above, it is not actually squelched per se, but instead just waiting to be passed on to anything added onto the chain at a later point in time. Run the example.

Taking this same code, but including an error handler, our error handler will catch this error to do with it as necessary (Example). When using the inline callbacks, they are appended to the Deferred chain in the form [loadFunc, null], [null, errorFunction].  That is, if an error occurs either as a result of the I/O operation or in the load callback, the error function will be called.   This is fine in many cases and can be exactly what you want.  With a long callback chain, however, and no error handlers or poor planning of how those error handlers are attached, it can seem as if the error was squelched.  More control can be had by adding the callbacks after creating the Deferred:

var cb = function(msg, d){
        console.log(msg, ": ", d);
};
 
var def = dojo.xhrGet({
         url: "/badUrl"
});
 
def.addCallbacks(dojo.partial(cb, "[CB,eb]"), dojo.partial(cb, "[cb,EB]"));

Our chain ends up with only a single tuple: [cb, eb]. If an error is returned from the request itself, it will get handled by the errback handler.   On the other hand if the request is successful, but there is an error generated in the callback, it will again be seemingly (but not really!) squelched.

 var cb = function(msg, d){
       console.log(msg, ": ", d);
 };
 
 var badCb = function(msg, d){
       console.log(msg, ": ", d);
       badvar.append("foo");
 };
 
 var def =dojo.xhrGet({
      url: "test.txt"
 });
 
 def.addCallbacks(dojo.partial(badCb, "[CB,eb]"), dojo.partial(cb, "[cb,EB]"));

The error is waiting for another errback handler to be added to the chain. Run the example.

As you can see, this provides insertion points so you can trigger errors and handle them in a variety of different ways.  Remember that Deferreds aren’t necessarily associated with an I/O operation.  They are simply a promise that something will happen in the future.  Just because that error happens, it doesn’t necessarily mean it is the end of the road as not all errors are fatal.   All of this can be summed up by simply stating that you have to think about how you add items on to the chain so you can insert at the desired point.   You are either adding [cb, null], [null, eb], or [cb, eb] .

What about a simple example of how switching states and all of this chaining can be useful. Let’s say you are going to request a resource that may or may not exist, and if it doesn’t you want to continue using a default value. If the errback handler is added with addErrback before the callback, this can easily be achieved. The following example ends up with [null, eb], [cb, null] as its callback stack, so that when the ‘eb’ function returns an onError, the deferred will then send that value to the normal callback function.

var eb = function(err){
    return /* default object */  {}
}
 
var cb = function(data){
    /* do something with data */ 
}
 
var def = dojo.xhrGet({url: "/fubar"});
 
// add the errorback function first, [null, eb], 
// so that if the xhr has an error it can catch it 
// and still be able to continue to the normal chain
def.addErrback(eb);
 
// this chain will process either a successful return 
// from xhrGet OR the result of the errback handler
def.addCallback(cb);

Now you may be thinking that this implies you have to always end with an errback or the deferred will seem to squelch the error.  This makes it difficult to add an arbitrary number of callback items and ensure an errback is added last to catch any problems.   It is the understanding of how Deferreds themselves can be chained together that puts everything into perspective.

Dojo developers often think of Deferreds at the level of I/O, but they are much more useful than that when well thought out.  Let’s create an example.   On our page we want to have a simple startup system  that does a number of different startup tasks.  For our init method, we want to initialize the communication system, and then show the user interface.  However, the initComms() section can be broken down into getting the data, processing the data, and then sending off a message.  By chaining deferred objects together (as opposed to adding a function onto a particular Deferred’s callback chain), complex procedures can be broken down into simple tasks, and error handling can be handled at any level or bubble up to the outermost Deferred.  To chain two Deferreds together, simply add a function that will return a Deferred to another Deferred’s callback chain!  For example:

var  def = new dojo.Deferred();
def.addCallback(function(){
     return new dojo.Deferred();
});

In this example, we are adding a function that returns a Deferred object.  The outer Deferred will trigger the launch of the inner Deferred’s process, and will block any more callbacks until the inner has exhausted is entire callback chain. Diving right into the details, here is an example.

// Init comms function.  This will call getData() which
// returns a deferred representing the process of retrieving data
// and processing it.  After getData() is complete, 
// send a message to the server, and then log completed.
 
initComms = function(){
       //getData returns a deferred
       var initDeferred = getData();
 
       // add sendMessage to the chain.
       initDeferred.addCallback(sendMessage);
 
       // log this as completed
       initDeferred.addCallback(function(){
               console.log("initComms Completed");
       });
 
       return initDeferred;
 }
 
 getData = function() {
        // Request data from the server
        console.log("getData()");
        var def = dojo.xhrGet({
               url: "test.txt"
        });
 
        // process it in some fashion, or if there was an xhr error
        // throw the error handler
        def.addCallback(processData);
 
        // log that we are finished
        def.addCallback(function(){
              console.log("Get Data Complete");
        });
 
        //return the deferred from the initial xhr
        return def;
}
 
// dummy process data func
processData = function(d){
        /* do something */
        console.log("Process Data: ", d);
        return d; /* return processed data */
}
 
// dummy send message func
sendMessage = function(msg){
        dojo.publish("/msg", [msg]);
        console.log("Sent Message");
        return msg;
}
 
// showUi creates a new dojo.deferred, adds a log 
// message callback, and then returns
// The setTimeout() call is used to represent the 
// work that might go on in this function
showUi = function(){
        console.log("showUi");
        var def = new dojo.Deferred();
        def.addCallback(function(){
               console.log("showUI completed");
        });
 
        // for demo, trigger the showUi work with a trigger
        setTimeout(function(){def.callback(true);}, 50);
        return def;
}
 
// "Main"
 
// call initComms and get a deferred back for the whole process
var initDef = initComms();
 
// add showUI to the callback chain.  showUI returns a deferred object, whose
// process will be kicked off here
 
initDef.addCallback(showUi);
 
// this will not be executed until the Deferred returned from showUi 
// and all its callbacks have been completed.  We didn't add any 
// error handling in the individual functions, though we could have,
// and instead let the error bubble up to the outermost Deferred.
 
initDef.addCallbacks(
        function(){
              console.log("READY");
        },
        function(err){
              console.warn("ERROR Starting APP: ", err);
        }    
);

Deferreds can be both powerful and simple.  After having gone through the exercise of testing each of these scenarios and making sure that they fit the original Twisted specification, I do not believe that there is a bug here, only a misunderstanding of the errback chain.  I believe that Deferreds provide a powerful and elegant way to work with asynchronous events, but at the same time they can be difficult to follow.  

While I personally prefer using the Deferreds, many people don’t use this power and in some cases are befuddled by its behavior. Some are of the opinion that they are too complicated and shouldn’t be used for things like XHRs in Dojo going forward. What does the Dojo developer community think about the future of Deferreds for Dojo 2.x? Should we be looking at getting rid of Deferreds in Dojo? Should we simplify this into two simple queues?  What’s your opinion?

Deciding on Functionality

Certain requirements were set in stone from the beginning. We had to use AIR’s database. We had to do an offline mode. We had to support a mode where Queued could run in the background. We had to demonstrate a tight Dojo build. We had to support drag and drop for queue re-ordering—stuff like that. Other things were optional; for example, we could demonstrate the Encrypted Local Store (ELS), but it wasn’t a blocker. After that, though, it was pretty wide open, so we had to decide how much to bite off and still be able to meet our deadlines.

To Be, or Not to Be (Multi-User)?

We did a lot of work around the user model. Should we support multiple accounts? That seems like an obvious feature to include, and as web application developers, we work with user authentication all the time. So, we prototyped a model where the application would store separate user credentials in the ELS and use those to authenticate with Netflix. Here are a few screenshots of the designs we went through:

And after the Less Red Redesign that Torrey talked about in a previous post:

Integrating the Pieces

Quite a few of Queued’s internals were still in the prototype stage at this point, and right about here was where the OAuth code started getting put in place. This presented us with a challenge. As a team, we’re so used to the web application authentication model—where authentication happens on some remote server—that we were automatically falling into that paradigm almost out of habit. It’s just the way our projects tend to work. However, Queued is essentially a desktop application, so that model doesn’t necessarily fit.

We had originally assumed we’d be able to store sets of login credentials in the ELS and use those at application startup to authenticate with the Netflix API and get rolling. However, Netflix uses the OAuth protocol, and OAuth fundamentally doesn’t work that way.

Traditional username/password-based authentication is like showing your ID to get into, say, a bar. You come and go as you please, but if you leave for long enough, be prepared to flash your ID again to get back in. OAuth, on the other hand, is like having your hand stamped when you enter an amusement park: if you leave, the stamp gets you back in. In this case, the stamp is a token you use to sign your API calls, and Queued gets this token after redirecting you to netflix.com so you can authorize the application to access your account. So, Queued never knows your Netflix login credentials at any time.. This is how OAuth is designed, and it has its benefits, but it didn’t really work with our defined interaction/login model.

Decision Time

We had to make a difficult decision: cut multi-user capability for the 1.0 release. We feel this decision made sense, not only because of the deadline constraints, but because by default, AIR provides different databases and ELS for each operating system user, and we were building a desktop application, after all. We decided to follow that model, and since that’s what AIR expects, the entire toolchain would inherently support that mode of operation.

In the final 1.0 release, therefore, you won’t find any login screens, but you will find a button that kicks off the OAuth handshake:

It was tough to say goodbye to multi-user capability, but in the end, it let us focus on our core requirements, so in retrospect, it was probably the right decision to make under the circumstances. Queued is Open Source, of course, so don’t feel like you have to wait on us if you want that implemented; aside from the necessary UI work, Queued’s internals are mostly set up so they don’t care whether the app is single-user or multi-user. Have at it :-)

That’s one example of the kind of iterative process we use. Let’s talk architecture.

Deciding on Architecture

Getting a new Dojo-based project started can sometimes be daunting, precisely because it doesn’t force a particular structure on you. The field is wide open. Our first task was to start putting dAIR-based tests together to figure out how we should handle things like window initialization and database access. While that was happening, we had to figure out how to organize our assets and divide up responsibility within the team. We landed on a few key things—nothing really new, but certainly, choices that we could have easily taken in a different direction.

Do as much as possible in CSS.

Dojo’s widget toolkit, Dijit, is robust. You get i18n & a11y out of the box, for example. However, those were outside the scope of this project, which was to be a small fast technology demo. One specific requirement was to keep the built .air package as small as possible, so we made the choice to cut everything that didn’t directly contribute to our project requirements. Since we were dealing with a single WebKit-based engine and didn’t need all of the great cross-engine compatibility stuff in Dijit, and since CSS is generally faster than JavaScript for just about anything that CSS can do, our simple rule was that if you can do something in CSS, you do it in CSS. Even if it took JavaScript code to trigger a change, we pushed as much of the change as possible into the stylesheet. This means, for example, that we created a lot of CSS classes to define state (stuff like “.movie.inQueue”) rather than changing node styles individually.

If you read through the code in the repository, you’ll certainly find some Dijit, but it’s mostly restricted to the initial interface layout (BorderContainer and friends).

Stick with Unobtrusive JavaScript.

Most people advocate the concept of Unobtrusive JavaScript, for a variety of reasons. For Queued, we didn’t have to worry about SEO, making sure our tags were semantic, or any of the big philosophical ideas of Correctness. In an app like this, HTML simply acts as a UI spec. What it came down to for us was separation of design and behavior. It allowed our CSS pros to work their magic on as sparse a DOM as we could provide, without distraction by things like inline event handlers. There was probably a tiny performance benefit to not including the Dojo parser and working through the DOM looking for dojoType attributes and instantiating widgets in markup, but that’s just a nice side effect.

Initializing widgets in this style usually means running some code with dojo.addOnLoad, selecting nodes with dojo.query or dojo.byId, and attaching event handlers to them with dojo.connect. The dojo.behavior module is a great way to do this en masse. We ended up structuring our application code where most modules followed a pattern like this:

dojo.provide("qd.app.someModule");
 
dojo.require("dojo.behavior");
dojo.require("qd.app");
 
qd.app.someModule = new (function(){
	// private variables
	var foo = null,
	    bar = null,
	    thinger = null;
 
	function privateMethod(){
		// code here
	}
 
	this.publicMethod = function(){
		// code here
	};
 
	// this module's behaviors
	dojo.behavior.add({
		".someSelector .someOtherOne":{
			onmouseover:function(e){
				// code here
			},
			onclick:function(e){
				// code here
			}
		},
		"#oneParticularNode .childNode img.thinger":{
			onclick:function(e){
				// code here
			}
		}
	});
})();

To use this code in the application, we simply do dojo.require("qd.app.someModule") in the initialization code; the closure gets put together immediately, and the module gets itself all set up.

Being closure-bound like this, each module now had “private” and “public” members, and we could use dojo.behavior in each module to provide that module’s behaviors. Used this way, dojo.behavior will do a bunch of dojo.query/dojo.connect sequences for us; the syntax is more direct than doing them all manually, so it makes for more readable code.

JSLint doesn’t particularly like the someModule = new (function(){ ... })() idiom, however. It always complains, “weird construction. Delete ‘new’”, but it’s a perfectly valid way to create singleton modules, so we stuck with it.

As an example, you may want to look at tooltip.js, which is the module that creates the movie hover info toolip. It’s fully self-contained, without a single public member. Also, try ratings.js, which builds the star rating widgets. Those are good examples of the code structure I’m talking about.

Abstract functionality into internal services.

Finally, even though there isn’t necessarily the concept of a server-side and a client-side in this kind of application, we decided to organize certain things as though there was, by creating various “services” to act as data sources to the application. They’re the middle layer between the application and AIR system-level stuff like the database or ELS or external data like the Netflix API. For example, we created a service to parse Netflix’s XML, a service to handle authentication, and a few others. To the rest of the application, those act as an internal API that abstracts the lower-level details away and keeps the app code focused on the interaction of our objects.

The biggest place where this structure helped was in creating the offline mode. One of our modules continually monitors network availability and stores the status in qd.services.network.available. The rest of the application will automatically toggle between online and offline modes because of this simple bit of code:

qd.__defineGetter__("service", function(){
	//	summary:
	//		Return the proper service (qd.services.online, qd.services.offline)
	//		based on the current network status.
	var b = qd.services.network.available;
	return b ? qd.services.online : qd.services.offline;	//	Object
});

That’s it! This getter transparently (to the caller) switches between the Netflix API and the local database cache as network availability varies, so calling qd.service.titles.find(...), for example, will search for titles using the best available source automatically. The calling code doesn’t need to figure that out for itself.

Check it Out

Queued lives in SitePen Labs, so check that out if you haven’t already. The latest code is hosted at Google Code. Check back soon for more in our Queued series.