The Dojo tree widget supports lazy loading, but it typically connects to a Dojo data store for all its data through a model-store adapter. Consequently, the tree itself does not define the actual data loading mechanism, as that is up to the store. The tree merely requests data from the store as needed. However, the JsonRestStore fully supports lazy loading with a well defined mechanism for retrieving deferred data from the server. When the JsonRestStore is used with the tree widget, it makes it possible to lazy load nodes in the tree as nodes are expanded. JsonRestStore supports lazy loading by using JSON referencing, whereby items can be referenced from other properties and arrays and the full item can later be retrieved when needed (with loadItem).

The model-store adapter now in Dojo 1.4 and later supports an option for a loading mechanism that facilitates single requests per node expansion. Previously (in 1.3), when a node was expanded the tree would request all the children and if any of the children had not been loaded, the tree would request that each child’s data item be fully loaded (this is done to ensure that each child has a label and information about whether or not it has children so the child nodes can be properly rendered) resulting in a request for each child of the expanded node, which is clearly inefficient. However, now when the model’s deferItemLoadingUntilExpand property is set to true, the tree will not attempt to load all of the children, but rather will only load the item for the expanded node if needed. This allows us to leverage the JsonRestStore’s support for partially loaded items. When a node is expanded and the item is loaded, the server can provide a JSON representation of the item which includes references to the children, and a partial set of properties for each child, which can include the label and children information for proper rendering. Each of these children can then be fully loaded when they are expanded (with their partial children objects). With this strategy, each node expansion will rely on only a single HTTP request.

Let’s take a look at how to build a lazy loaded tree with the JsonRestStore. We begin by creating a store:

Now, we will create the model adapter for the tree to access the store. This is where we use the new deferItemLoadingUntilExpand property:

Next, we can create a tree:

Now, we can build our data that is supplied from the server. The first request that the JsonRestStore will make to the server will be a request for the top level nodes (the children of the root). The tree will make the initial request using the query provided to the model, and the JsonRestStore will combine that with the target. In this case, the request will be made to the path “tree/root”. Leveraging the partial loading support in JsonRestStore, we can serialize references to each of the items that will be children of the root node in the response, and only include the properties necessary to render these nodes (label and children info):

This provides sufficient information to render the top level of the tree, as well as the link information for retrieving the full representation of each item. We don’t need to actually include the children, just the presence of a children property will indicate to the Tree that the node is expandable and an expansion icon will be included. Now when a user clicks on one of these nodes, the tree will ask the JsonRestStore to load the item and the JsonRestStore will request the resource for the URI specified by the $ref property. In this case, it will request node1. This URI is interpreted relative to the target URI of the store, so in this case, the JsonRestStore will request “tree/node1″. Your server can then respond:

Here we see the full representation of node1. This not only includes the name, but may include additional properties that are used in other application logic. We also include an array that lists all the children of this item. Once again we use partial representations of these children to minimize the data transferred over the wire to the client. Whenever any of these children are expanded the process will be repeated and another request will be made to the server for the full representation of that child.

The example code for lazy loaded trees is available in Dojo at /dijit/tests/Tree_with_JRS.html.

If you are updating data in the store, you should be aware of one more tip when using the Tree with the JsonRestStore. By default, the ForestStoreModel adapter will re-query the top nodes on every onNew notification event and every onDelete event that involves a top level item. This can result in queries to the server even though the server has not yet been sent all changes. This makes top level additions essentially disappear when the re-query takes place. You may need to override the _onNewItem and _onDeleteItem to provide your own logic about where new and deleted items should be placed in the hierarchy.

Another powerful feature of the referencing capabilities of JsonRestStore is that individual items can be referenced from multiple parent items. Consequently an item could exist in multiple places in the tree, under different parents.

Together, the Tree and the JsonRestStore provide a powerful combination for lazy loading data that allows for large extensive hierarchical data to be displayed without large upfront data transfers. The JsonRestStore’s partial loading support can be leveraged so that we can perform lazy loading with a single request per expansion.

Related posts:

1. Effective use of JsonRestStore: Referencing, Lazy Loading, and more
2. More Query and Caching Power for Data Stores: ClientFilter, JsonQuery, and JsonQueryRestStore
3. RESTful JSON + Dojo Data

Pintura-based applications normally consist of server-side data models with three layers: data stores, store models, and model facets. On top of this, different representation handlers (for serializing data to different formats) can be defined, but Pintura comes with a good set of these ( including JSON, JavaScript, multipart, and Atom), so usually that is not necessary. This provides a well-structured separation of concerns, distinguishing storage configuration (data stores), core data logic (models), varying capabilities of access to the data (facets), and data serialization (representations). Perhaps the easiest way to understand this approach to take a look at an example application.

Naturally we need to install Pintura first. Pintura is JavaScript based and can run on various CommonJS JavaScript servers including Node and Narwhal/Jack, and the various engines they can run on. The Pintura installation process is described here.

A basic Pintura application is very easy to create. If we start by making a copy of /template/, we can create an application with a server-side model class by simply adding this to the lib/app.js:

Once we start up our server for this application, we now have the full Persevere HTTP/REST interface for creating, querying, updating, and deleting objects from the MyTable model. We have everything we need on the server to write a web client application, which can utilize REST methods like GET, PUT, POST, and DELETE, (as well as JSON-RPC) to interact with the server model class to store and query data. We get an enormous amount of functionality for free, including cross-site capabilities (JSONP, window.name, and CORS support), cross-site request forgery protection, authentication, conditional request handling, and more.

However, most applications will require more server-side logic than simply a generic data store. Generally, application code will be written as model and facet definitions using Pintura’s persistence framework (Perstore). With this framework, we can define handlers for different aspects of the object’s interaction and lifecycle, such as get, put, and delete handlers. We can define data integrity contracts by adding JSON schema constraints in basically the same way as in Persevere 1.0 (except our model classes are defined with the Model constructor from the model module rather than a top-level Class constructor, to be more inline with CommonJS and standard web framework terminology). We can define additional methods for the model as well as methods for the persisted object instances (by defining methods on the model prototype), once again using the same structure as the first version of Persevere. For example, we could easily add a new method on object instances:

Pintura comes with an example wiki, we will examine how it is constructed to see how we can further understand what it looks like to build an application with Pintura. The example wiki is found in the example folder in the Pintura distribution, and we will break out the various parts and how they fit together.

We will start by looking at the app.js file found in the example/lib directory. This is the entry module which loads the other modules, starting from a user perspective, first defining the application-specific resource representations and user access levels. The example/lib/access.js module defines a set of facets for accessing data, for the different sets of users. For each user, a set of facets is returned and the facet provides the capability through which the user can access the application data. We can then drill down into example/lib/facet/page.js module, which defines how the underlying model class can be accessed through each facet. The example/lib/model/page.js defines the core application logic which in turn accesses the data store that provides the low-level interaction with the underlying storage medium.

Let’s look more closely at each of the levels, starting at the bottom with the store. A data store is the basic entity which allows for RESTful interaction with a storage system. Storage systems can be relational SQL-based databases, document based databases, OS files, remote servers, and so on. The object store API follows the W3C Indexed Database API for an object store. Pintura’s Perstore framework comes with a variety of data store implementations, including a SQL datastore for Rhino, and a JavaScript/JSON file-based data store. Creating your own store is not difficult, it essentially involves implementing get(id), put(object, id), query(query), and delete(id) functions (and optionally create, transaction, and subscribe for additional functionality). In our example application, we don’t define any special stores, we will simply use the default store, which out of the box is the js-file module, an object store which saves records into a JavaScript/JSON file.

The object store API is not only used at the bottom storage level in Pintura, but the model class and facets also follow the same API for consistency; classes and facets are simply wrappers that expose the same store API, adding their own functionality and/or constraining operations. Model classes act as the gateway to stores, providing the central core logic of the application. Any application data logic that should be enforced for all users should be implemented here. Models do not need to implement the entire store API themselves. Any store method that is implemented simply passes through the underlying object store. An empty class is perfectly valid, and would result in all REST actions being passed through to the object store. In example/lib/model/page.js, we can see the put handler is implemented to add a lastModifiedBy property and create a new PageChange instance:

We can also define the prototype for this class, which allows us to create methods that are available on all instances of the class.

At the next layer up, we have the set of facets used to access the Page class. Facets are defined in a similar way to model classes, they follow the store API, wrapping a class, and only methods with specific logic need to be implemented. There are two primary types of facets, restrictive and permissive facets. These differ in how they handle unimplemented methods. A permissive facet will default to passing all actions through to the underlying class. A restrictive facet defaults to only passing retrieval actions (get and query) through, methods have to be explicitly implemented to interact with the class.

In example/lib/facet/page.js we see three facets defined to accommodate different groups of users. The public facet is defined to be restrictive, essentially making the objects read-only for public users. The public facet also defines the query method so that only published pages are accessible. The user facet and admin facet are permissive, allowing all actions on the pages. All of these facets control access to the model class, and in fact, model classes are simply a special permissive facet that applies to data stores.

Continuing back up the stack, the access module can now be understood as the initial logic that determines which facets are available for the authenticated user, so their access is properly controlled.

The initial example/lib/app.js file also defines the custom media module used for generating the HTML for pages. Pintura comes with a good selection of media handlers out of the box; the JavaScript and JSON handlers are the primary use case handlers, allowing objects to easily be accessed through XHR by Ajax applications. Pintura is designed first and foremost for Ajax applications, so normally these will be used most frequently. However, we need a specific HTML handler in this case so that pages can be directly generated as HTML for easy indexing by search engines.

Media handlers are simple to implement. In the example/lib/media/wiki-html.js handler we can see that we just need to define the media type, the quality of the media type (for content negotiation), and the serializer. Bi-direction media handlers can also implement a deserialize method (for handling PUT and POST requests), but this isn’t needed for our Wiki, all updates to pages will be done through JSON-based XHR requests. The Wiki HTML handler simply uses the the wiky package to do the transformation of markup to HTML for us.

By default, Pintura does all the HTTP request handling for you, handling cross-site issues, authentication, transaction management, and converting requests to calls to facets and model classes. However, Pintura is highly modular, actually consisting of about 13 modular middleware appliances, many of which can be used on their own. With these set of modules, one can easily create new HTTP request handling stacks, or reuse individual modules for handling other requests. To see the Pintura stack, you can open up lib/pintura.js, and easily see the different middleware modules that come together in Pintura (this file also lists the set of media type handlers that Pintura registers by default).

For an application running on Jack, the Pintura application stack is defined as the request handler in jackconfig.js (in Node, it is defined in start-node.js). With these startup files we can define our own additional request handling logic, and still defer store related requests to the Pintura handler. This application definition defines static file handlers to simplify development. For the example application, we also created a very simple JSGI middleware module redirect-root that will check for requests to the root path, and redirect to a specific “root” wiki page. This middleware wraps the pintura.app request handler, handling the root path, and delegating all other requests to Pintura.

The example wiki gives us a good starting point for beginning to build with Pintura. Much of the persistence, model and facet APIs are defined at the Perstore project page. In the next article in this series, we will look at some more examples of how to leverage the persistence framework and its various stores.

Pintura provides the complete web client-server REST functionality of Persevere combined with a simple, modular, JavaScript-based, facet-oriented architecture, for easy to build to JavaScript applications, with end-to-end consistency, and flexible control and security.

Related posts:

1. Introducing Pintura
2. CommonJS/JSGI: The Emerging JavaScript Application Server Platform
3. Getting Started with Persevere Using Dojo

Pintura is not a traditional MVC web server framework, which often conflate presentation and interaction concerns across the client and server, but rather follows the REST prescription of maintaining simple storage and serialization oriented server also known as thin server architecture or SOFEA. Pintura is designed to cleanly separate the concerns of presentation and interaction on the client, and storage and model logic concerns on the server. This design fits perfectly with comprehensive JavaScript frameworks like Dojo, General Interface, Cappuccino, YUI, and ExtJS that provide client-side MVC. In particular, Dojo has excellent support for standards-based JSON REST interaction that matches perfectly with this server-side framework.

The core design principles behind Pintura:

• Standards should be leveraged as much as possible to maximize interoperability with other projects, and facilitate freedom in composing web applications with the tools of choice. Pintura is based on RFC 2616 (HTTP/REST), JSON Schema, CommonJS, JSGI, JSON-RPC, and other interoperable standards.
• Presentation and data model logic should be properly separated such that clients can provide the user interface and servers can handle storage and application modeling logic. One should be able to develop multiple web clients for a given application data server.
• Storage and data processing concerns should be separated, with pluggable storage options that can easily be switched and combined (such MongoDB plus a MySQL database).
• The best option choice between configuration and convention is neither. Integration between different entities without dependence upon certain programming patterns (convention or configuration) is optimal.
• End-to-end consistency in data (JSON/JavaScript) handling provides the smoothest communication and interaction from client to server and back.
• The most common cases, like standard CRUD style applications, should be very easy, while still making it possible to build more unusual or sophisticated applications.
• A framework should be simple enough to read the source and easily extend or modify. Applications and frameworks should be organized into layers of functionality.
• Utilize object-capability design for security to allow for flexible security design while protecting against confused deputy style vulnerabilities.
• Full support throughout for promises, providing asynchronous functionality (used extensively by Node), while preserving clean encapsulated interfaces.

Improvements over Persevere 1.0

The server-side JavaScript community has dramatically changed since Persevere development started. There have been numerous exciting new server-side projects that have started like node.js, flusspferd, and Narwhal, utilizing the new generation of advanced high-performance JavaScript engines like V8, TraceMonkey, and JavaScriptCore. Persevere’s main focus is on enabling efficient client server interaction and for well-structured web application development, and matching this functionality with powerful new platforms is a great fit.

In many ways, Pintura represents a simplification of Persevere. With an entirely JavaScript-based modular design, it is very easy to dig in and understand how it works, and the complicated Java-JavaScript bridges have been eliminated. The central request handling mechanism is composed of about 13 middleware modules that can easily be rearranged and used on their own. From the ground up, Pintura has a simple, minimal, direct approach to providing REST functionality.

There are numerous other improvements over Persevere 1.0. Pintura introduces a new security system that is based on the object capability model and is therefore much more flexible and powerful than the previous purely role-based mechanism. Much scientific research has pointed to the superiority of capability based design and how it avoids dangerous confused deputy issues. Pintura utilizes “facets” (also known as proxies) to provide fine-grained access control to different stored objects. Each facet defines what actions are available for objects, with the flexibility of programmatic guard functions. In the end, security is essentially as simple as defining what facets are available for given users, and then defining what actions the facets allow.

Pintura also has moved to implementing the W3C proposed standard Indexed Database APIs object store interface for all data stores. Pintura is based around the Perstore persistence library, which implements this simple API consisting of a REST-style interface. The stores expose get, put, and delete methods and is used at multiple levels in the framework for simplicity and consistency, including for facets, model classes, as well as lower level object stores. Furthermore, additional functionality is easily applied to stores with provided store wrappers that add support for inheritance, caching, replication, event notification and more. Not only that, but with the interface likely to be implemented in browsers soon, this API will provide true end-to-end consistency.

Pintura has also embraced a much more asynchronous oriented design. JavaScript in the browser uses shared-nothing event loop concurrency, and server side projects like Node and Narwhal are following suit. This not only improves client-server consistency, but there is compelling evidence that suggests that event-based design performs and scales much better than traditional thread-based web servers. One of the major challenges of event-based systems is the encapsulation of functionality when dealing with callbacks, but promises are used extensively to greatly simplify working with asynchronous code.

Pintura also leverages the link relation definitions specified by the JSON schema specification that is now a submitted Internet-Draft. Link relations superset the functionality of JSON referencing, and allow for much cleaner, more flexible referencing syntax. This further advances the flexibility of a loosely coupled hyperlink-driven REST architecture.

A core feature of Pintura is now content negotiation, following the REST architectural design of permitting multiple representations of a resource. Resources effectively are persistable JavaScript objects, and representations allows these objects to be serialized and deserialized in multiple formats. Pintura has excellent RFC 2616 conformance to content negotiation and provides a number of out of the box media type handlers, such JSON, JavaScript, CSV, and more, while making it easy to create new custom media type handlers.

Finally, Pintura uses greatly simplified querying syntax. It continues to provide powerful URL-based querying, but the format is simplified to work easily with standard HTML form URL-encoding, aligns with the feed item query language, and has an easy to use and extensible call syntax. Pintura provides a parsing mechanism to help developers implement their own custom query handlers.

Role in Persevere 2.0

Persevere 2.0 is not a single monolithic project, but will be a distribution of several web stack configurations composed of various components including Pintura, Perstore, and the store explorer. Pintura and Perstore are still alpha stage software, but the Persevere 2.0 system is shaping up quickly.

The Next Web Stack

Pintura brings together the cumulative advances of modern JavaScript and web architecture design patterns, including object capability based security, event-driven flow, immutable promises, JSON Schema, REST style, and thin server architecture. Coupled with the powerful new CommonJS platforms and remarkably fast JavaScript engines, this framework fulfills a critical role in the next generation web application stack, and is built to adapt to your JS platform of choice. The next article in this series will explore how to get started with Pintura.

Related posts:

1. Getting Started With Pintura
2. Persevere 1.0
3. CommonJS/JSGI: The Emerging JavaScript Application Server Platform

Introducing dojo.cache

Dojo 1.4 adds a new core utility called dojo.cache. To appreciate it we first have to review how Dijit’s templatePath works. When you define a _Templated widget with a templatePath property, the content from that URL is fetched the first time you instantiate the widget, and made available as a string. All subsequent instances get the cached string. Furthermore, when you run a build, your templatePaths get replaced with templateStrings and their content is inlined into the output from the build. This improves performance considerably by removing those synchronous XHR requests, while remaining transparent to the developer.

dojo.cache generalizes this pattern, making the same functionality available from Dojo Core—synchronous, cached content retrieval that gets inlined during the build. dojo.cache usage is much like dojo.moduleUrl:

Runtime configuration

You’ll find many uses for dojo.cache. The ability to define and cache strings was immediately useful for me in the context of runtime configuration files. This is a fairly common technique in which you define and load modules ahead of most of the components to your app, to externalize constants and properties for the application that you don’t want to hard-code into each component. So, you might have something like this:

…and a variation:

In this example, switching the “tone” in your app is as simple as switching the one line that requires your config. Your code can use e.g. this.attr("content", myapp.config.loadingText) without having to know or care which variation it is supposed to be using.

Template configuration

Now with dojo.cache, we can manage templates in the same way, using object paths which can be looked up and requested at runtime during development, and inlined by the build for production:

The config lets us retrieve template strings by using a look-up like myapp.config.templates.someWidget. With Dojo, we can treat these types of object paths as strings and use dojo.getObject to receive the value they represent. In this sense the object path is like an address—it gives us the ability to reference a variable and potentially a long string like a template with a short, symbolic one. I use these configuration properties by referencing them in a (new and custom) templateKey widget property. The key provides a level of indirection that enables you to pull template management outside of your widget definition entirely. It looks like this:

Template Reuse

When all that varies between a set of widget classes is the template markup, you can reuse one widget by simply passing the instance a new templateKey, for example:

The Implementation

To make this happen we need to have all widgets use an extended dijit._Templated to put the templateKey property on the prototype, and handle it in the buildRendering step:

.. which gets used by all our custom templated widgets, e.g.

Now, when we run our build, the config file you’ve specified gets the inlined templates, and the templateKey look-up works just as before. We could even monkey-patch this functionality into dijit._Templated and wring yet more goodness from the out-of-the-box Dijit & DojoX widgets.

Conclusion

Dijit’s architecture has always let you override templates, but in practice this has typically necessitated a subclass. The combination of runtime configuration and dojo.cache gives you new, powerful options that keep specifics out of reusable components, while maintaining resources in a manageable way. This is just one new technique that falls out of Dojo 1.4. Would you use it? What have been the highlights of 1.4 for you?

Related posts:

1. Creating Dojo Widgets with Inline Templates
2. HTML Widget Prototyping with the Dojo Toolkit
3. Dojo-Mini: Optimization Tricks with the Dojo Toolkit

CommonJS

CommonJS has focused on developing critical APIs for building reusable modules, particularly for server-side JavaScript environment. The server-side is generally based around database interaction, file I/O, HTTP serving, and the generation of data formats and HTML, whereas the client-side is based around DOM manipulation and the browser object model. There are certainly APIs that can be used on both sides, and JavaScript on the client and server invites the reuse of APIs where possible. The WebWorker, Indexed Database, and XHR APIs are promising to be enormously beneficial on the server side, and with excellent client server consistency. But still the server side requires special attention, and CommonJS is bringing the needed standards and conventions.

There have been a number of different areas that CommonJS has invested in for creating the necessary foundation for SSJS. The primary initial effort was in defining the module system, which provides an easy to use system for loading modules and their dependencies, while ensuring that each module is loaded only once. This module system is now very broadly implemented and extensively used by the other subsequent module APIs defined by CommonJS. The module system provides the essential basis for modular library and application development.

CommonJS is working on building up the low-level I/O components, defining the necessary binary data types, the APIs for file interaction, and encoding. There are proposals in place for unit testing, and a number of modules to provide concurrency support. And finally there is a HTTP interface specification known as JSGI.

Concurrency

One of the most central characteristics of a runtime is the concurrency approach. The inherent dangers of shared memory threading are widely understood, and the browsers wisely avoid exposing developers to the low level complications of shared-memory threads, alternately providing shared nothing event-loop concurrency. With the latter approach, data structures are not exposed to preemptive threads that could cause race conditions. Instead, events are queued up, and deterministically executed. Concurrent operations are handled with WebWorkers, in separate memory spaces, with the ability to communicate to other workers with message passing.

The CommonJS group defines APIs, so it is beyond the scope of CommonJS to dictate the concurrency model used by servers, but there is general consensus recommending that JavaScript-based servers utilize shared-nothing event loop concurrency, and CommonJS APIs are built to support this model. The CommonJS wiki includes proposals for exposing the WebWorker API through a worker module, and for exposing controls for the event loop with the event-queue module.

Promises

One key abstraction that is extremely useful for event-based systems is a promise. A promise is an abstraction that encapsulates the execution of an asynchronous action and its subsequent value that it returns. A promise can be thought of as representing the result of an asynchronous computation. The promise will receive the value returned by the call when it is finished, and acts as a placeholder for the returned value. Promises are critical to developing large scale applications that make heavy use of asynchronicity. Using callback-passing for asynchronous actions does not compose well. It is impossible to properly encapsulate functionality, switching an implementation from synchronous to asynchronous breaks APIs and creates complex flows of passing callbacks around to handle return values. With promises, values can be returned just like synchronous functions, and callers can listen for asynchronous results without requiring additional callback-passing APIs.

The promise API in CommonJS is an asynchronous handling system based on the collective experience of the promises in Dojo, Twisted, and ref_send. The ref_send promises focus on secure immutable promises, but lacks a reasonable API for promise implementors. Dojo provides a promise API (called Deferreds) with convenient chaining support, but has suffered from the use of mutable promises, with side-effect inducing callbacks complicating the data flow. The promise API combines the best of both, providing secure immutable promises with a very simple, easy to implement API for implementors, and full chaining support. The promise API in CommonJS provides a foundation that implementors can easily extend to provide the convenience functions from Dojo’s Deferred API as well as ref_send’s static operators.

An example of using CommonJS promise might look like:

JSGI: HTTP Gateway

The JavaScript gate interface (JSGI) API has been asymptotically reaching consensus in the CommonJS group, as a standard API for serving HTTP requests through JavaScript application servers. The JSGI provides a very easy to use API for responding to HTTP requests, influenced by the design of WSGI and Rack. A basic JSGI application looks like:

JSGI has been intentionally designed to make it easy to write “middleware”, modules that exist at different places in the request handling chain to add various forms of functionality. Middleware can include support for cross-domain requests, logging, error handling, special method handling, mapping URLs to different applications, and much more.

Most existing HTTP interfaces were designed around a synchronous model, and have had an extremely difficult time adapting to the asynchronous model, which is essential for efficient Comet support. JSGI on the other hand, has been developed from the ground up with asynchronicity in mind, both to support Comet, and to work properly in an event-based environment. JSGI achieves support for asynchronicity while still maintaining a simple function call and return style by leveraging promises. Here we can see the power of promises, JSGI applications can return a promise to indicate that the response is not yet complete, and no additional callback parameter need to be passed around. With promise support throughout, an asynchronous JSGI app can be as simple as:

Since JSGI effectively provides a JSON/JavaScript representation of the HTTP protocol, JSGI has powerful opportunities for use in communication outside of just HTTP server handling. JSGI can be used as an API for making HTTP requests as a client (as a promise based alternative to XHR), and can be used as an inter-process/inter-worker protocol. HTTP is almost certainly the most commonly used communication protocol, and the RESTful power of HTTP can easily be leveraged for worker communication with JSGI’s JSON-based format (that doesn’t require HTTP parsing).

Projects

CommonJS and JSGI are purely API specifications, not implementations. Lets look at some projects that implement these specifications.

Narwhal/Jack

Probably the most advanced and mature CommonJS implementation is Narwhal plus Jack. Basically all of the CommonJS APIs are implemented in Narwhal/Jack and/or their forks. Narwhal provides extensive I/O support, a large collection of useful library modules, and a great package management system called tusk with package resource handling built into its module loader. Narwhal is designed for running on various JavaScript platforms and has an intelligently architected layer of abstractions for dealing with different engines.

Narwhal has the most complete engine support for Rhino. Rhino is almost certainly the most popular JavaScript engine for server side JavaScript due to its seamless integration with Java. With Rhino, the virtually limitless expanse of Java libraries are available and easily accessible. While great efforts are being made to important libraries in other engines, the breadth of libraries that are available through the Rhino’s Java bridge is not likely to be matched anytime soon. It also comes with a good visual debugger. Rhino is also perhaps the most feature-rich JavaScript implementation available, as it implements most of the ECMAScript 5 specification and Mozilla’s JavaScript 1.8 features.

Narwhal also runs on JSCore, the JavaScript engine used by Safari. JSCore is a very high-performance engine (second only to V8, perhaps), and consequently there is great potential for high-performance applications running on Narwhal/JSCore.

Jack is a JSGI implementation that has been developed to run on Narwhal, and is also built to run on various web server technologies. Jack has been somewhat of the de facto reference implementation for JSGI, and runs on Simple, CGI, servlet containers (including Jetty and GAE), and others. Older versions of Jack relied on shared memory threading, but with the latest work on Jack, requests are handled with full event loop based shared-nothing concurrency, with request delegation across multiple workers for efficient multi-core utilization.

Node

Node is an exciting project in the SSJS realm, and has received a lot of publicity recently, for good reason. Node builds on the amazingly fast V8 engine and provides important low-level functionality. V8 is one of the fastest dynamic language engines on the planet. Node uses libev for an event-loop based asynchronous I/O model, providing the same type of event-loop shared-nothing concurrency used in Narwhal. However, Node has had a stronger emphasis on trying to provide asynchronous APIs for all I/O operations. By exposing almost all I/O operations as asynchronous events, Node rarely has to block while waiting for I/O, and is able to execute much more efficiently than spreading workloads across multiple blocking threads. The performance advantages to event-loops over threads are well-documented. This focus, in combination with the blazing fast V8 engine, promises to place Node as the performance king of dynamic language based application servers.

There are certainly gaps that need to be filled with Node. Node implements CommonJS’s module system, but does not implement other CommonJS APIs, and has been slow to add better support for interoperability. Node runs its HTTP server and handling on a single process/thread, which means it can’t utilize multi-cores yet (support for that is coming). Node does not support the JSGI specification; however, it is possible to run JSGI on Node with the JSGI-Node adapter that I built.

There are other important things like package support and module reloading that are definitely needed. From my experience, Node does not feel as complete as Narwhal. However, despite these concerns, Node has a lot of potential, and is an exciting project. Node primarily fulfills a role as a low-level JavaScript platform, on which higher level functionality can be built. The Node community is responsive. There is an effort to run Narwhal on top of Node, which would be a powerful combination.

Ejscript

Ejscript is a JavaScript platform that extends the JavaScript language with many new constructs. Many of these additions are based on the ECMAScript 4 specification that was eventually abandoned because many implementors felt it was too extensive of a revision of the language. However, many of the features included in ECMAScript 4 that Ejscript implements were technically excellent additions, and may be included in ECMAScript 6. Beyond extra language features, Ejscript boasts a very high performance engine with remarkably low memory footprints, claiming performance that rivals Node with less than a tenth of the memory consumption.

Persevere

Persevere was one of the first projects to implement the CommonJS module specification. However, the primary goal of the Persevere project is to provide consistent end-to-end data semantics leveraging RESTful architecture. Persevere 1.0 provided CommonJS module support to provide interoperability with other CommonJS libraries, but being a JavaScript platform is beyond the primary scope of the project. With the new array of CommonJS platforms, future Persevere development will now focus on building on top of these new platforms and providing its data persistence, security, REST architecture, Comet notifications, and advanced web communication capabilities for multiple platforms.

Others

Mozilla’s SpiderMonkey engine is also a capable engine for server side JavaScript. Flusspferd is a project to add low-level system functionality to SpiderMonkey and couples with Zest as a JSGI server and Juice as a web framework on top.

There are number of other important server-side projects including HelmaNG, with high-quality technology that predates CommonJS and now are adapting to work in harmony with the CommonJS ecosystem, but I won’t cover all of them.

Web Frameworks

Some exciting new web frameworks are building on these technologies such as Nitro, Bogart, and JuiceJS. Also some platforms come with their own MVC framework such as HelmaNG, and EJScript. However, for many the move to JavaScript has been motivated by investment in rich client interfaces that already have presentation capabilities (the V and C in MVC), so a more modern REST-style, “thin server architecture” may be a better fit than traditional server-side MVC frameworks for many next generation JavaScript-based web applications. Pintura is the core of the next generation of Persevere, a cross-platform REST-style architecture web framework that is designed specifically for Ajax-style applications that maintain presentation logic on the client separate from the server-side storage concerns. We will cover Pintura in the next article of this series.

Conclusion

The combination of improvements and maturation of ECMAScript standards, with the new line of astonishingly fast JavaScript, the Ajax driven interest in JavaScript, the collaborative efforts of CommonJS standards, and the emerging array of server-side JavaScript projects are all coming together to propel server-side JavaScript to the forefront of web application development. It is not unreasonable to expect that the CommonJS APIs implemented with a web stack of the new line of SSJS projects could be the most important new web platform of the next decade.

Related posts:

1. Introducing Pintura
2. Getting Started With Pintura
3. Narwhal on Persevere

These efforts are outstanding and highly useful, especially the JSMeter work and their valiant effort to redefine performance tests that are indicative of today’s web apps.

I completely agree with one of their stated goals:

We hope our results will convince the JavaScript community to develop and adopt benchmarks that are more representative of real web applications.

However, I disagree with their approach: they are testing the performance of today’s already optimized sites! There’s nothing in the middle of testing today’s sites and the more unrealistic “test every feature” ACID tests.

I believe more accurate tests for tomorrow would be useful, testing what’s pushing the limits of the web today, but are not currently top 100 sites. My main objection with comparing performance across the top 100 web sites is this: The top 100 web sites are already relatively highly performant, because they are optimized for what’s possible today. They have and continue to improve thanks in large part to the work of Steve Souders and others in the performance optimization community. Because it costs significant amounts of money in server operations fees and bandwidth, high traffic web sites generally dedicate considerable resources to highly optimizing their sites. High traffic web sites also face significant competition and are highly scrutinized for acceptable page load time. Budget and competition result in popular sites not deploying code that makes pages load slower than their desired performance threshold. Even more importantly, top 100 sites have the budget to make their app work in the future when things change. You can dedicate people on your team to squeezing out performance improvements in all aspects if you have the budget for it. Most web apps cannot afford to do this.

When we’re testing the performance of new browsers or analyzing page load performance, we should also really be looking at what the top 100 sites will look like in terms of features and expectations in five years! So how do we do that today? There’s no simple answer, but here are some ideas:

• Test popular web apps, e.g. mint.com populated with large amounts of data
• Test apps that don’t support IE6, e.g. Google Wave
• Test all sections of popular sites, not just the home page, through an automated performance test harness
• Test ridiculous configurations of popular applications, e.g. enable every feature in modular applications until they slow down
• Test apps over long amounts of time in the browser, not just initial page load time
• Test 50 apps, each in a different tab, all at once, and see how fast you can make a browser like Firefox or IE crash!
• Test throttled networks that emulate the profile of mobile and satellite networks, slow hotel wi-fi networks that often limit the length and duration of connections, corporate proxies, tech conferences, and countries with overloaded pipes (e.g. YouTube in New Zealand)

Only when browsers are pushed to their limits do we see where they break down, and how sites break them. We also need tests and tools (such as instrumented usage of YSlow, PageSpeed, SpeedTracer, etc.) comparing the most complex apps and how they perform across the various browsers, as today’s complex app is potentially tomorrow’s median site.

To be clear, I’m not saying “don’t optimize for today”. I’m saying, stop comparing cutting edge sites to Google search results. Lumping these two together in a common test is like putting apples against oranges because they are both round fruit.

Related posts:

1. String Performance: Getting Good Performance from Internet Explorer
2. Why Apple is Investing in WebKit Performance
3. Web Page Global Variable Performance

Of particular interest:

• IO Pipeline topics
• dojo.cache
• dojo.contentHandlers
• dojo.hash with native HTML5 onhashchange event support where available
• Traversal and manipulation for NodeLists (the return value for dojo.query)
• dojo.ready (easier to type than dojo.addOnLoad)
• Hundreds of refinements to the Dijit API and collection of Dijits, and a few new widgets in DojoX
• DataChart widget and other improvements to charting
• dojox.drawing lands!
• Editor improvements and new plug-ins in both Dijit and DojoX
• Grid is faster, and the EnhancedGrid lands!
• ForestStoreModel for the TreeGrid
• GFX improvements
• dojox.jq, a very experimental module aimed at trying to match the jQuery API as close as possible, but using Dojo underneath
• Dojo build system optionally supports the Google Closure Tools compiler
• Significant speed improvements, especially in IE

Read the full Dojo 1.4 release notes for more details! And thanks to everyone in the Dojo community that helped make this release great!

Related posts:

1. Dojo 0.9 released
2. Introducing the Dojo Toolkit 1.1
3. Dojo Grid Update

Android has the first mover advantage of being deployed today to many devices, but to really get the most out of it, you really should develop using Java, or employ a toolkit like PhoneGap. Chrome OS offers the promise of using web technologies that are popular today, but is not yet production-ready, or optimized for mobile devices like Palm’s webOS.

The long-term expected convergence from Google makes sense. Convergence of Palm’s webOS with Chrome OS makes sense to us as well as they are both pushing towards having a very similar WebKit-based operating system for delivering web applications. Palm already appears to be moving in the direction of converging with Chrome through their adoption of key technologies in webOS such as Google’s V8 JavaScript engine. There’s nothing confirming that this will happen, other than it just making sense.

While there is a lot of short term interest in apps, you can still get significant results from mobile web apps, and the gap between a native app and a web-based app will quickly shrink, as evidenced by apps like Pie Guy. After all, web apps don’t require app store approval!

Related posts:

1. An Android without Gears?
2. Why We Love Chrome Frame
3. SDK showdown: iPhone vs. Android

In their post, Read-Write Web asks if offline access is even necessary any longer. I guess they don’t spend enough time on airplanes or at hotels and conferences with poor internet connectivity! It’s certainly necessary, and while other browsers have developed significant offline capabilities, older browsers still need a plug-in like Gears.

In the interim, what should you do if you want an offline application? Do you develop for Gears and HTML 5 features? Do you wait for Chrome Frame to integrate offline capabilities? Do you use a toolkit like Dojo which will wrap the various possibilities? Or do you rely on something else like AIR, Titanium, Prism, or Fluid? The answer really depends on your application. If it is live now, you plan for the future but you keep going with Gears and/or Dojo Offline. If you won’t be launching for some time, you may want to talk to us about your offline app options.

At the end of the day, Gears is open source. If there’s a long-term need for its existence, the community can pick it up and run with it! Thankfully end-of-life isn’t the certain death-knell that it is with closed-source software.

Related posts:

1. Google Gears, Dojo Offline, and SitePen
2. Why We Love Chrome Frame
3. An Android without Gears?

• Create, read, update, and delete access to persistent data through a standard JSON HTTP/REST web interface
• Dynamic object persistence – expando objects, arrays, and JavaScript functions can be stored, for extensive JavaScript persistence support
• Remote execution of JavaScript methods on the server through JSON-RPC for a consistent client/server language platform
• Flexible and fast indexed query capability through JSONQuery/JSONPath
• Comet-based data monitoring capabilities through HTTP Channels with Bayeux transport plugin/negotiation support
• Data-centric role-based object level security with user management, Persevere is designed to be accessed securely through Ajax with public-facing sites
• Comprehensive referencing capabilities using JSON referencing, including circular, multiple, lazy, non-lazy, cross-data source, and cross-site referencing for a wide variety of object structures
• Data integrity and validation through JSON Schema based definitions
• Class-based data hierarchy – typed objects can have methods, inheritance, class-based querying
• Pluggable data source architectures – SQL tables, XML files, remote web services can be used as data stores
• Object versioning with transactional history of record states

Persevere in use

Recently, Cramer Development put together a slick little application for making sticky notes. They discuss how quickly the application came together, as Persevere allowed them to quickly establish a data API, and then focus on the client side interface.

Other users include:

• DataStream Content Solutions is using Persevere to build an XML repository for legal data in combination with MarkLogic.
• Montana State University is using Persevere for their Yogo Data Management Project.
• Another multi-national company is using Persevere in production for Intranet applications, with consistent usage from a number of users.
• And, of course, we at SitePen are using Persevere for a number of the applications we are developing.

Numerous others are using Persevere in a variety of ways.

Learning more

There are a number of resources for learning more about Persevere and getting started with it.

• Persevere Documentation
• Download Persevere
• SitePen Persevere Support
• SitePen Persevere Blog
• Getting Started with Persevere Using Dojo
• Persevere Mailing List
• Persevere + Dojo (using Comet) demonstration

What’s Next

With Persevere 1.0 finished, we are already working on the next version which will be based on the new Pintura architecture. Pintura is the new JavaScript core for the Persevere HTTP interface that is based on the CommonJS and JSGI API. Pintura will run on any CommonJS/JSGI capable JavaScript engine (support for V8, JSCore, and Spidermonkey coming).

Related posts:

1. When to Use Persevere: a Comparison with CouchDB and Others
2. The Latest from SitePen Labs
3. Using the Persistent Object Model in Persevere