The NoSQL database approach is characterized by a move away from the complexity of SQL based servers. The logic of validation, access control, mapping querieable indexed data, correlating related data, conflict resolution, maintaining integrity constraints, and triggered procedures is moved out of the database layer. This enables NoSQL database engines to focus on exceptional performance and scalability. Of course, these fundamental data concerns of an application don’t go away, but rather must move to a programmatic layer. One of the key advantages of the NoSQL-driven architecture is that this logic can now be codified in our own familiar, powerful, flexible turing-complete programming languages, rather than relying on the vast assortment of complex APIs and languages in a SQL server (data column, definitions, queries, stored procedures, etc).

In this article, we’ll explore the different aspects of data management and suggest an architecture that uses a data management tier on top of NoSQL databases, where this tier focuses on the concerns of handling and managing data like validation, relation correlation, and integrity maintenance. Further, I believe this architecture also suggests a more user-interface-focused lightweight version of the model-viewer-controller (MVC) for the next tier. I then want to demonstrate how the Persevere 2.0 framework is well suited to be a data management layer on top of NoSQL databases. Lets look at the different aspects of databases and how NoSQL engines affect our handling of data and architecture.

Architecture with NoSQL

In order to understand how to properly architect applications with NoSQL databases you must understand the separation of concerns between data management and data storage. The past era of SQL based databases attempted to satisfy both concerns with databases. This is very difficult, and inevitably applications would take on part of the task of data management, providing certain validation tasks and adding modeling logic. One of the key concepts of the NoSQL movement is to have DBs focus on the task of high-performance scalable data storage, and provide low-level access to a data management layer in a way that allows data management tasks to be conveniently written in the programming language of choice rather than having data management logic spread across Turing-complete application languages, SQL, and sometimes even DB-specific stored procedure languages.

Complex Data Structures

One important capability that most NoSQL databases provide is hierarchical nested structures in data entities. Hierarchical data and data with list type structures are easily described with JSON and other formats used by NoSQL databases, where multiple tables with relations would be necessary in traditional SQL databases to describe these data structures. Furthermore, JSON (or alternatives) provide a format that much more closely matches the common programming languages data structure, greatly simplifying object mapping. The ability to easily store object-style structures without impedance mismatch is a big attractant of NoSQL.

Nested data structures work elegantly in situations where the children/substructures are always accessed from within a parent document. Object oriented and RDF databases also work well with data structures that are uni-directional, one object is accessed from another, but not vice versa. However, if the data entities may need to be individually accessed and updated or relations are bi-directional, real relations become necessary. For example, if we had a database of employees and employers, we could easily envision scenarios where we would start with an employee and want to find their employer, or start with an employer and find all their employees. It may also be desirable to individually update an employee or employer without having to worry about updating all the related entities.

In some situations, nested structures can eliminate unnecessary bi-directional relations and greatly simplify database design, but there are still critical parts of real applications where relations are essential.

Handling Relational Data

The NoSQL style databases has often been termed non-relational databases. This is an unfortunate term. These databases can certainly be used with data that has relations, which is actually extremely important. In fact, real data almost always has relations. Truly non-relational data management would be virtually worthless. Understanding how to deal with relations has not always been well-addressed by NoSQL discussions and is perhaps one of the most important issues for real application development on top of NoSQL databases.

The handling of relations with traditional RDBMSs is very well understood. Table structures are defined by data normalization, and data is retrieved through SQL queries that often make extensive use of joins to leverage the relations of data to aggregate information from multiple normalized tables. The benefits of normalization are also clear. How then do we model relations and utilize them with NoSQL databases?

There are a couple approaches. First, we can retain normalization strategies and avoid any duplication of data. Alternately, we can choose to de-normalize data which can have benefits for improved query performance.

With normalized data we can preserve key invariants, making it easy to maintain consistent data, without having to worry about keeping duplicated data in sync. However, normalization can often push the burden of effort on to queries to aggregate information from multiple records and can often incur substantial performance costs. Substantial effort has been put into providing high-performance JOINs in RDBMSs to provide optimally efficient access to normalized data. However, in the NoSQL world, most DBs do not provide any ad-hoc JOIN type of query functionality. Consequently, to perform a query that aggregates information across tables often requires application level iteration, or creative use of map-reduce functions. Queries that utilize joining for filtering across different mutable records often cannot be properly addressed with map-reduce functions, and must use application level iteration.

NoSQL advocates might suggest that the lack of JOIN functionality is beneficial; it encourages de-normalization that provides much more efficient query-time data access. All aggregation happens for each (less frequent) write, thus allowing queries to avoid any O(n) aggregation operations. However, de-normalization can have serious consequences. De-normalization means that data is prone to inconsistencies. Generally, this means duplication of data; when that data is mutated, applications must rely on synchronization techniques to avoid having copies become inconsistent. This invariant can easily be violated by application code. While it is typically suitable for multiple applications to access database management servers, with de-normalized data, database access becomes fraught with invariants that must be carefully understood.

These hazards do not negate the value of database de-normalization as an optimization and scalability technique. However, with such an approach, database access should be viewed as an internal aspect of implementation rather than a reusable API. The management of data consistency becomes an integral compliment to the NoSQL storage as part of the whole database system.

The NoSQL approach is headed in the wrong direction if it is attempting to invalidate the historic pillars of data management, established by Edgar Codd. These basic rules for maintaining consistent data are timeless, but with the proper architecture a full NoSQL-based data management system does not need to contradict these ideas. Rather it couples NoSQL data storage engines with database management logic, allowing for these rules to be fulfilled in much more natural ways. In fact, Codd himself, the undisputed father of relational databases, was opposed to SQL. Most likely, he would find a properly architected database management application layer combined with a NoSQL storage engine to fit much closer to his ideals of a relational database then the traditional SQL database.

Network or In-process Programmatic Interaction?

With the vastly different approach of NoSQL servers, it is worth considering if the traditional network-based out-of-process interaction approach of SQL servers is truly optimal for NoSQL servers. Interestingly, both of the approaches to relational data point to the value of more direct in-process programmatic access to indexes rather than the traditional query-request-over-tcp style communication. JOIN style queries over normalized data is very doable with NoSQL databases, but it relies on iterating through data sets with lookups during each loop. These lookups can be very cheap at the index level, but can incur a lot of overhead at the TCP handling and query parsing level. Direct programmatic interaction with the database sidesteps the unnecessary overhead, allowing for reasonably fast ad-hoc relational queries. This does not hinder clustering or replication across multiple machines, the data management layer can be connected to the storage system on each box.

De-normalization approaches also work well with in-process programmatic access. Here the reasons are different. Now, access to the database should be funneled through a programmatic layer that handles all data synchronization needs to preserve invariants so that multiple higher level application modules can safely interact with the database (whether programmatically or a higher level TCP/IP based communication such as HTTP). With programmatic-only access, the data can be more safely protected from access that might violate integrity expectations.

Browser vendors have also come to similar conclusions of programmatic access to indexes rather than query-based access in the W3C process to define the browser-based database API. Earlier efforts to provide browser-based databases spurred by Google Gears and later implemented in Safari were SQL-based. But the obvious growing dissatisfaction with SQL among developers and the impedance mismatches between RDBMS style data structures and JavaScript style data structures, has led the W3C, with a proposal from Oracle (and supported by Mozilla and Microsoft), to orient towards a NoSQL-style indexed key-value document database API modeled after the Berkeley DB API.

Schemas/Validation

Most NoSQL databases could also be called schema-free databases as this is often one of the most highly touted aspects of these type of databases. The key advantage of schema-free design is that it allows applications to quickly upgrade the structure of data without expensive table rewrites. It also allows for greater flexibility in storing heterogeneously structured data. But while applications may benefit greatly from freedom from storage schemas, this certainly does not eliminate the need to enforce data validity and integrity constraints.

Moving the validity/integrity enforcement to the data management layer has significant advantages. SQL databases had very limited stiff schemas, whereas we have much more flexibility enforcing constraints with a programming language. We can enforce complex rules, mix strict type enforcements on certain properties, and leave other properties free to carry various types or be optional. Validation can even employ access to external systems to verify data. By moving validation out of the storage layer, we can centralize validation in our data management layer and have the freedom to create rich data structures and evolve our applications without storage system induced limitations.

ACID/BASE and Relaxing Consistency Constraints

One aspect of the NoSQL movement has been a move away from trying to maintain completely perfect consistency across distributed servers (everyone has the same view of data) due to the burden this places on databases, particularly in distributed systems. The now famous CAP theorem states that of consistency, availability, and network partitioning, only two can be guaranteed at any time. Traditional relational databases have kept strict transactional semantics to preserve consistency, but many NoSQL databases are moving towards a more scalable architecture that relaxes consistency. Relaxing consistency is often called eventual consistency. This permits much more scalable distributed storage systems where writes can occur without using two phase commits or system-wide locks.

However, relaxing consistency does lead to the possibility of conflicting writes. When multiple nodes can accept modifications without expensive lock coordination, concurrent writes can occur in conflict. Databases like CouchDB will put objects into a conflict state when this occurs. However, it is inevitably the responsibility of the application to deal with these conflicts. Again, our suggested data management layer is naturally the place for the conflict resolution logic.

Data management can also be used to customize the consistency level. In general, one can implement more relaxed consistency-based replication systems on top of individual database storage systems based on stricter transactional semantics. Customized replication and consistency enforcements can be very useful for applications where some updates may require higher integrity and some may require the higher scalability of relaxed consistency.

Customizing replication can also be useful for determining exactly what constitutes a conflict. Multi-Version Concurency Control (MVCC) style conflict resolution like that of CouchDB can be very naive. MVCC assumes the precondition for any update is the version number of the previous version of the document. This certainly is not necessarily always the correct precondition, and many times unexpected inconsistent data may be due to updates that were based on other record/document states. Creating the proper update logs and correctly finding conflicts during synchronization can often involve application level design decisions that a storage can’t make on its own.

Persevere

Persevere is a RESTful persistence framework, version 2.0 is designed for NoSQL databases while maintaining the architecture principles of the relational model, providing a solid complementary approach. Persevere’s persistence system, Perstore, uses a data store API that is actually directly modeled after W3C’s No-SQL-inspired indexed database API. Combined with Persevere’s RESTful HTTP handler (called Pintura), data can be efficiently and scalably stored in NoSQL storage engines and accessed through a data modeling logic layer that allows users to access data in familiar RESTful terms with appropriate data views, while preserving data consistency. Persevere provides JSON Schema based validation, data modification, creation, and deletion handlers, notifications, and a faceted-based security system.

Persevere’s evolutionary schema approach leans on the convenience of JSON schema to provide a powerful set of validation tools. In Persevere 2.0, we can define a data model:

var Model = require("model").Model;
// productStore provides the API for accessing the storage DB.
Model("Product", productStore, {
  properties: {
    name: String, // we can easily define type constraints
    price: { // we can create more sophisticated constraints
      type: "number",
      miminum: 0,
    },
    productCode: {
      set: function(value){
        // or even programmatic validation
      }
    }
  },
  // note that we have not restricted additional properties from being added
  // we could restrict additional properties with:
  // additionalProperties: false

  // we can specify different action handlers. These are optional, they will
  // pass through to the store if not defined.
  query: function(query, options){
    // we can specify how queries should be handled and 
    //delivered to the storage system
    productStore.query(query, options
  },
  put: function(object, directives){
     // we could specify any access checks, or updates to other objects 
     // that need to take place
     productStore.put(object, directives);
  }
});

The Persevere 2.0 series of articles provides more information about creating data models as well as using facets for controlling access to data.

Persevere’s Relational Links

Persevere also provides relation management. This is also based on the JSON Schema specification, and has a RESTful design based on the link relation mechanism that is used in HTML and Atom. JSON Schema link definitions provide a clean technique for defining bi-directional links in a way that gives link visibility to clients. Building on our product example, we could define a Manufacturer model, and link products to their manufacturers:

Model("Product", productStore, {
   properties: {
      ...
      manufacturerId: String
   },
   links: [
     { 
        rel: "manufacturer",
        href: "Manufacturer/{manufacterId}"
     }
   ],
...
});

Model("Manufacturer", manufacturerStore, {
   properties: {
      name: String,
      id: String,
      ...
   },
   links: [
     { 
        rel: "products",
        href: "Product/?manufacterId={id}"
     }
   ]
});

With this definition we have explicitly defined how one can traverse back and forth (bi-directionally) between a product and a manufacturer, using a standard normalized foreign key (no extra effort in synchronizing bi-direction references).

The New mVC

By implementing a logically complete data management system, we have effectively implemented the “model” of the MVC architecture. This actually allows the MVC layer to stay more focused and lightweight. Rather than having to handle data modeling and management concerns, the MVC can focus on the user interface (the viewer and controller), and simply a minimal model connector that interfaces with the full model implementation, the data management layer. I’d suggest this type of user interface layer be recapitalized as mVC to denote the de-emphasis on data modeling concerns in the user interface logic. This type of architecture facilitates multiple mVC UIs connecting to a single data management system, or vice versa, a single mVC could connect to multiple data management systems, aggregating data for the user. This decoupling improves the opportunity for independent evolution of components.

Conclusion

The main conceptual ideas that I believe are key to the evolution of NoSQL-based architectures:

• Database management needs to move to a two layer architecture, separating the concerns of data modeling and data storage. Persevere demonstrates this data modeling type of framework with a web-oriented RESTful design that complements the new breed of NoSQL servers.
• With this two layered approach, the data storage server should be coupled to a particular data model manager that ensures consistency and integrity. All access must go through this data model manager to protect the invariants enforced by the managerial layer.
• With a coupled management layer, storage servers are most efficiently accessed through a programmatic API, preferably keeping the storage system in-process to minimize communication overhead.
• The W3C Indexed Database API fits this model well, with an API that astutely abstracts the storage (including indexing storage) concerns from the data modeling concerns. This is applicable on the server just as well as the client-side. Kudos to the W3C for an excellent NoSQL-style API.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Roy Fielding‘s REST architecture is broken down into seven constraints (and the four sub-constraints of the uniform interface). The individual concepts here are certainly not new, but collectively looking at these concepts as resource oriented programming may provide an interesting new perspective. I will also look at how these principles are exemplified in Persevere 2.0 in its object store framework, Perstore, and its web stack Pintura.

Client-Server

Of all the REST principles the first principle is most targeted at the network paradigm, and is the least compelling for advising programming. However, it does suggest the importance of an asymmetric relationship between modules with separation of concerns. When modules interact, one module should stick to the role of providing a service (acting as the server), and one should act as the consumer of the service. The roles and concerns of the two modules thus stay cleanly separated.

Stateless

If you follow computer science research, you are probably aware that one of the predominant focuses of modern programming languages is seeking to minimize the hazards of stateful imperative programming. Many developers are moving towards functional languages, or at least functional elements of languages, which perform computations on inputs, yielding outputs, rather than being defined as a series of state changes. Concurrency, security, scalability, and many other programming concepts become drastically simpler when mutating state is removed, and applications become much easier to develop. This directly corresponds with the stateless principle of REST. Maintaining transitive state information puts an enormous burden on modules in terms of scalability and complexity. Minimizing or eliminating state and using functional paradigms yields much better code.

When we consider the stateless constraint of REST, there is an important distinction to be made between “state” and “data” or “resources”. Since REST is clearly resource oriented, data/resources are obviously not something that we are trying to eliminate. When we are advising against state, we are advising against the use of transitive state information as the primary programming technique for developing and implementing application flow and logic. The resources and data to which an application provides a user interface are still mutable structures, but this interaction is carefully handled through the uniform interface (coming later).

Cache

Another key concept of a resource oriented approach is that it provides visibility for interaction. REST’s interaction explicitly defines cacheability and idempotence of actions. Certain operations are known to be safe and cacheable. In HTTP, GET requests fit this criteria. In standard object oriented programming conventions, a getter is safe and cacheable as well. If a uniform interface has been defined, one can define one or more retrieval function that are safely cacheable. The uniform interface must explicitly disclose which operations have side-effects. This can have very significant implications for creating well-optimized code. We can re-utilize local copies of data without re-calling retrieval functions if the functions are known to be cacheable. In the realm of purely opaque functions we are afforded no such opportunities (who knows if a function might have side-effects).

Getters should be safe and cacheable (and idempotent), and setters should be idempotent. Getters and setters provide an extremely powerful concept within programming. Distilling everything down to opaque calls hides important semantics from users that can distinctly alter their expectations and use of modules, but getters and setters expose these expectations in a way that benefits users. Developers can expect that object.foo can be evaluated multiple times with the same result unless the property has been altered. Setting object.foo is an idempotent operation (i.e., it can be safely repeated) and can be expected to alter the value returned by object.foo (if successful).

Uniform Interface

The most central concept of REST is the uniform interface, often characterized by HTTP’s standard methods: GET, PUT, POST, DELETE, etc. as the means for navigation and interaction. These methods have clear semantics. GET is safe, PUT, POST, DELETE have side-effects. GET, PUT, and DELETE are idempotent, POST is not. PUT is supposed to update a resource in a way that predictably alters the expected response to a GET.

These methods have analogies in object programming with getters and setters as we have discussed; however a uniform interface can be more than simply getters and setters. The uniform interface does not act solely on the internal state of individual objects, but can affect the collection of some type of objects, providing means for creating and deleting objects, querying for objects, retrieving by unique identifier, and even locking and unlocking the objects.

Perhaps one excellent example of the uniform interface principle in JavaScript is Dojo‘s Data API. While it doesn’t utilize the traditional HTTP-style set of verbs, this API does follow the uniform interface principle and provides a single generic uniform interface through which all widgets can access data, and the data source can implement this interface to provide generic access to various different storage systems or file formats.

The W3C is working on a compelling new API called the indexed database API that could provide a more broadly accepted generic uniform interface for JavaScript, and even bears quite a bit of resemblance to the traditional HTTP verbiage for verbs, with get(), put(), and delete() methods. This API is used by Persevere 2.0 in it’s object store framework, Perstore, and further leveraged by its web stack Pintura. With this interface, various different storage systems, especially NoSQL servers, can very easily be modeled and leveraged at different layers within the Pintura/Perstore ecosystem.

The uniform interface is further broken down into four important sub-constraints.

Identification of Resources

A fundamental concept of REST is that resources must have a unique identifier which can be used to retrieve that resource (a URL). The identifier must be unique and carry sufficient information to locate the resource. If the identifier is only unique within a given subsystem, additional information must be added to the identifier if it is referenced outside the subsystem. This is in fact how URLs work. The paths are unique to a server, but the path is combined with the server host name to make them unique across the entire Internet. When applied to programming, resources are objects that are identifiable. Within an application runtime, it is often sufficient to simply use the server unique identifier, or even within the scope of certain modules use an identifier specific to a given table or data store.

Within Perstore, all resource objects must have an identifier property or getId() function that returns the identity of the object. Resource objects can be retrieved using the store.get(id). These mechanisms guarantee a way to discover a resource’s identity and retrieve the resource by the identity.

Manipulation of Resources through Representations

REST defines a distinction between resources and representations. Resources are not necessarily simple files. Resources may represent abstract concepts such people, computers, events, and more, and may be stored and accessed through various means such as database records. In the scope of network communication, representations provide a means for describing these resources using byte-level serialization. REST recognizes that different consumers may need or prefer different types of representations, and supporting multiple representations for a given resource can be highly valuable.

In intra-machine programming, of course modules can interact with higher level constructs than byte-level data, and can use objects, numbers, strings, and language constructs. However, there still may be multiple ways of representing an abstract resource. These can be particularly important if different data consumers expect data to be structured in different ways.

One of the ways that the abstraction between a representation and a resource can be achieved is with facets. I have previously discussed how facets are a powerful tool for securely controlling access to resources in the object capability model. Facets act as a wrapper for an object store, and can not only attenuate access, but can provide alternate “views” of data/resources. For example, facets can easily be used for localization. You could create one facet for a set of resources that provides methods and properties for English speakers, and then another set of methods and properties for French speakers. The facet could include any necessary logic for pulling data from different data sources based on locale. This use of facets is a clear demonstration of separation of representation/view and resource.

Self-descriptive Messages

Self-descriptive messages describe how we view and interact with a resource. There are a couple aspects of this principle. First, we should be able to easily introspect a representation of a resource to comprehend the structure and content, and use this information to intelligently update the object. Objects with properties (or hash/map style name-value pairs, which is the same thing in JavaScript) are well suited for this since the introspection clearly implies how the object can be updated. Self-descriptive objects are easier to interact with because generic tools can interact with them. The API for interacting with the object is evident from the object itself.

Self-descriptive messages also means that in addition to the raw representation of a resource, we should also be able to view additional metadata that describes extra information about the resource such as cacheability, attribution information, etc. Here, Pintura defines a getMetadata() function for resource objects that provides access to metadata information for resources and their representations. This makes metadata readily available for processing conditional requests (based on modification times or etags) and defining caching levels.

Hypermedia as the Engine of Application State

The ability to navigate resources through hyperlinks/hypermedia is the hallmark of REST. This is easy to relate to object oriented programming where property values can be references to other objects or getters can return other objects, making it easy to navigate through data structures. We frequently utilize object mapping to maintain these types of object references from relationships indicated by underlying data sources.

Perstore leverages JSON Schema’s hyperlink description capabilities to define links based on data. The JSON Schema implementation used by Perstore provides a getLink() function to the target of a resource link as defined in the schema definition. For example, we can define a model in Perstore:

var Model = require("model").Model;
Task = Model("Task", dataStore, {
  links: [
    {rel: "project", href: "/Project/{projectId}"}
  ]
});

And then navigate through the data:

var getLink = require("json-schema/links").getLink,
    someTask = Task.get("some task id");
someTask.projectId -> id of a the project it is part of
someTask.get("project") -> retrieves the target project through the defined link

getLink(“project”, someTask, Task) -> “/Project/project-id”

Layered

REST advises layering as the technique for composing functionality and scalability from multiple agents. In programming, layering can also be used, and is best described as wrapping or composition. With programmatic layering, one object “wraps” another source object and handles all messages before sending them on to the source object. The wrapper can then add functionality to the source object and still expose the same interface to object users.

The programmatic layering principle is also seen in middleware. The HTTP interface of middleware-centric web applications consists of layers of middleware, where each middleware acts as an HTTP intermediary, adding functionality along the way. Pintura, the HTTP interface for Persevere 2.0, is completely driven by this middleware-centric approach. Pintura is basically a set of middleware providing authorization, CSRF protection, content negotiation, and more.

Perstore makes extensive use of the layering approach as well. Perstore uses a single REST-style uniform interface object store API (with get, put, delete from the W3C Indexed DB API), that is used throughout the framework. Low level data storage adapters (for MongoDB, JSON files, CouchDB, SQL, etc), can implement this API, and then store wrappers can be layered on top to add additional functionality such as caching, subscription/notification, aggregation, replication, and adaptable-indexing. For example, the we can take a store and add notification and caching to it:

var Notifying = require("store/notifying").Notifying,
    Cache = require("store/cache").Cache,
    Memory = require("store/memory").Memory,
var enhancedStore = Notifying( // add notifications
   Cache( // add caching
      dataStore, // our original source store
      Memory())); // cache in memory

Perstore’s models and facets are layers themselves, also implementing the same interface, and provide schema validation and access control. These various layers are effectively an object store form of middleware.

Code on Demand

The Code on Demand principle provides a pathway to extensibility for applications. A traditional basic resource may simply be data, but this principle advises that resources may contain code (or be code entirely), providing language level flexibility for richer content than is possible for simple static data structures. As the LISP mantra goes, “code is data and data is code”. JavaScript can easily be used in homoiconic ways, supporting this paradigm well. The object literal syntax of JavaScript (the proper superset of JSON), is well adapted to include functions holding programmatic logic. By supporting storage of code/functions in databases, one can easily define individual resource-specific logic, and resource consumers can easily interface to resource functions as means for allowing resource-specific extensibility in applications.

Summary

These seven principles (and the four sub-principles of uniform interface) collectively integrate to form the powerful architectural style known as REST. This important network architecture can be applied as a programming style in resource oriented programming, and Persevere 2.0 provides a great framework for building applications with this methodology.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

CommonJS

CommonJS has focused on developing critical APIs for building reusable modules, particularly for server-side JavaScript environment. The server-side is generally based around database interaction, file I/O, HTTP serving, and the generation of data formats and HTML, whereas the client-side is based around DOM manipulation and the browser object model. There are certainly APIs that can be used on both sides, and JavaScript on the client and server invites the reuse of APIs where possible. The WebWorker, Indexed Database, and XHR APIs are promising to be enormously beneficial on the server side, and with excellent client server consistency. But still the server side requires special attention, and CommonJS is bringing the needed standards and conventions.

There have been a number of different areas that CommonJS has invested in for creating the necessary foundation for SSJS. The primary initial effort was in defining the module system, which provides an easy to use system for loading modules and their dependencies, while ensuring that each module is loaded only once. This module system is now very broadly implemented and extensively used by the other subsequent module APIs defined by CommonJS. The module system provides the essential basis for modular library and application development.

CommonJS is working on building up the low-level I/O components, defining the necessary binary data types, the APIs for file interaction, and encoding. There are proposals in place for unit testing, and a number of modules to provide concurrency support. And finally there is a HTTP interface specification known as JSGI.

Concurrency

One of the most central characteristics of a runtime is the concurrency approach. The inherent dangers of shared memory threading are widely understood, and the browsers wisely avoid exposing developers to the low level complications of shared-memory threads, alternately providing shared nothing event-loop concurrency. With the latter approach, data structures are not exposed to preemptive threads that could cause race conditions. Instead, events are queued up, and deterministically executed. Concurrent operations are handled with WebWorkers, in separate memory spaces, with the ability to communicate to other workers with message passing.

The CommonJS group defines APIs, so it is beyond the scope of CommonJS to dictate the concurrency model used by servers, but there is general consensus recommending that JavaScript-based servers utilize shared-nothing event loop concurrency, and CommonJS APIs are built to support this model. The CommonJS wiki includes proposals for exposing the WebWorker API through a worker module, and for exposing controls for the event loop with the event-queue module.

Promises

One key abstraction that is extremely useful for event-based systems is a promise. A promise is an abstraction that encapsulates the execution of an asynchronous action and its subsequent value that it returns. A promise can be thought of as representing the result of an asynchronous computation. The promise will receive the value returned by the call when it is finished, and acts as a placeholder for the returned value. Promises are critical to developing large scale applications that make heavy use of asynchronicity. Using callback-passing for asynchronous actions does not compose well. It is impossible to properly encapsulate functionality, switching an implementation from synchronous to asynchronous breaks APIs and creates complex flows of passing callbacks around to handle return values. With promises, values can be returned just like synchronous functions, and callers can listen for asynchronous results without requiring additional callback-passing APIs.

The promise API in CommonJS is an asynchronous handling system based on the collective experience of the promises in Dojo, Twisted, and ref_send. The ref_send promises focus on secure immutable promises, but lacks a reasonable API for promise implementors. Dojo provides a promise API (called Deferreds) with convenient chaining support, but has suffered from the use of mutable promises, with side-effect inducing callbacks complicating the data flow. The promise API combines the best of both, providing secure immutable promises with a very simple, easy to implement API for implementors, and full chaining support. The promise API in CommonJS provides a foundation that implementors can easily extend to provide the convenience functions from Dojo’s Deferred API as well as ref_send’s static operators.

An example of using CommonJS promise might look like:

requestSomeData("http://example.com/foo") // returns a promise for the response
    .then(function(response){ // 'then' is used to provide a promise handler 
        return JSON.parse(response.body); // parse the body
    }) // returns a promise for the parsed body
    .then(function(data){
        return data.price; // get the price
    }) // returns a promise for the price
    .then(function(price){ // print out the price when it is fulfilled
        print("The price is " + price);
    });

JSGI: HTTP Gateway

The JavaScript gate interface (JSGI) API has been asymptotically reaching consensus in the CommonJS group, as a standard API for serving HTTP requests through JavaScript application servers. The JSGI provides a very easy to use API for responding to HTTP requests, influenced by the design of WSGI and Rack. A basic JSGI application looks like:

app = function(request){
    return {
        status: 200,
        headers: {},
        body: ["Hello World"]
    };
};

JSGI has been intentionally designed to make it easy to write “middleware”, modules that exist at different places in the request handling chain to add various forms of functionality. Middleware can include support for cross-domain requests, logging, error handling, special method handling, mapping URLs to different applications, and much more.

Most existing HTTP interfaces were designed around a synchronous model, and have had an extremely difficult time adapting to the asynchronous model, which is essential for efficient Comet support. JSGI on the other hand, has been developed from the ground up with asynchronicity in mind, both to support Comet, and to work properly in an event-based environment. JSGI achieves support for asynchronicity while still maintaining a simple function call and return style by leveraging promises. Here we can see the power of promises, JSGI applications can return a promise to indicate that the response is not yet complete, and no additional callback parameter need to be passed around. With promise support throughout, an asynchronous JSGI app can be as simple as:

app = function(request){
    return doSomethingAsynchronous().then(function(result){
        return {
            status: 200,
            headers: {},
            body: ["Result ", result]
        };
    });
};

Since JSGI effectively provides a JSON/JavaScript representation of the HTTP protocol, JSGI has powerful opportunities for use in communication outside of just HTTP server handling. JSGI can be used as an API for making HTTP requests as a client (as a promise based alternative to XHR), and can be used as an inter-process/inter-worker protocol. HTTP is almost certainly the most commonly used communication protocol, and the RESTful power of HTTP can easily be leveraged for worker communication with JSGI’s JSON-based format (that doesn’t require HTTP parsing).

Projects

CommonJS and JSGI are purely API specifications, not implementations. Lets look at some projects that implement these specifications.

Narwhal/Jack

Probably the most advanced and mature CommonJS implementation is Narwhal plus Jack. Basically all of the CommonJS APIs are implemented in Narwhal/Jack and/or their forks. Narwhal provides extensive I/O support, a large collection of useful library modules, and a great package management system called tusk with package resource handling built into its module loader. Narwhal is designed for running on various JavaScript platforms and has an intelligently architected layer of abstractions for dealing with different engines.

Narwhal has the most complete engine support for Rhino. Rhino is almost certainly the most popular JavaScript engine for server side JavaScript due to its seamless integration with Java. With Rhino, the virtually limitless expanse of Java libraries are available and easily accessible. While great efforts are being made to important libraries in other engines, the breadth of libraries that are available through the Rhino’s Java bridge is not likely to be matched anytime soon. It also comes with a good visual debugger. Rhino is also perhaps the most feature-rich JavaScript implementation available, as it implements most of the ECMAScript 5 specification and Mozilla’s JavaScript 1.8 features.

Narwhal also runs on JSCore, the JavaScript engine used by Safari. JSCore is a very high-performance engine (second only to V8, perhaps), and consequently there is great potential for high-performance applications running on Narwhal/JSCore.

Jack is a JSGI implementation that has been developed to run on Narwhal, and is also built to run on various web server technologies. Jack has been somewhat of the de facto reference implementation for JSGI, and runs on Simple, CGI, servlet containers (including Jetty and GAE), and others. Older versions of Jack relied on shared memory threading, but with the latest work on Jack, requests are handled with full event loop based shared-nothing concurrency, with request delegation across multiple workers for efficient multi-core utilization.

Node

Node is an exciting project in the SSJS realm, and has received a lot of publicity recently, for good reason. Node builds on the amazingly fast V8 engine and provides important low-level functionality. V8 is one of the fastest dynamic language engines on the planet. Node uses libev for an event-loop based asynchronous I/O model, providing the same type of event-loop shared-nothing concurrency used in Narwhal. However, Node has had a stronger emphasis on trying to provide asynchronous APIs for all I/O operations. By exposing almost all I/O operations as asynchronous events, Node rarely has to block while waiting for I/O, and is able to execute much more efficiently than spreading workloads across multiple blocking threads. The performance advantages to event-loops over threads are well-documented. This focus, in combination with the blazing fast V8 engine, promises to place Node as the performance king of dynamic language based application servers.

There are certainly gaps that need to be filled with Node. Node implements CommonJS’s module system, but does not implement other CommonJS APIs, and has been slow to add better support for interoperability. Node runs its HTTP server and handling on a single process/thread, which means it can’t utilize multi-cores yet (support for that is coming). Node does not support the JSGI specification; however, it is possible to run JSGI on Node with the JSGI-Node adapter that I built.

There are other important things like package support and module reloading that are definitely needed. From my experience, Node does not feel as complete as Narwhal. However, despite these concerns, Node has a lot of potential, and is an exciting project. Node primarily fulfills a role as a low-level JavaScript platform, on which higher level functionality can be built. The Node community is responsive. There is an effort to run Narwhal on top of Node, which would be a powerful combination.

Ejscript

Ejscript is a JavaScript platform that extends the JavaScript language with many new constructs. Many of these additions are based on the ECMAScript 4 specification that was eventually abandoned because many implementors felt it was too extensive of a revision of the language. However, many of the features included in ECMAScript 4 that Ejscript implements were technically excellent additions, and may be included in ECMAScript 6. Beyond extra language features, Ejscript boasts a very high performance engine with remarkably low memory footprints, claiming performance that rivals Node with less than a tenth of the memory consumption.

Persevere

Persevere was one of the first projects to implement the CommonJS module specification. However, the primary goal of the Persevere project is to provide consistent end-to-end data semantics leveraging RESTful architecture. Persevere 1.0 provided CommonJS module support to provide interoperability with other CommonJS libraries, but being a JavaScript platform is beyond the primary scope of the project. With the new array of CommonJS platforms, future Persevere development will now focus on building on top of these new platforms and providing its data persistence, security, REST architecture, Comet notifications, and advanced web communication capabilities for multiple platforms.

Others

Mozilla’s SpiderMonkey engine is also a capable engine for server side JavaScript. Flusspferd is a project to add low-level system functionality to SpiderMonkey and couples with Zest as a JSGI server and Juice as a web framework on top.

There are number of other important server-side projects including HelmaNG, with high-quality technology that predates CommonJS and now are adapting to work in harmony with the CommonJS ecosystem, but I won’t cover all of them.

Web Frameworks

Some exciting new web frameworks are building on these technologies such as Nitro, Bogart, and JuiceJS. Also some platforms come with their own MVC framework such as HelmaNG, and EJScript. However, for many the move to JavaScript has been motivated by investment in rich client interfaces that already have presentation capabilities (the V and C in MVC), so a more modern REST-style, “thin server architecture” may be a better fit than traditional server-side MVC frameworks for many next generation JavaScript-based web applications. Pintura is the core of the next generation of Persevere, a cross-platform REST-style architecture web framework that is designed specifically for Ajax-style applications that maintain presentation logic on the client separate from the server-side storage concerns. We will cover Pintura in the next article of this series.

Conclusion

The combination of improvements and maturation of ECMAScript standards, with the new line of astonishingly fast JavaScript, the Ajax driven interest in JavaScript, the collaborative efforts of CommonJS standards, and the emerging array of server-side JavaScript projects are all coming together to propel server-side JavaScript to the forefront of web application development. It is not unreasonable to expect that the CommonJS APIs implemented with a web stack of the new line of SSJS projects could be the most important new web platform of the next decade.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Android has the first mover advantage of being deployed today to many devices, but to really get the most out of it, you really should develop using Java, or employ a toolkit like PhoneGap. Chrome OS offers the promise of using web technologies that are popular today, but is not yet production-ready, or optimized for mobile devices like Palm’s webOS.

The long-term expected convergence from Google makes sense. Convergence of Palm’s webOS with Chrome OS makes sense to us as well as they are both pushing towards having a very similar WebKit-based operating system for delivering web applications. Palm already appears to be moving in the direction of converging with Chrome through their adoption of key technologies in webOS such as Google’s V8 JavaScript engine. There’s nothing confirming that this will happen, other than it just making sense.

While there is a lot of short term interest in apps, you can still get significant results from mobile web apps, and the gap between a native app and a web-based app will quickly shrink, as evidenced by apps like Pie Guy. After all, web apps don’t require app store approval!

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

The rest of iTunes remains a custom desktop user interface, but it would not surprise me to see iTunes 10 be completely rendered using open web techniques. We already see competing products like Songbird using open web technologies for rendering media players, and it could eventually lead to a version of iTunes that lives inside Mobile Me, with songs stored in the Apple cloud.

Given Apple’s recent push on efforts like Time Machine and Mobile Me, it seems like Apple is working hard towards the important goal of making it very easy for users to not lose their data and work, and moving the iTunes franchise to the web would seem to be the next logical step!

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

I recently posted a proposal for an addition to JSON Schema, called JSON Hyper Schema, for defining the properties of a JSON structure that represent links or references within data structures. This is intended to provide the same linking capabilities of JSON Referencing, but in a much more flexible manner such that schemas can be used to describe link information in existing data structures without requiring a fixed convention. I wanted to exposit one of the further benefits of using this type of schema: satisfying the goals of namespacing in JSON.

Many useful mechanisms and conventions have been created in the world of JSON based on the prior similar work in the XML world, such as JSON-RPC and JSON Schemas. Done properly, JSON counterparts have used some of the best ideas from XML and learned from the mistakes. Namespacing is an important concept from XML, and occasionally there have been discussions considering the potential need for namespacing in JSON. XML’s namespacing is indeed valuable in its capability to provide distributed definition of meanings from multiple parties, and this goal is of value in JSON as well. JSON Hyper Schema provides a mechanism for defining hypertext/hyperlink semantics within JSON data structures for URI-based linking that forms a foundation that is capable of achieving these goals.

A naive import of the ideas from XML might simply attempt to apply the semantics and even syntax of XML to JSON. Such an approach fails to account for the fact that JSON is a very different paradigm than XML, fundamentally created for simple object oriented data structures. In order to properly move forward with namespacing in JSON, rather than looking at the syntax in XML, lets look at the goals of namespacing. First, XML and JSON do both share a similar idea in that they provide means for defining the meaning of entities within their structure with names. In XML, tag names are used to give meaning to the contents of elements, and attribute names give meaning to the attribute values. Likewise in JSON, property names give meaning to the property values. The primary goal of namespacing is to build on this to allow distributed definition of meanings from various parties. The key here is the authoritative meaning can be ascribed from different groups beyond that which can be attained from just a short opaque string (tag name or property name).

JSON Schema provides a natural means for this goal. One of the basic functionalities provided by the JSON Schema specification is to define the meaning and nature of properties within an instance JSON data structure. A property definition is a schema with the purpose of describing the value of a property. JSON Schema defines a number of attributes that can be used to define the type of the value, the constraints on the value, and a description of the value, all of which come together to ascribe cumulative meaning to properties of JSON objects.

The base JSON Schema format for describing properties forms the foundation of namespacing’s goal of distributed definition by providing a format for the definition, but alone does not provide the means for linking to the authoritative definition. Distributed efforts almost always are reliant on a global registry for coordination of different parties. When the challenges of this are considered the solution is URIs. This is the basis of the web, XML namespacing, and is the only reasonable means for global referencing to resources. Here JSON Hyper Schema provides the means to associate URIs with definitions such that they can be universally referenced. JSON Hyper Schema’s self-descriptive nature means that agents can determine URIs for every property definition unambiguously. With the JSON Hyper Schema specification to lean on, JSON structures can indicate the link to an authoritative definition of the meaning of its properties.

JSON Hyper Schemas can be referenced from instances by Link headers or media type parameters. A simple example illustrates how JSON properties have universally locatable definitions:

Content-Type: application/json; schema=http://www.book-warehouse.com/book-schema
[
  {"title": "Oliver Twist", "price": 16.99},
  {"title": "Robinson Crusoe", "price": 15.99}
]

This message gives the authoritative URI for the schema. With a look at the schema, we can see how each property has a corresponding definition with an authoritative URI as well:

Content-Type: application/schema+json; schema=http://json-schema.org/hyper-schema
{
  "properties": {
     "title": {"type": "string", "description": "The title of the book"},
     "price": {"type": "number", "description": "The price of the book in US"},
  }
}

Since JSON Hyper Schema specifies how fragment identifiers can be used to reference individual entities in the schema using dot delimited property names, we can construct a full URI for the definition of the title and price properties. The title’s definition URI is http://www.book-warehouse.com/book-schema#properties.title and the price’s definition URI is http://www.book-warehouse.com/book-schema#properties.price. Just as with XML’s namespaces, each JSON namespace provides a fully qualified URI for each property.

Collisions

An oft cited motivation for namespacing is collision avoidance. When simple identifiers are used for element and property names without coordination, it is easy to see how different parties can use the same name with conflicting meanings. However, while it is easy to theorize about potential collisions, when collision avoidance techniques are used, a more frequent and serious issue than collisions can arise: missed “collisions”. Missed collisions may sound good, but when two parties use the exact same identifier (remember there is pretty large domain of words that are available), more often than not they actually mean the same thing. Blindly avoiding collisions misses many opportunities for shared use and meaning applied to entities. For example, suppose we were combining schemas for information on books from another party at http://uk-books.com/schema/book:

{
  "properties": {
     "title": {"type": "string", "description": "The title"},
     "author": {"type": "string", "description": "The author"},
     "price": {"type": "number", "description": "The price in pounds"}
  }
}

Here the title property clearly has the same meaning as the first schema. If a namespacing technique was used that blindly separated the title properties into distinct properties, we would need to specify the title twice in an instance structure. The surface level affect is the obvious redundancy of data, but ignoring DRY principles can have more serious consequences in trying to maintain data synchronization. If the same title value was stored in two properties, someone could easily update one and not realize they needed to update the other one.

A simple look at linguistics illustrates the value of shared vocabulary. Our ability to communicate with others within our region is based on the fact that we have mutually accepted meanings for words. If individuals in a society refused to accept or understand words that were not defined by each individual or even by their family, it would undermine the process of gradually reaching mutual consensus on the usage and meanings of words that is necessary for the development and evolution of a language. Effective communication across a substantial population relies on this convergence of meanings.

The key point here is that property definitions with identical names that are combined from different sources should default to being treated as the same property. Mapping these definitions to separate properties to avoid collisions should be treated as the exception rather than the rule.

Collisions (really avoiding them)

But still, exceptions do happen, and sometimes different parties do use identical names for properties that really have different meaning and need different values. The hyperlinking capabilities that JSON Hyper Schema affords can still be used to solve this situation. When we need to avoid collisions we can map our own property names to specific property definitions from various schemas to create a composite schema with explicitly chosen names. In these two schemas, one price property is expecting a price in US dollars, and the other in UK pounds. We can create a composite schema:

{
  "extends": {"$ref": "http://www.book-warehouse.com/"},
  "properties": {
     "ukPrice": {"$ref": "http://uk-books.com/schema/book#properties.price"},
  }
}

We now have created a new schema that maps the property with the name “ukPrice” to the property definition from the uk-books.com domain of http://uk-books.com/schema/book#properties.price.

Alternately, one could also map property names to property definitions from different schemas to combine property definitions that may have different property names in their containing schemas.

Extensible Schemas

One particular form of schemas that can benefit from authoritative definitions are meta-schemas. Schemas are a quintessential example of data that can benefit from additional properties beyond those defined by the actual JSON Schema specification. JSON Schema defines a number of properties for schemas, but it can be very convenient to include extra data on schemas, such as information about suggested visual layout, persistence information, and associated class implementation information. With JSON Hyper Schemas, one can define a meta-schema that defines the structure and additional properties of your schemas, giving a concrete URL-backed meaning to the properties you add to your schema. For example, we could define a meta-schema:

http://some-site.com/deprec-schema:

Content-Type: application/schema+json; schema=http://json-schema.org/hyper-schema
{
  "extends": {"$ref": "http://json-schema.org/hyper-schema"},
  "properties": {
     "deprecated": {
        "type": "string",
        "description": "When present, this indicates that the associated property \
             is deprecated. The schema value provides information on what \
             property should be used instead".
      }
  }
}

Now we can use the “deprecated” property in our own schemas:

Content-Type: application/schema+json; schema=http://some-site.com/deprec-schema;
{
  "properties": {
     "name":{"type": "string", "deprecated": "Please use 'title' instead"},
     "title":{"type": "string", "description": "The title of this document"},
     ....
  }
}

Namespacing, the JSON way

JSON Hyper Schema is a sufficient building block upon which we can realize the goals of namespacing, namely, meanings from distributed parties can be authoritatively ascribed to elements of a data structure with property name sharing and conflict resolution control. What is elegant about this approach is that basically nothing needs to be modified in the actual instance data structures themselves. URI attribution is done through media type parameters in combination with schemas. The actual data stays unchanged, it is still the original simple, easy to read, compact JSON. JSON namespacing not only fits with JSON technically, but it fits in spirit: data is simple and light.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Mashable has summarized Microsoft’s position that IE6 cannot die until Windows XP dies, even though Microsoft strongly encourages users to upgrade.

We recognize that IE6 was the best browser on the market when released — in 2001. We, also understand that many organizations have invested significantly, buying and building apps on proprietary, non-forward compatible technology such as ActiveX or extensions to IE6. While extensions are not inherently evil, they have a tendency to lock you into a single-vendor solution, which may not be supported in future versions of their product, in this case IE7 and later.

This is one of the reasons that we’re proponents of open source software. We build web applications using free and open source works such as Dojo, which we help adapt and update for new browsers as they are released, to prevent getting stuck with untenable solutions. If you’re looking to unshackle yourself from the confines of an IE6-strapped web application and empower your users to participate in the joys of modern-day web browsing, contact SitePen for a free 30-minute consultation today.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Consider the following:

PhoneGap XUI

“We hear your words. Why another JavaScript framework?! When development of PhoneGap was under way we noticed slow load times for modern JavaScript frameworks (such as Prototype, MooTools, YUI, Ext and (yes) even jQuery. A big reason why these libraries are so big is because is mostly they contain a great deal of cross browser compatability code. The mobile space has less browser implementations (so far) and different needs. Thus XUI.

XUI strives to be a framework for first class mobile device browsers such as WebKit, Fennec and Opera with future support under consideration for IE Mobile and BlackBerry.”

Joe Hewitt’s iUI

“First and foremost, iUI is not meant as a “JavaScript library”. Its goal is simply to turn ordinary standards-based HTML into a polished, usable interface that meets the high standards set by Apple’s own native iPhone apps.”

John Resig’s war on War on Browser Sniffing

“As of 1.3, jQuery no longer uses any form of browser/userAgent sniffing internally – and is the first major JavaScript library to do so.

Browser sniffing is a technique in which you make assumptions about how a piece of code will work in the future. Generally this means making an assumption that a specific browser bug will always be there – which frequently leads to code breaking when browsers make changes and fix bugs.

Instead we use a technique called feature detection where we simulate a particular browser feature or bug to verify its existence. We’ve encapsulated all the checks that we use in jQuery into a single object: jQuery.support. More information about it, feature detection, and what this feature provides can be found in the jQuery.support documentation.”

Sizzle, a micro-toolkit for CSS Selectors that is now part of the Dojo Foundation

“Sizzle is a pure-JavaScript CSS selector engine designed to be easily dropped in to a host library.

A completely standalone selector engine (no library dependencies), Sizzle provides competitive performance for the most frequently used CSS selectors. At only 3KB minified and gzipped, it is highly extensible with an easy-to-use API.

Sizzle is designed for optimal performance with event delegation, provides meaningful error messages for syntax problems, uses a single code path (no XPath), uses no browser-sniffing (feature-detection only), and is Caja-compatible.”

TIBCO General Interface

GI builds an optimized code tree for each major browser release (the opposite of the approach jQuery 1.3 and others have taken, as code only makes it into a build for that browser if that browser is known to support that functionality).

dojox.gfx

GFX supports SVG, VML, Canvas, Silverlight, and Flash with the same simple procedural drawing API, so you can target native and plugin rendering environments with one simple API. In order to make this possible, dojox.gfx selectively loads only the code it needs based on the target rendering environment. Loading the code for all platforms would be prohibitively expensive from a performance perspective.

excludeStart(“webkitMobile”)

In recent discussions on the Dojo mailing list, discussions turned to how to make Dojo faster for mobile users. In most cases, this involves removing code and features that are not needed on that platform given the precious resources available on mobile devices and over mobile networks.

Alex Russell checked in some work he started last year that looks like this:

//>>excludeStart("webkitMobile", kwArgs.webkitMobile); 
!dojo.isIE &&  
//>>excludeEnd("webkitMobile"); 

The syntax is a bit verbose, but it excludes the code between the start and end in a Dojo build created for webkitMobile:

./build.sh profile=base action=clean,release webkitMobile=true

While we would love to switch to pure feature detection over browser version detection like John has done with jQuery 1.3, version detection is generally more concise and precise, and does not necessarily make Dojo any less forwards compatible.

Version detection also makes it easy to exclude code created solely for browsers that require major workarounds, such as IE 6. To get the most out of this though would require doing something similar for each major browser, which would make Dojo more challenging to understand and maintain, would probably require a build step even during development (not just in production to improve performance).

At this point, there are no clear answers, just a lot of questions on how to create a toolkit that offers edge of the web features for desktop users, and still preserves performance for mobile web users. What approach do you think is best?

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Dojo is Undocumented

Far from it! Early on, Dojo’s only documentation was the source code. Today, we have a tremendous amount of information available, and we’re working to better organize and refine it.

To start, take a look at the Dojo QuickStart Guide, Dojo API Viewer, and Dojo Campus Documentation Project (the forthcoming replacement for the venerable Dojo Book).

Want offline docs? Check out the Dojo Toolbox for now, and soon the Dojo Campus Documentation Project.

Need a book? There are now four great dead tree Dojo books to choose from.

Need a great how-to or walthrough? Try one of these in depth Dojo tutorials.

Want to see demos? Look at the Dojo Campus Feature Explorer or the Dojo Spotlight.

Need hands-on Training? Contact SitePen for assistance.

Will Dojo continue to simplify and improve its documentation? Absolutely, and we welcome your help and feedback.

Dojo is Slow

Any JavaScript library can be slow if used incorrectly. In the days of Dojo 0.4, it was too easy to create apps that were slow by having the parser on by default, and include extraneous files and source code.

Today, the parser is a problem unique to Dijit. Dojo != Dijit, and today you only accept the performance hit of a parser (one that is much less than the days of 0.4) when you need a full-featured widget system.

For the past 18 months, Dojo has made it easy to create things with Dojo that are lean and fast, and any functionality that was in Dojo 0.4 is at least 100% faster in Dojo today, and some things are 1000% faster or more. In measuring performance with querying, dojo.query is as fast if not faster than other major leading toolkits with the SlickSpeed test suite.

And Dojo offers a plethora of performance optimization techniques and suggestions to squeeze every last KB and millisecond out of your source code. Reliable, cross-browser performance is a data-driven art that requires leaving many assumptions at the door, listening to data, and making compromises as necessary.

Dojo is Bloated, Larger and More Complex than Prototype, jQuery, MooTools, YUI, etc.

Dojo is designed to give you the necessary tools to solve complex engineering problems, while keeping the simple things simple. This myth exists simply because we use a package system, and we expect developers to leverage a build tool in production code to optimize performance. If you download the full Dojo source, you will see many files. Out of all of those files, you only need to include a single one with a script tag in your markup.

We make it easy to get started by just including dojo.js from AOL or Google’s CDN services. We call dojo.js “Dojo Base”. It is 26KB (gzipped) in size, and comparable in size, features, performance, and functionality to other major toolkits.

Dojo Base provides an extremely rich and lean set of functionality useful to any web developer. I think you’ll be hard-pressed to find more functionality and performance in less KB.

If 26KB is too much, you can even get a stripped down version of Dojo in just 5.5KB, which represents the module loading framework for Dojo, and then add back just the exact features you need. You can also take the entire source Dojo SDK and reduce it down to what you need.

In the browser, every feature adds a slight performance hit, so Dojo values the flexibility necessary to include just what you need, and no more, while offering a wide amount of features in a coherent, organized manner. If you want to use more of Dojo’s functionality, it’s completely optional, and you just dojo.require the extra features you need. The class structure in most cases just follows the logical path structure of your project. For example, dijit.foo.Bar would be expected in /dijit/foo/Bar.js.

The simple provide and require namespace system system makes it easy to use Dojo code and your code alike. This makes it possible to easily create advanced, powerful applications as well as simple features such as the Dojo Flickr Badge.

The Dojo Package system, while adding a bit of complexity that takes two minutes for new developers to learn, makes it much more readable and organized, and provides you with the ability manage your code when script includes and dependencies otherwise become unwieldy. Dojo is a library, and behaves as such. Like Java and many other languages, we automatically include only the bare minimum amount of code, with plenty of other packages that you have to include manually.

There are frameworks within the various libraries inside Dojo, the most widely-known one being Dijit. The Dijit widgets have dependencies on the framework code. These sort of long dependency chains are rare within Dojo, but exist when needed to provide more involved functionality.

Dojo Requires Java

No, not at all—even though the full Dojo SDK does include a few jar files.

In addition to browser-based capabilities, Dojo provides completely optional Rhino-based tools to optimize your code in a build step to improve performance. Rhino is the Mozilla Foundation’s JavaScript engine implemented in the Java programming language.

Why would you ever want to “build” JavaScript? To reduce the size of your source code, to optimize caching, and reduce the calls to the server. For example, you can merge CSS rules together, and take HTML resources used by widgets and turn them into JavaScript strings to reduce the number of HTTP requests, overall file size, and more.

Dojo is Verbose and is “Java for JavaScript”

The only thing in Dojo that resembles Java is our desire to include things in namespaces. Our library structure still encourages brevity, with Dojo Base features living in the dojo.* namespace. Dojo prefers dojo.byId and dojo.query over $, though it is easy to roll your own Dojo bling function if you prefer!

Dojo prefers putting everything in namespaces to reduce our footprint in the global namespace, to allow Dojo code to coexist with other libraries and user code without naming conflicts. Also, Dojo minimizes function overloading, so using full APIs helps with code maintenance.

For brevity, anything that is widely used typically makes its way into the top level namespace in a short and easy to remember manner, e.g. dojo.connect and dojo.attr. The terseness of our API and namespace structure tends to surprise people who haven’t used Dojo since 0.4 or earlier.

Each component is a distinct “thing” and our only options are to address this in the global namespace which would create conflicts, or to hang it off of Dojo. Admittedly, some projects and toolkits don’t have namespaces simply because they don’t provide enough code to need namespaces.

Java developers do like Dojo (or DWR or even GWT or Ext), most likely because of the breadth, depth, and organization of the Dojo Toolkit APIs.

Much of the inspiration for Dojo is derived from Python, AOP, PHP, Java, and a variety of functional programming language constructs like hitch, mixins, delegation, and more. However, we explicitly avoid using any patterns from other languages, and we make great efforts to make our JavaScript leverage the patterns unique to JavaScript.

Toolkit constructs such as dojo.declare() may be considered Java-like, though most libraries have some sort of constructor and inheritance model. Dojo takes advantage of the flexibility of JavaScript rather than trying to provide a domain specific language on top of JavaScript.

Dojo is only Useful for Enterprise Apps and not for Small Sites or Blogs

This myth exists because Dojo was not as small and simple in the Dojo 0.4 days as it is now.

Dojo Base at 26KB gzipped or Dojo Mini down to 5.5KB, both with the same easy to use API, make it possible to do simple progressive enhancement or unobtrusive JavaScript, handle events, Ajax, animations, DOM, querying, and more.

Dojo is Not for Single-Page Apps

The developers from Cappuccino, Objective-J, and Sprout Core argue that they created their efforts because other toolkits are not optimal for true applications in the browser. In our opinion this is rubbish.

Dojo is used widely in feature-rich applications by a variety of companies, such as Sun Convergence, Cisco’s WebEx Connect and IBM’s Project Zero. Dojo is also very widely used for creating applications that live inside the firewall in what I call Ajax Dark Matter.

Dojo is not for Multi-Page Apps and Web Sites

While it’s unlikely that anyone believes both this myth and the previous one, this viewpoint is based on perspective. Given the perceived bloat and toolkit size, some people have suggested that Dojo not be used for multi-page sites, but Dojo’s extensive optimization options make this usage scenario a snap. The Eye-Fi Manager and even the Dojo Foundation web site itself show how snappy these apps and sites are.

Dojo is Just About Widgets

Dojo has a highly flexible, and extremely fast widget system called Dijit. This is a completely optional package for people needing to use and create their own widgets. It is designed to handle a variety of use cases in a modular manner, including accessibility, internationalization, layout, containment, templating, and much more. The underlying concept is that each widget is simply an HTML template, a CSS template, and a JavaScript file for logic.

Many of our users never even touch Dijit.

There are other ways to get lightweight reusable behavior on the fly, including Dojo’s implementation of the Django Templating Language (DTL).

Interest in Dojo is Waning

Why do people think this? Perhaps it is from badly argued stats. People do things like compare Google stats for searches like “jQuery JavaScript” “Prototype JavaScript” “Dojo JavaScript”, etc. because the words on their own conflict with general search terms. Lies, damn lies, and statistics, right?

While interest in other great toolkits has also grown significantly, the usage of Dojo continues to grow aggressively, based on the number of deployed applications, downloads, site traffic, books sold, and a variety of other usage statistics. By all appreciable measures, interest in Dojo is growing faster than ever.

Ajax has become increasingly more popular, and JavaScript is now considered a real language. Dojo has always treated JavaScript as a first-class language, and has a stricter focus on utility than on one particular aspect of the workflow, such as the DOM.

Dojo Doesn’t Validate

A long standing complaint is that Dojo doesn’t validate. DTD != valid HTML. Custom attributes have always been valid HTML, they just don’t validate when tested against a DTD.

If DTD validation is truly important to your project, it is simple to use Dojo with graceful degradation as well; the use of custom attributes is an implementation pattern only, not a hard requirement for using Dojo. Or you can always create a DTD.

In addition to performance benefits and efficiencies of custom attributes, they are a lot friendlier and human-friendly than other options. For example, some toolkits use the rel attribute, and stuff it with data. jQuery’s popular meta plugin looks like this: “<div class=”hilight { background: ‘red’, foreground: ‘white’ }”>”. Why is this better than this non-real, but Dojo style markup: “<div jQueryType=”hilight” background=”red” foreground=”white”>”?

The HTML specification states that any attribute not recognized is to be ignored by the HTML rendering engine in user agents, and Dojo optionally takes advantage of this to improve ease of development.

Dojo is Ugly

Creating beautiful web applications is about having a great sense of style, and interaction/user experience design capabilities. Dojo provides 3 professionally designed themes: Tundra, Soria and Nihilo. If you would like to make them look better, or create a new theme, then get involved!

Dojo’s themes are deliberately understated to make it easier to drop components in alongside existing designs and brands. But at SitePen, in our work with clients we’ve applied our CSS and design skills to Dojo-based applications to create excellent designs and amazing experiences.

Many other companies are also succeeding with theming their Dojo-based applications.

Dojo Doesn’t Offer Certain Features Provided by a Particular Library

This may be the case, or it might be called something different or available through a different API style. But Dojo offers extreme flexibility. For example, check out Peter Higgins work to add the missing jQuery API footprint he likes to Dojo in less than 1 KB.

Dojo Doesn’t Work with a Particular Toolkit, Environment or Server

We go out of our way to make sure Dojo works with everything, even when a toolkit or server doesn’t behave the way we’d like. If something is not working for you, just ask on the Dojo Forums or through SitePen Support, and if there’s a bug, file a ticket and it will get fixed ASAP. We’ve gone to great lengths to make sure Dojo works well with popular toolkits, as well as in a variety of environments including: XUL, command-line, Jaxer, AIR, etc., support for initiatives by the OpenAjax Alliance, as well as integration with DWR, Persevere, Zend Framework, IBM Websphere, Django, Ruby On Rails, and much, much more.

Dojo was initially started to stop reinventing the DHTML toolkit wheel, with the original Dojo source code based significantly on the work of earlier toolkits such as netWindows, BurstLib, and f(m).

The People at Dojo Don’t Like a Particular Project

In general, on an interpersonal level, we’re friends with the people creating jQuery, Prototype, YUI, MooTools and more. While Dojo developers of course have differences of opinion in the best approach to development (great developers should disagree, as long as they keep the debate focused on merit rather than the form of the message), we find that fans of Dojo and other toolkits attempt to make things a bigger rivalry than toolkit authors do. After all, we are talking about open-source toolkits that are liberally licensed (BSD, AFL, MIT, Apache, etc.), so it’s hard for us to not get along. Alex Russell, Peter Higgins and I are also on record as inviting other toolkits to collaborate and cooperate to again stop reinventing the wheel.

In many cases, this is a matter of perspective. I know that many of us get bugged because so much of the mud flinging happens not because someone’s done something wrong, but because features of a particular toolkit are criticized because the person doing the criticizing doesn’t understand why those features even exist in a particular toolkit.

It is difficult to contribute to Dojo or Get Involved

Unlike most other toolkits, Dojo requires a CLA which protects your IP rights, and also requires that when you contribute code, you have the right to contribute it. The Dojo Foundation does this to ensure that we are able to redistribute every piece of code in the Toolkit completely free and with liberal licensing. It’s a simple process that should take at most fifteen minutes, and it is well worth the effort. For more information about getting involved, contributing, donating, visit the Dojo Foundation and Dojo Toolkit web sites, or visit the Dojo IRC channel on irc.freenode.net in #dojo.

Dojo lacks Commercial Support

A number of companies including SitePen offer Dojo development, support and training services. The Dojo community offers great free support, but when an issue goes beyond what is reasonable to ask of a volunteer, or needs to occur immediately and/or under an NDA, companies like SitePen are available to help you get productive, from fixing a bug in Dojo to building and designing your entire application.

Summary

As you can see, Dojo has come a long way since its initial creation 4+ years ago. Over the next several months, Dojo developers will continue to improve the toolkit source code, documentation and marketing to make Dojo easy to use regardless of the type of application or web site you are developing.

I hope you will make the decision to use, or not use, Dojo based on merit, rather than on myths or information that is no longer or was never accurate.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Although the lecture didn’t really focus in any specific product categories, the overall concept seemed to be aimed at electronic consumer devices, social networks, and the kind of fast & fun web 2.0 applications that are popping up like mushrooms. It got me thinking: If playfulness has value, it stands to reason that play could be incorporated into more serious contexts as well. What about products that are notoriously unplayful?

Some companies are finding playful success in the technological gaps that users expect to be dehumanizing, such as the usually stale e-commerce purchasing process. Woot.com, the ubiquitous retailer of zero-day bargains, has made an art of infusing a pretty standard process with a lot of nudges, winks, and playful jibes. For example, instead of just telling customers “your order has been completed” after checkout, Woot does a good job of humanizing the interaction, telling the customer that everything appears to have gone smoothly. Order history pages are populated with playful language such as “What’s going on here?” as well as a tongue in cheek nods to their business model, e.g. “Hey, here’s a fun game: try to get the longest list of purchased woots of anyone! Better get started though, some people have a pretty good start on you. If you win, we’ll be really happy for you.” Perhaps because Woot treats its customers like a friend with whom the site plays around, Woot’s customer base tends to be loyal, engaged, and unsurprisingly, just as playful towards Woot.

Another fun example is Twitter. When Twitter gets flooded, the server responds with a playful image that is far from serious, but incredibly endearing.

Some other playful examples:

• Flickr uses the same playful strategy for humanizing error messages. Although Flickr usually has solid uptime, when Flickr is down for servicing, users can expect tongue in cheek error messages such as “Flickr is having a Massage.” Sometimes it is the little things that count. Many Web 2.0 companies have followed Flickr’s great lead.
• Pownce: Makes not finding pages a little more fun.
• Renkoo: a site focused on event planning, that offers the popular BoozeMail Facebook app, has an offline message about the service having a hang-over, and an invalid page error message from the Renkoo Bots: “Oh these silly humans. Little do they know that we are destroying their precious webpages. With lasers!“

Other sites like Digg and YouTube have rather boring and unplayful error or away messages, though on the rare occasion when Digg is down, they offer a nice touch for their news-starved user base: a list of every Digg employee’s favorite blog.

These are all great little features, but there’s a bigger question lingering:

Can we work the same kind of playfulness into enterprise applications?

Playfulness in a business context would have to be executed carefully, but the goal remains the same: Engaging users and creating an enjoyable user experience.

Business applications are very task-oriented beasts. They are used to get things done, and that means that any included playfulness should either be helpful or at the very least not get in the way of users.

The current generation of open-ended games, including massive sandbox games like World of Warcraft and Grand Theft Auto, can really be boiled down to long chains of deliverables and task goals, the completion of which lead to psychological payoffs which have no real value outside of entertainment. It may be a little much to ask that our office apps be as fun as a car chase in a video game, but it seems like we could bootstrap the same ideas of challenge, feedback, and psychological payoff in order to instill some sense of playful gamesmanship in productivity software.

How could we work a playful context into a ho-hum application like a task manager / to-do list? Taking a cue from exploratory games, we’d want to tie in pleasurable feedback (e.g. mission complete!), some kind of progress indicator (two more goals left), and perhaps performance based recognition (level up).

One example of a playful infusion to an otherwise boring set of tasks can be seen at LinkedIn. In order to encourage users to fully complete their online profiles, LinkedIn suggests which sections of the user profile to fill in next and rewards users with a percentage-complete score. This is a much more fun interaction model than the standard, dry web form we might expect. This kind of model can inform the possibility of yoking playfulness in robust applications like our previously mentioned task manager. It also makes the network more useful overall by encouraging users to add more detail to their profiles over time.

Task priority ratings, difficulty estimates and due dates are already found in many task managers, so using this data to create playful feedback seems reasonably straight forward.

Our playful feature might manifest itself by keeping track of user progress and comparing the statistics across a historical data set. Did the user complete more goals today than their historical average? If so, let them know with a pleasant notification. Did the user just finish a task with a very high difficulty rating? A different message might be called for when the user checks that big task off their list. To tie this playfulness to our idea of noting progress, completion data and accomplishment notifications could be charted historically within the task manager for later reference. If a user is actually improving their historical performance regularly, that might call for another feedback cycle, preferably something engagingly visual, like a slick chart that shows their progress.

How these kinds of notifications are implemented can also make the difference between fun and lame. A sterile text message reading “You have completed 3 tasks today. Good work.” could be dull and even patronizing. The user might be more engaged by a quirky little animation with a distinctly non business message, congratulating the user on hitting a performance indicator, slogging through a challenging product launch, or achieving a sale that has been on the user’s calendar for a while.

It’s actually not that hard to imagine this kind of playfulness being useful from a business context. I could even envision a more social structure in which team members also get notified when someone is having a particularly productive day. If the members of a team all complete their current batch of tasks ahead of time, could the system give them a collective pat on the back ( or optionally inform their manager of excellent performance)?

Would there be value in assigning real-life bounties or rewards to certain tasks either explicitly or randomly? It’s tempting to consider the possibilities, though we’d want to make sure to avoid creating perverse incentives and unintentional stressors. Above all, a playful context always means -positive- interaction. If it edges into negativism, playfulness just turns into pressure.

There might also be value in engaging in some purely playful, non-functional elements. These could be features that activate based on predefined conditions and which do nothing other than offer the user a chance to smile or to be surprised. These elements would be discovered spontaneously by users, and should be clever enough to spur an inter-office email along the lines of “Hey, check this out!”

For example, imagine building a playful egglet into a text editor. For example, let’s consider an egglet that provides useless but fun feedback based on typing speed. When the user starts typing at a very high rate, a small egglet in the form of a speed gauge would appear at the bottom right of the interface, possibly along with a subtly implemented sound, like that of a revving engine. What’s the point? None. It’s purely there as playful fun, a way for us to tell the user “wow, you sure are typing fast.” As long as it is executed tactfully and doesn’t actually keep the user from accomplishing his goal, the egglet can add an element of quirky humor to an otherwise sterile software tool.

The danger here is overshooting the idea of playfulness and delving into the simply annoying. Some companies just can’t control themselves, resulting in too-clever-by-half abominations like Clippy, Microsoft’s infamous Office Assistant.

The trick is to offer something that is playful, entertaining, but only slightly obtrusive. Bonus points if the playful feature ends up being useful or offers valuable feedback. If the playful feature is purely for entertainment purposes, its probably best to give users a quick and easy way to permanently turn the feature off. Learn from Clippy’s mistakes and remember that cute and annoying have a lot in common.

The idea of playfulness is really just to give an experiential gift to the end user. In a world with increasingly dehumanizing elements and rampant focus on efficiency, genuine efforts to do something extra for the user are usually appreciated. The extent to which playfulness can be incorporated into applications without becoming annoying is a matter of art. Erring on the side of quality instead of quantity is probably a safe bet, but a nudge and a wink towards end users can be the perfect way to ask “Want to play?”

More often than not, they probably do.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!