define(, , );

This simple API can be used in a variety of different ways for different situations.

Anonymous Modules

The recommended general purpose way of writing your own modules with this API is to omit the first argument, and just include the dependencies and the module factory. A simple module can be written:

define(["math"], function(math){
  return {
    addTen: function(x){
      return math.add(x, 10);
    }
  };
});

The first argument defines the set of dependencies (as module names) for this module. The modules exported values are passed in as the arguments to the callback function once the dependent modules have been loaded.

When the module name (of the current module) is omitted, as in the example, the module is anonymous. This is a powerful way to write modules since the module source code is not coupled to any identity. The module can easily be moved to a different location without requiring any code change. This technique follows basic DRY principles, avoiding duplicate information about the identity of a module (the filename/path information does not need to be duplicated in code). This not only makes writing modules easier, but gives greater flexibility in module reuse.

Let’s look at how we load this module from a web page. We will assume the above module was put in a file called adder.js. Using RequireJS, we can load our module like this:

Once the initial entry require is called, RequireJS takes care of downloading all the dependencies of adder. We can use this same require call to load any of the module forms described below as well.

There are some rules for anonymous module usage. There may only be one anonymous module per file, and the anonymous modules must be loaded through the loader (you can just include a <script> pointing to the module).

Data Wrapping: The New JSON-P

Modules that only provide data, or dependency free methods can simply provide an object as the only argument to define():

define({
  name:"some data"
});

This is similar to JSON-P, but actually provides a significant advantage over JSON-P since it can be written in a static file without requiring the dynamic callback generation. This makes this format cacheable and CDN friendly.

CommonJS module wrapping

CommonJS modules can easily be used with the asynchronous module loader by simply wrapping them with a define call. Here you can also omit the first and second parameter, and the module’s require calls will be scanned to determine the appropriate dependencies. For example:

define(function(require, exports){
// standard CommonJS module:
  var math = require("math");
  exports.addTen = function(x){
    return math.add(x, 10);
  };
});

It is important to note that this form requires the module loader to inspect the function for require calls. The require calls must be written in the form of require("...") in order for this analysis to work properly. There are also certain browsers where this will simply not work (some versions of Opera mobile and PS3). This may not be an issue at all if your modules will be run through a build process before deployment. However, you can also wrap CommonJS modules, and manually specify dependencies. The specification allows us to also reference CommonJS variables, so we can include the standard “require” and “exports” variables:

define(["require", "exports", "math"], function(require, exports){
// standard CommonJS module:
  var math = require("math");
  exports.addTen = function(x){
    return math.add(x, 10);
  };
});

Full Module Definitions

A full module definition includes the module name, dependencies, and the factory. This form has the advantage that the module can be included in another file or it can be directly loaded with a script tag. This is the format generated by the build tools so that multiple dependencies can be combined in a single file. An example of this format:

define("adder", ["math"], function(math){
  return {
    addTen: function(x){
      return math.add(x, 10);
    }
  };
});

Finally, the last permutation of these arguments is to include the module id, but omit the dependencies list. This can be used when you want to indicate the module id (for use with direct script tag), but there are no dependencies. With no dependency array, the arguments default to “require”, “exports”, and “module”. We could alternately create our adder module:

define("adder", function(require, exports){
  exports.addTen = function(x){
      return x + 10;
  };
});

Again all these module forms can be handled by RequireJS and can be loaded by listing them as dependencies in another module or directly loading them with a require() call.

This simple API provides flexibility to declare modules for a variety of situations, from anonymous modules that can easily be moved around, to “built” modules that are ready to be used from script tags. This API is implemented by RequireJS and Dojo, as well Nodules for Node. You can also use Transporter to do on-the-fly concatenation of AMD modules and automatic wrapping of CommonJS modules that are delivered in this format.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

The NoSQL database approach is characterized by a move away from the complexity of SQL based servers. The logic of validation, access control, mapping querieable indexed data, correlating related data, conflict resolution, maintaining integrity constraints, and triggered procedures is moved out of the database layer. This enables NoSQL database engines to focus on exceptional performance and scalability. Of course, these fundamental data concerns of an application don’t go away, but rather must move to a programmatic layer. One of the key advantages of the NoSQL-driven architecture is that this logic can now be codified in our own familiar, powerful, flexible turing-complete programming languages, rather than relying on the vast assortment of complex APIs and languages in a SQL server (data column, definitions, queries, stored procedures, etc).

In this article, we’ll explore the different aspects of data management and suggest an architecture that uses a data management tier on top of NoSQL databases, where this tier focuses on the concerns of handling and managing data like validation, relation correlation, and integrity maintenance. Further, I believe this architecture also suggests a more user-interface-focused lightweight version of the model-viewer-controller (MVC) for the next tier. I then want to demonstrate how the Persevere 2.0 framework is well suited to be a data management layer on top of NoSQL databases. Lets look at the different aspects of databases and how NoSQL engines affect our handling of data and architecture.

Architecture with NoSQL

In order to understand how to properly architect applications with NoSQL databases you must understand the separation of concerns between data management and data storage. The past era of SQL based databases attempted to satisfy both concerns with databases. This is very difficult, and inevitably applications would take on part of the task of data management, providing certain validation tasks and adding modeling logic. One of the key concepts of the NoSQL movement is to have DBs focus on the task of high-performance scalable data storage, and provide low-level access to a data management layer in a way that allows data management tasks to be conveniently written in the programming language of choice rather than having data management logic spread across Turing-complete application languages, SQL, and sometimes even DB-specific stored procedure languages.

Complex Data Structures

One important capability that most NoSQL databases provide is hierarchical nested structures in data entities. Hierarchical data and data with list type structures are easily described with JSON and other formats used by NoSQL databases, where multiple tables with relations would be necessary in traditional SQL databases to describe these data structures. Furthermore, JSON (or alternatives) provide a format that much more closely matches the common programming languages data structure, greatly simplifying object mapping. The ability to easily store object-style structures without impedance mismatch is a big attractant of NoSQL.

Nested data structures work elegantly in situations where the children/substructures are always accessed from within a parent document. Object oriented and RDF databases also work well with data structures that are uni-directional, one object is accessed from another, but not vice versa. However, if the data entities may need to be individually accessed and updated or relations are bi-directional, real relations become necessary. For example, if we had a database of employees and employers, we could easily envision scenarios where we would start with an employee and want to find their employer, or start with an employer and find all their employees. It may also be desirable to individually update an employee or employer without having to worry about updating all the related entities.

In some situations, nested structures can eliminate unnecessary bi-directional relations and greatly simplify database design, but there are still critical parts of real applications where relations are essential.

Handling Relational Data

The NoSQL style databases has often been termed non-relational databases. This is an unfortunate term. These databases can certainly be used with data that has relations, which is actually extremely important. In fact, real data almost always has relations. Truly non-relational data management would be virtually worthless. Understanding how to deal with relations has not always been well-addressed by NoSQL discussions and is perhaps one of the most important issues for real application development on top of NoSQL databases.

The handling of relations with traditional RDBMSs is very well understood. Table structures are defined by data normalization, and data is retrieved through SQL queries that often make extensive use of joins to leverage the relations of data to aggregate information from multiple normalized tables. The benefits of normalization are also clear. How then do we model relations and utilize them with NoSQL databases?

There are a couple approaches. First, we can retain normalization strategies and avoid any duplication of data. Alternately, we can choose to de-normalize data which can have benefits for improved query performance.

With normalized data we can preserve key invariants, making it easy to maintain consistent data, without having to worry about keeping duplicated data in sync. However, normalization can often push the burden of effort on to queries to aggregate information from multiple records and can often incur substantial performance costs. Substantial effort has been put into providing high-performance JOINs in RDBMSs to provide optimally efficient access to normalized data. However, in the NoSQL world, most DBs do not provide any ad-hoc JOIN type of query functionality. Consequently, to perform a query that aggregates information across tables often requires application level iteration, or creative use of map-reduce functions. Queries that utilize joining for filtering across different mutable records often cannot be properly addressed with map-reduce functions, and must use application level iteration.

NoSQL advocates might suggest that the lack of JOIN functionality is beneficial; it encourages de-normalization that provides much more efficient query-time data access. All aggregation happens for each (less frequent) write, thus allowing queries to avoid any O(n) aggregation operations. However, de-normalization can have serious consequences. De-normalization means that data is prone to inconsistencies. Generally, this means duplication of data; when that data is mutated, applications must rely on synchronization techniques to avoid having copies become inconsistent. This invariant can easily be violated by application code. While it is typically suitable for multiple applications to access database management servers, with de-normalized data, database access becomes fraught with invariants that must be carefully understood.

These hazards do not negate the value of database de-normalization as an optimization and scalability technique. However, with such an approach, database access should be viewed as an internal aspect of implementation rather than a reusable API. The management of data consistency becomes an integral compliment to the NoSQL storage as part of the whole database system.

The NoSQL approach is headed in the wrong direction if it is attempting to invalidate the historic pillars of data management, established by Edgar Codd. These basic rules for maintaining consistent data are timeless, but with the proper architecture a full NoSQL-based data management system does not need to contradict these ideas. Rather it couples NoSQL data storage engines with database management logic, allowing for these rules to be fulfilled in much more natural ways. In fact, Codd himself, the undisputed father of relational databases, was opposed to SQL. Most likely, he would find a properly architected database management application layer combined with a NoSQL storage engine to fit much closer to his ideals of a relational database then the traditional SQL database.

Network or In-process Programmatic Interaction?

With the vastly different approach of NoSQL servers, it is worth considering if the traditional network-based out-of-process interaction approach of SQL servers is truly optimal for NoSQL servers. Interestingly, both of the approaches to relational data point to the value of more direct in-process programmatic access to indexes rather than the traditional query-request-over-tcp style communication. JOIN style queries over normalized data is very doable with NoSQL databases, but it relies on iterating through data sets with lookups during each loop. These lookups can be very cheap at the index level, but can incur a lot of overhead at the TCP handling and query parsing level. Direct programmatic interaction with the database sidesteps the unnecessary overhead, allowing for reasonably fast ad-hoc relational queries. This does not hinder clustering or replication across multiple machines, the data management layer can be connected to the storage system on each box.

De-normalization approaches also work well with in-process programmatic access. Here the reasons are different. Now, access to the database should be funneled through a programmatic layer that handles all data synchronization needs to preserve invariants so that multiple higher level application modules can safely interact with the database (whether programmatically or a higher level TCP/IP based communication such as HTTP). With programmatic-only access, the data can be more safely protected from access that might violate integrity expectations.

Browser vendors have also come to similar conclusions of programmatic access to indexes rather than query-based access in the W3C process to define the browser-based database API. Earlier efforts to provide browser-based databases spurred by Google Gears and later implemented in Safari were SQL-based. But the obvious growing dissatisfaction with SQL among developers and the impedance mismatches between RDBMS style data structures and JavaScript style data structures, has led the W3C, with a proposal from Oracle (and supported by Mozilla and Microsoft), to orient towards a NoSQL-style indexed key-value document database API modeled after the Berkeley DB API.

Schemas/Validation

Most NoSQL databases could also be called schema-free databases as this is often one of the most highly touted aspects of these type of databases. The key advantage of schema-free design is that it allows applications to quickly upgrade the structure of data without expensive table rewrites. It also allows for greater flexibility in storing heterogeneously structured data. But while applications may benefit greatly from freedom from storage schemas, this certainly does not eliminate the need to enforce data validity and integrity constraints.

Moving the validity/integrity enforcement to the data management layer has significant advantages. SQL databases had very limited stiff schemas, whereas we have much more flexibility enforcing constraints with a programming language. We can enforce complex rules, mix strict type enforcements on certain properties, and leave other properties free to carry various types or be optional. Validation can even employ access to external systems to verify data. By moving validation out of the storage layer, we can centralize validation in our data management layer and have the freedom to create rich data structures and evolve our applications without storage system induced limitations.

ACID/BASE and Relaxing Consistency Constraints

One aspect of the NoSQL movement has been a move away from trying to maintain completely perfect consistency across distributed servers (everyone has the same view of data) due to the burden this places on databases, particularly in distributed systems. The now famous CAP theorem states that of consistency, availability, and network partitioning, only two can be guaranteed at any time. Traditional relational databases have kept strict transactional semantics to preserve consistency, but many NoSQL databases are moving towards a more scalable architecture that relaxes consistency. Relaxing consistency is often called eventual consistency. This permits much more scalable distributed storage systems where writes can occur without using two phase commits or system-wide locks.

However, relaxing consistency does lead to the possibility of conflicting writes. When multiple nodes can accept modifications without expensive lock coordination, concurrent writes can occur in conflict. Databases like CouchDB will put objects into a conflict state when this occurs. However, it is inevitably the responsibility of the application to deal with these conflicts. Again, our suggested data management layer is naturally the place for the conflict resolution logic.

Data management can also be used to customize the consistency level. In general, one can implement more relaxed consistency-based replication systems on top of individual database storage systems based on stricter transactional semantics. Customized replication and consistency enforcements can be very useful for applications where some updates may require higher integrity and some may require the higher scalability of relaxed consistency.

Customizing replication can also be useful for determining exactly what constitutes a conflict. Multi-Version Concurency Control (MVCC) style conflict resolution like that of CouchDB can be very naive. MVCC assumes the precondition for any update is the version number of the previous version of the document. This certainly is not necessarily always the correct precondition, and many times unexpected inconsistent data may be due to updates that were based on other record/document states. Creating the proper update logs and correctly finding conflicts during synchronization can often involve application level design decisions that a storage can’t make on its own.

Persevere

Persevere is a RESTful persistence framework, version 2.0 is designed for NoSQL databases while maintaining the architecture principles of the relational model, providing a solid complementary approach. Persevere’s persistence system, Perstore, uses a data store API that is actually directly modeled after W3C’s No-SQL-inspired indexed database API. Combined with Persevere’s RESTful HTTP handler (called Pintura), data can be efficiently and scalably stored in NoSQL storage engines and accessed through a data modeling logic layer that allows users to access data in familiar RESTful terms with appropriate data views, while preserving data consistency. Persevere provides JSON Schema based validation, data modification, creation, and deletion handlers, notifications, and a faceted-based security system.

Persevere’s evolutionary schema approach leans on the convenience of JSON schema to provide a powerful set of validation tools. In Persevere 2.0, we can define a data model:

var Model = require("model").Model;
// productStore provides the API for accessing the storage DB.
Model("Product", productStore, {
  properties: {
    name: String, // we can easily define type constraints
    price: { // we can create more sophisticated constraints
      type: "number",
      miminum: 0,
    },
    productCode: {
      set: function(value){
        // or even programmatic validation
      }
    }
  },
  // note that we have not restricted additional properties from being added
  // we could restrict additional properties with:
  // additionalProperties: false

  // we can specify different action handlers. These are optional, they will
  // pass through to the store if not defined.
  query: function(query, options){
    // we can specify how queries should be handled and 
    //delivered to the storage system
    productStore.query(query, options
  },
  put: function(object, directives){
     // we could specify any access checks, or updates to other objects 
     // that need to take place
     productStore.put(object, directives);
  }
});

The Persevere 2.0 series of articles provides more information about creating data models as well as using facets for controlling access to data.

Persevere’s Relational Links

Persevere also provides relation management. This is also based on the JSON Schema specification, and has a RESTful design based on the link relation mechanism that is used in HTML and Atom. JSON Schema link definitions provide a clean technique for defining bi-directional links in a way that gives link visibility to clients. Building on our product example, we could define a Manufacturer model, and link products to their manufacturers:

Model("Product", productStore, {
   properties: {
      ...
      manufacturerId: String
   },
   links: [
     { 
        rel: "manufacturer",
        href: "Manufacturer/{manufacterId}"
     }
   ],
...
});

Model("Manufacturer", manufacturerStore, {
   properties: {
      name: String,
      id: String,
      ...
   },
   links: [
     { 
        rel: "products",
        href: "Product/?manufacterId={id}"
     }
   ]
});

With this definition we have explicitly defined how one can traverse back and forth (bi-directionally) between a product and a manufacturer, using a standard normalized foreign key (no extra effort in synchronizing bi-direction references).

The New mVC

By implementing a logically complete data management system, we have effectively implemented the “model” of the MVC architecture. This actually allows the MVC layer to stay more focused and lightweight. Rather than having to handle data modeling and management concerns, the MVC can focus on the user interface (the viewer and controller), and simply a minimal model connector that interfaces with the full model implementation, the data management layer. I’d suggest this type of user interface layer be recapitalized as mVC to denote the de-emphasis on data modeling concerns in the user interface logic. This type of architecture facilitates multiple mVC UIs connecting to a single data management system, or vice versa, a single mVC could connect to multiple data management systems, aggregating data for the user. This decoupling improves the opportunity for independent evolution of components.

Conclusion

The main conceptual ideas that I believe are key to the evolution of NoSQL-based architectures:

• Database management needs to move to a two layer architecture, separating the concerns of data modeling and data storage. Persevere demonstrates this data modeling type of framework with a web-oriented RESTful design that complements the new breed of NoSQL servers.
• With this two layered approach, the data storage server should be coupled to a particular data model manager that ensures consistency and integrity. All access must go through this data model manager to protect the invariants enforced by the managerial layer.
• With a coupled management layer, storage servers are most efficiently accessed through a programmatic API, preferably keeping the storage system in-process to minimize communication overhead.
• The W3C Indexed Database API fits this model well, with an API that astutely abstracts the storage (including indexing storage) concerns from the data modeling concerns. This is applicable on the server just as well as the client-side. Kudos to the W3C for an excellent NoSQL-style API.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Roy Fielding‘s REST architecture is broken down into seven constraints (and the four sub-constraints of the uniform interface). The individual concepts here are certainly not new, but collectively looking at these concepts as resource oriented programming may provide an interesting new perspective. I will also look at how these principles are exemplified in Persevere 2.0 in its object store framework, Perstore, and its web stack Pintura.

Client-Server

Of all the REST principles the first principle is most targeted at the network paradigm, and is the least compelling for advising programming. However, it does suggest the importance of an asymmetric relationship between modules with separation of concerns. When modules interact, one module should stick to the role of providing a service (acting as the server), and one should act as the consumer of the service. The roles and concerns of the two modules thus stay cleanly separated.

Stateless

If you follow computer science research, you are probably aware that one of the predominant focuses of modern programming languages is seeking to minimize the hazards of stateful imperative programming. Many developers are moving towards functional languages, or at least functional elements of languages, which perform computations on inputs, yielding outputs, rather than being defined as a series of state changes. Concurrency, security, scalability, and many other programming concepts become drastically simpler when mutating state is removed, and applications become much easier to develop. This directly corresponds with the stateless principle of REST. Maintaining transitive state information puts an enormous burden on modules in terms of scalability and complexity. Minimizing or eliminating state and using functional paradigms yields much better code.

When we consider the stateless constraint of REST, there is an important distinction to be made between “state” and “data” or “resources”. Since REST is clearly resource oriented, data/resources are obviously not something that we are trying to eliminate. When we are advising against state, we are advising against the use of transitive state information as the primary programming technique for developing and implementing application flow and logic. The resources and data to which an application provides a user interface are still mutable structures, but this interaction is carefully handled through the uniform interface (coming later).

Cache

Another key concept of a resource oriented approach is that it provides visibility for interaction. REST’s interaction explicitly defines cacheability and idempotence of actions. Certain operations are known to be safe and cacheable. In HTTP, GET requests fit this criteria. In standard object oriented programming conventions, a getter is safe and cacheable as well. If a uniform interface has been defined, one can define one or more retrieval function that are safely cacheable. The uniform interface must explicitly disclose which operations have side-effects. This can have very significant implications for creating well-optimized code. We can re-utilize local copies of data without re-calling retrieval functions if the functions are known to be cacheable. In the realm of purely opaque functions we are afforded no such opportunities (who knows if a function might have side-effects).

Getters should be safe and cacheable (and idempotent), and setters should be idempotent. Getters and setters provide an extremely powerful concept within programming. Distilling everything down to opaque calls hides important semantics from users that can distinctly alter their expectations and use of modules, but getters and setters expose these expectations in a way that benefits users. Developers can expect that object.foo can be evaluated multiple times with the same result unless the property has been altered. Setting object.foo is an idempotent operation (i.e., it can be safely repeated) and can be expected to alter the value returned by object.foo (if successful).

Uniform Interface

The most central concept of REST is the uniform interface, often characterized by HTTP’s standard methods: GET, PUT, POST, DELETE, etc. as the means for navigation and interaction. These methods have clear semantics. GET is safe, PUT, POST, DELETE have side-effects. GET, PUT, and DELETE are idempotent, POST is not. PUT is supposed to update a resource in a way that predictably alters the expected response to a GET.

These methods have analogies in object programming with getters and setters as we have discussed; however a uniform interface can be more than simply getters and setters. The uniform interface does not act solely on the internal state of individual objects, but can affect the collection of some type of objects, providing means for creating and deleting objects, querying for objects, retrieving by unique identifier, and even locking and unlocking the objects.

Perhaps one excellent example of the uniform interface principle in JavaScript is Dojo‘s Data API. While it doesn’t utilize the traditional HTTP-style set of verbs, this API does follow the uniform interface principle and provides a single generic uniform interface through which all widgets can access data, and the data source can implement this interface to provide generic access to various different storage systems or file formats.

The W3C is working on a compelling new API called the indexed database API that could provide a more broadly accepted generic uniform interface for JavaScript, and even bears quite a bit of resemblance to the traditional HTTP verbiage for verbs, with get(), put(), and delete() methods. This API is used by Persevere 2.0 in it’s object store framework, Perstore, and further leveraged by its web stack Pintura. With this interface, various different storage systems, especially NoSQL servers, can very easily be modeled and leveraged at different layers within the Pintura/Perstore ecosystem.

The uniform interface is further broken down into four important sub-constraints.

Identification of Resources

A fundamental concept of REST is that resources must have a unique identifier which can be used to retrieve that resource (a URL). The identifier must be unique and carry sufficient information to locate the resource. If the identifier is only unique within a given subsystem, additional information must be added to the identifier if it is referenced outside the subsystem. This is in fact how URLs work. The paths are unique to a server, but the path is combined with the server host name to make them unique across the entire Internet. When applied to programming, resources are objects that are identifiable. Within an application runtime, it is often sufficient to simply use the server unique identifier, or even within the scope of certain modules use an identifier specific to a given table or data store.

Within Perstore, all resource objects must have an identifier property or getId() function that returns the identity of the object. Resource objects can be retrieved using the store.get(id). These mechanisms guarantee a way to discover a resource’s identity and retrieve the resource by the identity.

Manipulation of Resources through Representations

REST defines a distinction between resources and representations. Resources are not necessarily simple files. Resources may represent abstract concepts such people, computers, events, and more, and may be stored and accessed through various means such as database records. In the scope of network communication, representations provide a means for describing these resources using byte-level serialization. REST recognizes that different consumers may need or prefer different types of representations, and supporting multiple representations for a given resource can be highly valuable.

In intra-machine programming, of course modules can interact with higher level constructs than byte-level data, and can use objects, numbers, strings, and language constructs. However, there still may be multiple ways of representing an abstract resource. These can be particularly important if different data consumers expect data to be structured in different ways.

One of the ways that the abstraction between a representation and a resource can be achieved is with facets. I have previously discussed how facets are a powerful tool for securely controlling access to resources in the object capability model. Facets act as a wrapper for an object store, and can not only attenuate access, but can provide alternate “views” of data/resources. For example, facets can easily be used for localization. You could create one facet for a set of resources that provides methods and properties for English speakers, and then another set of methods and properties for French speakers. The facet could include any necessary logic for pulling data from different data sources based on locale. This use of facets is a clear demonstration of separation of representation/view and resource.

Self-descriptive Messages

Self-descriptive messages describe how we view and interact with a resource. There are a couple aspects of this principle. First, we should be able to easily introspect a representation of a resource to comprehend the structure and content, and use this information to intelligently update the object. Objects with properties (or hash/map style name-value pairs, which is the same thing in JavaScript) are well suited for this since the introspection clearly implies how the object can be updated. Self-descriptive objects are easier to interact with because generic tools can interact with them. The API for interacting with the object is evident from the object itself.

Self-descriptive messages also means that in addition to the raw representation of a resource, we should also be able to view additional metadata that describes extra information about the resource such as cacheability, attribution information, etc. Here, Pintura defines a getMetadata() function for resource objects that provides access to metadata information for resources and their representations. This makes metadata readily available for processing conditional requests (based on modification times or etags) and defining caching levels.

Hypermedia as the Engine of Application State

The ability to navigate resources through hyperlinks/hypermedia is the hallmark of REST. This is easy to relate to object oriented programming where property values can be references to other objects or getters can return other objects, making it easy to navigate through data structures. We frequently utilize object mapping to maintain these types of object references from relationships indicated by underlying data sources.

Perstore leverages JSON Schema’s hyperlink description capabilities to define links based on data. The JSON Schema implementation used by Perstore provides a getLink() function to the target of a resource link as defined in the schema definition. For example, we can define a model in Perstore:

var Model = require("model").Model;
Task = Model("Task", dataStore, {
  links: [
    {rel: "project", href: "/Project/{projectId}"}
  ]
});

And then navigate through the data:

var getLink = require("json-schema/links").getLink,
    someTask = Task.get("some task id");
someTask.projectId -> id of a the project it is part of
someTask.get("project") -> retrieves the target project through the defined link

getLink(“project”, someTask, Task) -> “/Project/project-id”

Layered

REST advises layering as the technique for composing functionality and scalability from multiple agents. In programming, layering can also be used, and is best described as wrapping or composition. With programmatic layering, one object “wraps” another source object and handles all messages before sending them on to the source object. The wrapper can then add functionality to the source object and still expose the same interface to object users.

The programmatic layering principle is also seen in middleware. The HTTP interface of middleware-centric web applications consists of layers of middleware, where each middleware acts as an HTTP intermediary, adding functionality along the way. Pintura, the HTTP interface for Persevere 2.0, is completely driven by this middleware-centric approach. Pintura is basically a set of middleware providing authorization, CSRF protection, content negotiation, and more.

Perstore makes extensive use of the layering approach as well. Perstore uses a single REST-style uniform interface object store API (with get, put, delete from the W3C Indexed DB API), that is used throughout the framework. Low level data storage adapters (for MongoDB, JSON files, CouchDB, SQL, etc), can implement this API, and then store wrappers can be layered on top to add additional functionality such as caching, subscription/notification, aggregation, replication, and adaptable-indexing. For example, the we can take a store and add notification and caching to it:

var Notifying = require("store/notifying").Notifying,
    Cache = require("store/cache").Cache,
    Memory = require("store/memory").Memory,
var enhancedStore = Notifying( // add notifications
   Cache( // add caching
      dataStore, // our original source store
      Memory())); // cache in memory

Perstore’s models and facets are layers themselves, also implementing the same interface, and provide schema validation and access control. These various layers are effectively an object store form of middleware.

Code on Demand

The Code on Demand principle provides a pathway to extensibility for applications. A traditional basic resource may simply be data, but this principle advises that resources may contain code (or be code entirely), providing language level flexibility for richer content than is possible for simple static data structures. As the LISP mantra goes, “code is data and data is code”. JavaScript can easily be used in homoiconic ways, supporting this paradigm well. The object literal syntax of JavaScript (the proper superset of JSON), is well adapted to include functions holding programmatic logic. By supporting storage of code/functions in databases, one can easily define individual resource-specific logic, and resource consumers can easily interface to resource functions as means for allowing resource-specific extensibility in applications.

Summary

These seven principles (and the four sub-principles of uniform interface) collectively integrate to form the powerful architectural style known as REST. This important network architecture can be applied as a programming style in resource oriented programming, and Persevere 2.0 provides a great framework for building applications with this methodology.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Continue reading

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Persevere’s new command-line script can be found in the Persevere directory at bin/persvr (or bin\persvr.bat for windows users). The first thing we will want to do is to make sure that the Persevere command-line script is available in the path. We can create a link to or add the Persevere bin directory to the environmental path.

sudo ln -s /path/to/persevere/bin/persvr /usr/bin

The first option for the Persevere command-line script we will look at is the server generator. We can create a new Persevere server instance by running:

persvr --gen-server myserver

Now we have generated a myserver server with exists in the subdirectory myserver. This subdirectory acts like a normal web directory when we run this application server. You can put HTML, JavaScript, CSS, and other files directly in the directory and they will be accessible from the web from this server. To start our new application server, we can run persvr from inside our new server directory:

cd myserver
persvr

Now let’s look at the directory structure of our new server directory. The generator creates a starting directory structure to help organize our application:

• css – For CSS files
• images – For images
• index.html – The root page that is accessible from /
• WEB-INF – This contains all the data, configuration, and server side code that should not be accessed directly as static files from the server. This directory contains these subdirectories:
  • config – This contains the Persevere configuration files which specify the data sources, schemas, and class methods for the data in Persevere
  • data – This is where the actual database files are stored
  • tests – This is where unit tests are placed

Persevere also includes two server instances in the distribution. The examples directory is a server instance with code samples. The core-tests is a server instance with Persevere’s own core unit tests.

Server Inheritance

While the newly created server instance consists of a virtually empty, clean, skeletal directory structure, Persevere provides a server inheritance mechanism so the static files and configuration information from the core Persevere directory are still available from the new server instance. This means that all the JavaScript client files and the database explorer are accessible from the server instance without any need to copy the files into the instance. The database explorer can be loaded from explorer.html in all instances, as well as the included Dojo files. Server instances also inherit the core data source, class and server-side JavaScript environmental settings from the core installation. Multiple instances can leverage a single Persevere installation for both server side configuration and library files. Therefore, the Persevere core can be updated in a single place for multiple server instances.

You can also add your own extra static files and configuration files to the core installation to be inherited by all instances. Whenever a resource file exists at the same relative path in both the server instance and the core directory, the server instance will give precedent to the instance resource. This makes it easy to override resources that would be inherited from the core. For example, one could create a blank explorer.html page to ensure that the database explorer is not accessible in a production app (although exposing the database explorer should not be a security threat, since security is enforced on the server in Persevere).

Unit Tests

Persevere provides an easy to use framework for writing unit tests for Persevere applications. To create a unit test, you can create a JavaScript file with tests in it and put it in the application’s WEB-INF/tests directory. The JavaScript file defines its unit tests with the tests method which takes an array of functions. When the unit tests are being run, each function in the array is executed, and if no exceptions are thrown, than the test passes. The test code can utilize the assert method and the assertEqual method as well. The assert method takes a single argument which when false, throws an exception. assert takes an optional second parameter that can include a message in the exception. The assertEqual method takes two arguments and throws an exception if they are not equal. For example:

// you can do test setup/initialization code here

tests([
	function testMyClass(){
		var myInstance = new MyClass();
		assert(myInstance instanceof MyClass);
		var returnVal = myInstance.inverse(33);
		assertEqual(returnVal, 1/33);
	}
]);

// any cleanup from the tests can be performed here

We can then run our test suite from the command-line:

persvr --tests

When Persevere is running the test suite, it uses a separate data directory. Rather than using the normal WEB-INF/data directory, the tests are run with WEB-INF/tests/data as the database directory. This makes it easy to run the unit tests on controlled test data. Often one of the biggest challenges of unit testing is dealing with a test database, but Persevere makes this completely automatic.

Other Command-Line Options

The Persevere command-line script includes other useful function as well, including defining the port number for the server, defining the database directory, and checking the version. To see a list of all the options, run the help command:

persvr -h

Conclusion

Persevere committer Liam Staskawicz is responsible for most of this new functionality. Persevere’s new command-line script, testing framework, and server generator provides a robust infrastructure for developing organized, deployable applications. Server instances have a clean structure that can be easily versioned and managed, complete with a built-in testing system.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Using the StoreExplorer module is intended to be very simple to hook up to a data store. Generally you would create a simple test page that you can use for store interaction. You create your store and then you can simply connect to it with the StoreExplorer:

The StoreExplorer utilizes the Dojo grid and tree widgets to view your data. The grid is automatically configured with appropriate columns for you data, and the tree can be used for view individual attributes of each item in the store and easily edit the values in place. The StoreExplorer also provides an interface for easily entering queries for the data store, and the grid provides sorting as well.

The tree part of the StoreExplorer allows you to introspect individual attributes of an item, and with hierarchical data stores, one can drill down into the nested items. Multiple values can also be viewed from the store as well as various other JavaScript data types like nested objects, dates, and functions. You can also modify data in the items from the tree view. Click on an attribute to change its value with the property editor. The property editor allows you to enter any type of JSON or JavaScript value, including strings, numbers, dates, arrays, etc. The editor also allows you define a value to a reference to another item in the store. Back in the tree view, you can right click on attributes in the tree to delete or add new attributes.

In the DojoX data tests directory (/dojox/data/tests/stores/) there is an explorer that is connected to an ItemFileWriteStore. You can use this as a starting template for connecting to your own data store.

It should be noted that connecting a tree directly to a data store will appear much different than the tree in the StoreExplorer. The tree in the StoreExplorer is not a per-item tree, but rather is a view of the attributes of each item, similar to the way that Firebug and other debuggers give you a view of the properties of an object. StoreExplorer is a developer tool. Also, the tree view is actually a self-contained module itself called the ItemExplorer. This could be used standalone to introspect a single store item.

Here is an example of using the StoreExplorer to interact with the ItemFileWriteStore:

Persevere’s Database Explorer

One of our other projects, Persevere, is now using the StoreExplorer as its foundation for its database explorer for inspecting tables and their data and metadata, complete with the ability to create complex deeply hierarchical objects, arrays, and even functions:

Front-end for Databases

The StoreExplorer provides a great base for developing a user interface to databases, particular databases with hierarchical capabilities. The StoreExplorer would make an excellent foundation for other databases as well. The StoreExplorer uses a self-configuring grid with a familiar table-like view of the data and has intuitive in-place editing for modifying data. This new module provides a great web-based database front-end, with a view of data that precisely aligns with the API that is available to developers through the corresponding Dojo Data store.

Store Explorer is another tool available for rapidly developing robust data-driven applications using Dojo’s data infrastructure. This tool can help test and interact with your data stores to help modularize the task of application development with the Dojo data framework.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

The Persevere server is an open source JSON application and storage server. Persevere pairs well with Dojo; the Dojo Data paradigm has a strong correlation to Persevere’s data storage structure. Virtually everything you can create or action you can perform with Dojo Data can be persisted in Persevere’s data storage including subobjects, arrays, circular references, and functions. Combining Persevere with Dojo allows you to rapidly develop simple database applications with create, read, update, and delete (CRUD) capabilities with minimal effort.

Starting Persevere

To get Persevere running, first download the latest version of Persevere. Next, unzip the Persevere zip file into the directory you want to use for Persevere. Finally, go to the directory where you unzipped Persevere and run:

java -jar start.jar

Persevere should start up as long as Java is installed on your system. You can now access Persevere in your web browser by visiting http://localhost:8080/.

Creating a Persevere Class

The basic container for storage in Persevere is a class; analogous to a table in relational databases. A Persevere class holds object instances, and class style structures can be defined for the object instances including methods and type definitions for the object properties. To create a Persevere class, first open the object browser at http://localhost:8080/browser.html?id=root. The object browser allows you to browse and navigate the objects in Persevere. To create a class, click the button (make sure no object is selected). A dialog will ask you for the name of your class and what class to extend. We will call the class “Product” and we will accept the default base class of “Object”. You now have data storage structure which can hold Product object instances.

Connecting with Dojo

Persevere is compliant with the HTTP/REST protocol, so the JsonRestStore is very effective with Persevere. However, dojox.data.PersevereStore is an extension of JsonRestStore specifically designed to simplify connecting to Persevere and take advantage of extra features of Persevere. This architecture is designed for a data store instance per server table. With the PersevereStore we can easily get a set of data stores corresponding to the server tables/classes:

dojo.require("dojox.data.PersevereStore");
var deferred = dojox.data.PersevereStore.getStores();

This deferred object will provide an object map of the stores when finished. We can get the data store for our Product table:

deferred.addCallback(function(stores){
	productStore = stores.Product;
});

The data store can be used with various widgets that support data stores. For example to use this table with the Dojo Grid:

grid.setStore(productStore)

or you can set the productStore as the store in the constructor. For example:

grid = new dojox.grid.Grid({
	store: productStore,
	query: "",
}, dojo.byId("gridId"));

With the ability to define any type of object structure, a Persevere data store can be used with virtually any widget. With the referencing capabilities it can be used hierarchically, so it is also well suited for use with the Tree widget.

Create, Read, Update, and Delete

Via Dojo, interacting with Persevere can be done directly with the Dojo Data API. To create our first object instance in the Product table:

paintballGun = productStore.newItem({name:"Paintball Gun",price:129.99});
productStore.save();

We can easily read properties using directy property access or the Dojo Data API:

paintballGun.price -> 129.99
productStore.getValue(paintballGun,"price") -> 129.99

We can update properties:

productStore.setValue(paintballGun,"price",119.99);
productStore.save();

Note that we do not need to define any type of schema, columns, or structure for a table ahead of time, we can simply dynamically create properties on objects that we create.

And delete the objects as well:

productStore.deleteItem(paintballGun);

One of the powerful aspects of Persevere is that you can dynamically persist a large variety of data structures. For example, we could save a sub object with manufacturer information:

productStore.setValue(paintballGun,"manufacturer",{
	name:"HotShots",
	rating:4,
	started: new Date(Date.parse("Jul 09 2002")),
	topProduct:paintballGun
});

In this example, we are saving an object as a value, and even including Dates, and a circular reference (back to the product object). Virtually anything you can create with JavaScript can be persisted to Persevere.

Querying

An essential aspect of database interaction is querying. Here we can use the standard query-object convention used by Dojo Data to search for objects. A query object is an object with name-values corresponding to the search filters to apply (and wildcards are supported). For example to find all the objects with a name that starts with “Paintball”:

productStore.fetch({
	query: {name: "Paintball*"},
	onComplete: function(results){
		results[0] -> paintballGun
	}
});

We can also utilize sorting and paging as well. To sort by price (lowest to highest) and return items 20-29:

productStore.fetch({
	query: {name: "*"},
	sort: [{attribute:"price",descending:false}],
	start: 20,
	count: 10,
	onComplete: function(results){
		...
	}
});

Persevere also supports JSONPath/JSONQuery queries, which have a much greater level of expressibility and can be used for more sophisticated queries. For example to search for all the items with a price less than $100 or a rating greater than 3:

productStore.fetch({
	query: "[?price<100 | manufacturer.rating > 3]",
	onComplete: function(results){
		...
	}
});

Live Data with Comet

Dojo and Persevere support HTTP Channels which allows for addition of live data updates through Comet notifications. Adding Comet capability is simple, just add the HttpChannels module after the PersevereStore has been loaded:

dojo.require("dojox.data.PersevereStore");
dojo.require("dojox.cometd.HttpChannels");

This is all that is necessary. Dojo will automatically subscribe to all data that is accessed, notifications will be delivered to the client, and the notifications will result in cached data updates and Dojo Data API notifications (which will update the user interface on notification aware widgets like the Grid). No additional coding is necessary to utilize live data updates.

Setting up Security

Out of the box, security is disabled in Persevere to make it easier to begin development. However, security is a key feature of Persevere, allowing Persevere to be safely used directly on the web, accessible from your JavaScript environment. Prior to deployment you should of course enable security, and it is also enabled once the first user account is created. To create a user account, go to the object browser (http://localhost:8080/browser.html?id=root), click on the sign in button , and choose to create a new user. This first user is given the administrative privileges for the system. All data is accessible to this user, but is read only by default for all other users and public access. Additional users may be created by the same process (or programmatically), but subsequent users must be granted access to data.

To enable higher level access for the public or other created users, you can select an object or table and click on the grant access button in the object browser. You can then enter “public” for public access, or a user name to enable write access to the data.

Conclusion

You can see these simple data interaction techniques in sample code in the customers example included in the Persevere download (which can be viewed locally at http://localhost:8080/examples/customer.html) There are numerous other important capabilities of Persevere including server side JavaScript execution through JSON-RPC, schema definition, prototype objects, cross-site referencing, accessing existing SQL databases, and more that are described in the Persevere documentation and we will explore in future tutorials. You should now have enough information to get a quick start to building database applications with Persevere using the oft-used create, read, update, delete, and querying operations, and easily plugging into Dojo widgets, with support for live data updates. You can start building applications almost instantly; no need to create schemas or table definitions ahead of time, simply start creating and building dynamic persistent objects in your data stores, and connect them to widgets for display. You can build your entire application, client and server, in JavaScript.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

The client/server model can be categorized into three parts:

• User Interface
• Business or Application Logic
• Data Management

Traditional web application development has distributed the implementation of the user interface across the network, with much of the user interface logic and code executed on the server (thin client, fat server). This has several key problems:

• Poor distribution of processing – With a large number of clients, doing all the processing on the server is inefficient.
• High user response latency – Traditional web applications are not responsive enough. High quality user interaction is very sensitive to latency, and very fast response is essential.
• Difficult programming model – Programming a user interface across client/server is simply difficult. When every interaction with a user must involves a request/response, user interface design with this model is complicated and error prone. The vast number of web frameworks for simplifying web development testifies to this inherent difficulty. Some have mitigated this difficulty to some degree.
• Increased vector of attack – Unorganized mingling of user interface code with business code can increase security risks. If access rules are distributed across user interface code, as user interface code grows and evolves, new vectors of attack emerge. With mixed code, new user interface features can easily create new security holes.
• Heavy state management on the servers – When client user interface state information is maintained by the server, this requires a significant increase in resource utilization as server side sessions must be maintained with potentially large object structures within them. Usually these resources can’t be released until a session times out, which is often 30 minutes after a user actually leaves the web site. This can reduce performance and scalability.
• Offline Difficulties – Adding offline capabilities to a web application can be a tremendous project when user interface code is predominantly on the server. The user interface code must be ported to run on the client in offline situations.
• Reduced opportunity for interoperability – When client/server communication is composed of transferring internal parts of the user interface to the browser, it can be very difficult to understand this communication and utilize it for other applications.

With the massive move of development to the web, developers have frequently complained of the idiosyncrasies of different browsers and demanded more standards compliance. However, the new major shift in development is towards interconnectivity of different services and mashups. We will once again feel the pain of programming against differing APIs. However, we won’t have browser vendors to point our fingers at. This will be the fault of web developers for creating web applications with proprietary communication techniques.

Much of the Ajax movement has been related to the move of user interface code to the client. The maturing of the browser platform and the availability of HTTP client capabilities in the XMLHttpRequest object, have allowed much more comprehensive client side user interface implementations. However, with these new found capabilities, it is important to understand how to build client/server applications.

So how do you decide what code should run on the client and what should the run on the server? I have mentioned the problems with user interface code running on the server. Conversely, running business logic and/or data management on the client is simply not acceptable for security reasons. Therefore, quite simply, user interface code is best run on the browser, and application/business logic and data management is best run on the server side. We can take a valuable lesson from object oriented programming to guide this model. Good OO design involves creating objects that encapsulate most of their behavior and have a minimal surface area. It should be intuitive and easy to interact with a well designed object interface. Likewise, client and server interaction should be built on a well-designed interface. Designing for a modular reusable remote interface is often called service oriented architecture (SOA); data is communicated with a defined API, rather than incoherent chunks of user interface. A high quality client/server implementation should have a simple interface between the client and server. The client side user interface should encapsulate as much of the presentation and user interaction code as possible, and the server side code should encapsulate the security, behavior rules, and data interaction. Web applications should be a cleanly divided into two basic elements, the user interface and the web service, with a strong emphasis on minimal surface area between them.

An excellent litmus test for a good client/server model is how easy is it to create a new user interface for the same application. A well designed client/server model should have clearly defined web services such that a new user interface could easily be designed without having to modify server side application logic. A new client could easily connect to the web services and utilize them. Communication should be primarily composed of data, not portions of user interface. The advantages of a clean client/server model where user interface logic and code is delegated to the browser:

• Scalability – It is quite easy to observe the significant scalability advantage of client side processing. The more clients that use an application, the more client machines that are available, whereas the server processing capabilities remain constant (until you buy more servers).
• Immediate user response – Client side code can immediately react to user input, rather than waiting for network transfers.
• Organized programming model – The user interface is properly segmented from application business logic. Such a model provides a cleaner approach to security. When all requests go through user interface code, data can flow through various interfaces before security checks take place. This can make security analysis more complicated, with complex flows to analyze. On the other hand, with a clean web service interface, there is well-defined gateway for security to work on and security analysis is more straightforward, holes can be quickly found and corrected.
• Client side state management – Maintaining transient session state information on the client reduces the memory load on the server. This also allows clients to leverage more RESTful interaction which can further improve scalability and caching opportunities.
• Offline applications – If much of the code for an application is already built to run on the client, creating an offline version of the application will almost certainly be easier.
• Interoperability – By using structured data with minimal APIs for interaction, it is much easier to connect additional consumers and producers to interact with existing systems.

Difficulties with the Client Server Model on the Web

There are certainly difficulties with applying the client/server model to the web. Accessibility and search engine optimization can certainly be particular challenges with the web services approach. Handling these issues may suggest a hybrid approach to web applications, some user interface generation may be done on the server to create search engine accessible pages. However, having a central architectural approach based around a client/server model, with extensions for handling search engines may be a more solid and future oriented technique for many complex web applications.

Our Efforts to Facilitate the Client Server Model

SitePen is certainly not alone in working to facilitate client/server architecture. However, since I am familiar with the projects we help create, I did want to mention our approaches to the client/server model:

DWR – From inception, DWR has provided an excellent framework for building client side user interfaces that can easily connect with server side business logic. DWR was years ahead of its time in establishing a framework that encouraged good client/server modeling. DWR has a solid structure for interacting with Java business logic objects. DWR has continued to progress, providing means for bi-directional Comet based communication (Reverse Ajax), and is adding more interoperability capabilities as well.

Dojo Toolkit – It should be obvious that building a good client-side user interface can benefit from a good toolkit, and Dojo has long provided just that. However, Dojo is more than just a library and set of widgets. Dojo provides real tools for building client/server applications. Dojo RPC can provides tools for connecting to web services, and can even auto-generate services based on SMD definitions. Dojo Data provides a powerful API for interacting with a data model. Dojo has lead the way with Comet technology, creating standards around browser based two-way communication. Recently we have built the JsonRestStore which allows one to connect to a REST web service and interact with it using the Dojo Data read and write API. This greatly simplifies the construction of user interfaces by simplifying the user interface-business logic interaction, and encouraging standards-based communication that can easily be used by others. Furthermore, Dojo provides comprehensive tools for robust data-driven applications; even templating can be done on the client instead of the server with Dojo’s DTL support.

The benefits of using standards-based client/server communication has facilitated integration with server frameworks like Zend, jabsorb, Persevere, and interoperability with other frameworks will be coming soon.

Cometd – Cometd provides real-time duplex communication between clients and servers. However, the distinguishing characteristic of the Cometd project is the focus on not only achieving Comet-style duplex communication, but doing so with an interoperable standard protocol, Bayeux. Comet uses a quintessential client/server approach. Any Cometd (Bayeux implementing) server can interact with any Cometd/Bayeux client. One can easily connect various different client implementations to a single server, by using the Bayeux standard.

Persevere – Persevere is a recently launched project, built with this service oriented client/server approach. Persevere is a web object database and application server with RESTful HTTP/JSON interfaces, allowing applications to quickly be built with a database backend that can be directly and securely accessed through Ajax. Persevere is focused on provided a comprehensive set of web services interaction capabilities through standard interoperable communication. Data can be accessed and modified with basic RESTful JSON interaction, clients can invoke methods on the server with simple JSON-RPC, and data can be queried with JSONQuery/JSONPath. With Dojo’s new REST data store, and SMD driven RPC services, Dojo clients can seamlessly build applications and interact with Persevere services using the Dojo APIs. Complex application logic can be added to the persisted objects in Persevere to facilitate building service oriented applications with a straightforward interface to the user interface code on the browser. Persevere is integrated with Rhino, so model and application logic can be written in JavaScript, providing a consistent language and environment for distributing client and server roles.

For more information about any of these open source projects, visit the SitePen Labs

Summary

As the web platform matures, as applications evolve to use more interactive and rich interfaces, and as web services increasingly interact, architecting web applications with an intelligent client/server model will become increasingly important. A properly designed client/server model will provide a foundation for modular, adaptable, and interoperable applications equipped for future growth.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

A new data querying tool for has been added to Dojo 1.2. JSONQuery is a new module intended to succeed and improve upon the JSONPath module introduced in Dojo 1.1. JSONQuery provides a comprehensive set of data querying tools including filtering, recursive search, sorting, mapping, range selection, and flexible expressions with wildcard string comparisons and various operators.

JSONQuery provides safe evaluation with language agnostic expressions that prevents arbitrary code execution. It also uses intuitive result-based evaluation that allows successive query operations. Furthermore, the new JSONQuery module provides significant performance improvements, with 20-100x faster execution with the common filter operation on large arrays than the JSONPath module. JSONQuery generally supersets the functionality of JSONPath and provides syntax that matches and behaves like JavaScript where the syntax intersects for maximum ease of use.

Usage API

A JSONQuery can be executed with the following call:

results = dojox.json.query(query,object);

Where query is the JSONPath query to execute and object is the root object or array to query. You can also create a “compiled” evaluation function that can be reused for multiple evaluations by only passing a query string:

evaluator = dojox.json.query(query);

The evaluator function can be then be performed on a query on data by calling it with the data as a parameter:

results = evaluator(object);

In situations where a single query may be executed multiple times, doing a single parse/compilation with dojox.json.query(query) and reusing the returned evaluation function will provide better performance. It is worth noting that dojox.json.query(query,object) is functionally equivalent to dojox.json.query(query)(object).

Query Syntax

JSONQuery evaluations begin with the provided object, which can referenced within queries with $. From the starting object, various operators can be successively applied, each operating on the result of the last operation. You may explicitly begin your query with $, but this is implicitly auto-inserted, so it is not necessary. This allows you to start queries with operators. JSONQuery uses syntax that is similar to JavaScript (with a number of extra operators), so a simple query could be performed like:

data = {foo:"bar"};
results = dojox.json.query("$.foo",data);
results -> "bar"

Query Capabilities

JSONQuery provides all the functionality of JSONPath, and additional query capabilities. By analogy, JSONQuery is to JSONPath, as XQuery is to XPath. In particular, JSONQuery provides the expressive equivalence of XQuery FLOWR expressions for mapping, sorting, and filtering object data. Several of the fundamental operators of JSONQuery follow the same syntax as JavaScript, as well behaving exactly as with JavaScript evaluation:

• .property – This will return the provided property of the object
• [expression] – This returns the property name/index defined by the evaluation of the provided expression

For example, the following JSONQuery will find the foo property of the 2nd item in the provided array (the $ is omitted):

[1].foo

The following operators are new in JSONQuery:

• [?expression] – This will perform a filter operation on an array, returning all the items in an array that match the provided expression. This operator does not need to be in brackets, you can simply use ?expression, but since it does not have any containment, no operators can be used afterward when used without brackets. The following JSONQuery will find all the array items that have a price less than 15:

  [?price < 15]

  And to add a condition for the rating property to be greater than 3 (and omit the brackets):

  ?price < 15 & rating > 3

• [/expression], [\expression], [/expression, /expression] – This performs a sort operation on an array, with sort based on the provide expression. Multiple comma delimited sort expressions can be provided for multiple sort orders (first being highest priority). / indicates ascending order and \ indicates descending order. For example to sort an array by lastName first and then firstName as the second priority:

  [/lastName,/firstName]

• [=expression] – This performs a map operation on an array, creating a new array with each item being the evaluation of the expression for each item in the source array. For example, to create a list of the price value from an array of objects, we could use the query:

  [=price]

  You can also use object literals and and create a new array of objects that with a name and price properties generated from the source array:

  [={price:price,name:firstName + " " + lastName}]

• expr = expr – Performs a comparison (like JavaScript’s ==). When comparing to a string, the comparison string may contain wildcards * (matches any number of characters) and ? (matches any single character). For example to find all objects in an array where the name starts with “Mr”, one could use the query:

  [?name='Mr*']

• expr ~ expr – Performs a string comparison with case insensitivity. For example to find all objects in an array with the word “the” in the description regardless of case:

  [?description~'*the*']

• ..[?expression] – This will perform a deep search filter operation on all the objects and subobjects of the current data. Rather than only searching an array, this will search property values, arrays, and their children.
• $1, $2, $3... – This can be used to reference additional parameters passed to the query call. For example:

  results = dojox.json.query("[?firstName=$1&lastName=$2]",
  					myData,"John","Doe");

  or it can be applied to the evaluator function:

  evaluator = dojox.json.query("[?firstName=$1&lastName=$2]");
  results = evaluator(myData,"John","Doe");

The following operators from JSONPath are also supported:

• [start:end:step] – This performs an array slice/range operation, returning the elements from the optional start index to the optional end index, stepping by the optional step parameters. For example to get the first ten items in an array:

  [0:10]

• [expr,expr] – The union operator returns an array of all the property/index values from the evaluation of the comma delimited expressions.
• .* or [*] – Returns the values of all the properties of the current object.
• $ – This is the root object.
• @ – This is the current object in filter, sort, and map expressions. Note that names are auto-converted to property references of the current object in expressions, but @ can be used for index access on the current object. The following queries are identical:

  [?name='Fred']
  [?@.name='Fred']
  [?@['name']='Fred']

• ..property – Performs a recursive search for the given property name, returning an array of all values with such a property name in the current object and any subobjects.
• +, -, /, *, &, |, %, (, ), <, >, <=, >=, != - These operators behave just as they do in JavaScript.

Multiple operators can be used successively to create complex queries. For example, to find all the objects from the array in the products property that have a price under 15 and then sort them by descending order of rating and show the first twenty items from the resultant list, we could query:

$.products[?price < 15][\rating][0:20]

Queries can use the regular operators to form general expressions based on more complex query operations. For example, to find the difference between the lowest priced item and the highest priced item in an array:

$.store.book[\price][0].price - $.store.book[/price][0].price

Summary

The new Dojo JSONQuery module provides a powerful tool for general purpose data querying, and can be used in variety of situations. The JSONQuery module is already used by Persevere to parse and execute queries in it’s server side JavaScript object storage environment. JSONQuery is a flexible and complete query format for handling large JSON/object data structures with an intuitive JavaScript-like syntax.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!