The dojox/socket module is designed to be simple, lightweight, and protocol agnostic. In the past Dojo has provided protocol specific modules like CometD and RestChannels, but there are numerous other Comet protocols out there, and dojox/socket provides the flexibility to work with virtually any of them, with a simple foundational interface. The dojox/socket module simply passes strings over the HTTP or WebSocket connection, making it compatible with any system.

The simplest way to start a dojox/socket is to simply call it with a URL path:

require(["dojox/socket"], function (Socket) {
	// Create socket instance
	var socket = new Socket("/comet");
});

The socket module will then connect to the origin server using WebSocket, or HTTP as a fallback. We can now listen for message events from the server:

socket.on("message", function(event){
	var data = event.data;
	// do something with the data from the server
});

Here we use the socket.on() event registration method (inspired by socket.io and NodeJS’s registration method) to listen to “message events” and retrieve data when they occur. This method is also aliased to the deprecated Dojo style socket.connect().

We can also use send() to send data to the server. If you have just started the connection, you should wait for the open event to ensure the connection is ready to send data:

socket.on("open", function(event){
  socket.send("hi server");
});

Finally, we can listen for the connection being closed by the server or network by listening for the close event. And we can initiate the close of a connection from the client by calling socket.close().

The dojox/socket method can also be called with standard Dojo IO arguments to initiate the communication with the server. This makes it easy to provide any necessary headers for the requests. For example:

var socket = new Socket({
	url:"/comet",
	headers: {
		"Accept": "application/json",
		"Content-Type": "application/json"
	}});

This will automatically translate the relative URL path to a WebSocket URL (using ws:// scheme) or an HTTP URL depending on the browser capability.

For some applications, the server may only support HTTP/long-polling (without real WebSocket support). We can also explicitly create a long-poll based connection:

var socket = new Socket.LongPoll({
	url:"/comet",
	headers: {
		"Accept": "application/json",
		"Content-Type": "application/json"
	}});

We can also provide alternate transports in the socket arguments object. This would allow us to use the get() method in dojo/io/script to connect to a server. However, a more robust solution is to use the dojox/io/xhrPlugins for cross-domain long-polling, which will work properly with dojox/socket.

Auto-Reconnect

In addition to dojox/socket, we have also added a dojox/socket/Reconnect module. This wraps a socket, adding auto-reconnection support. When a socket is closed by network or server problems, this module will automatically attempt to reconnect to the server on a periodic basis, with a back-off algorithm to minimize resource consumption. We can upgrade a socket to auto-reconnect by this simple code fragment:

require(["dojox/socket", "dojox/socket/Reconnect"], 
	function (Socket, Reconnect) {
	// Create socket instance
	var socket = new Reconnect(new Socket("/comet"));
});

Using Dojo WebSocket with Object Stores

One of the other big enhancements in Dojo is the Dojo object store API (which supercedes the Dojo Data API), based on the HTML5 IndexedDB object store API. Dojo comes with several store wrappers, and the Observable store provides notification events that work very well with Comet driven updates. Observable is a store wrapper. To use it, we first create a store, and then wrap it with Observable:

require([
	"dojo/store/JsonRest",
	"dojo/store/Observable"
], function(JsonRest, Observable){
	var store = Observable(new JsonRest({data:myData}));
});

This store will now provide an observe() method on query results that widgets can use to react to changes in the data. We can notify the store of changes from the server by calling the notify() method on the store:

 
socket.on("message", function(event){
  var existingId = event.data.id;
  var object = event.data.object;
  store.notify(object, existingId);
});

We can signal a new object by calling store.notify() and omitting the id, and a deleted object by omitting the object (undefined). A changed/updated object should include both.

Handling Long-Polling from your Server

Long-polling style connection emulation can require some care on the server-side. For many applications, the server may have sufficient information from request cookies (or other ambient data) to determine what messages to send the browser. However, other applications may vary on what information should be sent to the browser during the life of the application. Different topics may be subscribed to and unsubscribed from. In these situations, the server may need to correlate different HTTP requests with a single connection and its associated state. While there are numerous protocols, one could do this very easily be defining a unique connection and adding that as a header for the socket (the headers are added to each request in the long-poll cycles). For example, we could do:

require([
	"dojox/socket"
], function(Socket){
	var socket = Socket.LongPoll({
		url:"/comet",
		headers: {
			"Accept": "application/json",
			"Content-Type": "application/json",
			"Client-Id": Math.random()
		}});
});

In addition, dojox/socket includes a Pragma: long-poll to indicate the first request in a series of long-poll requests to help a server ensure that the connection setup and timeout is properly handled.

We can easily use dojox/socket with other protocols as well:

CometD

To initiate a Comet connection with a CometD server, we can do a CometD handshake, connection, and subscription:

var socket = new Socket("/cometd");
function send(data){
  return socket.send(json.stringify(data));
}
socket.on("connect", function(){
  // send a handshake
  send([
    {
       "channel": "/meta/handshake",
       "version": "1.0",
       "minimumVersion": "1.0beta",
       "supportedConnectionTypes": ["long-polling"] // or ["callback-polling"] for x-domain
     }
  ])
  socket.on("message", function(data){
    // wait for the response so we can connect with the provided client id
    data = json.parse(data);
    if(data.error){
      throw new Error(error);
    }
    // get the client id for all future messages
    clientId = data.clientId;
    // send a connect message
    send([
      {
         "channel": "/meta/connect",
         "clientId": clientId,
         "connectionType": "long-polling"
       },
       {  // also send a subscription message
         "channel": "/meta/subscribe",
         "clientId": clientId,
         "subscription": "/foo/**"
       }
    ]);
    socket.on("message", function(){
      // handle messages from the server
    });
  });
});

Socket.IO

Socket.IO provides a lower-level interface like dojox/socket, providing simple text-based message passing. Here is an example of how to connect to a Socket.IO server:

require([
	"dojo/request", "dojox/socket"
], function(request, Socket){
	var
		args = {},
		ws = typeof WebSocket != "undefined",
		url =  ws ? "/socket.io/websocket" : "/socket.io/xhr-polling";
		
	var socket = new Socket(args = {
		url:url,
		headers:{
			"Content-Type":"application/x-www-urlencoded"
		},
		transport: function(args, message){
			args.data = message; // use URL-encoding to send the message instead of a raw body
			request.post(url, args);
		}
	});
	var sessionId;
	socket.on("message", function(){
		if (!sessionId){
			sessionId = message;
			url += '/' + sessionId;
		}else if(message.substr(0, 3) == '~h~'){
			// a heartbeat
		}
	});
});

Comet Session Protocol

And here is an example of connecting to a Comet Session Protocol server (the following example was tested with Orbited, but could work with Hookbox, APE, and others):

require([
	"dojo/json", "dojox/socket"
], function(json, Socket){
	var args, socket = new Socket(args = {
		url: "/csp/handshake"
	});
	function send(data){
		return socket.send(json.stringify(data));
	}
	var sessionId = Math.random().toString().substring(2);
	socket.on("connect", function(){
		send({session:sessionId});
		socket.on("message", function(){
			args.url = "/csp/comet";
			send({session:sessionId});
		});
	});
});

Tunguska

Tunguska provides a Comet-based interface for subscribing to data changes. This is a very simple protocol which allows us to communicate with a Tunguska server:

var socket = new Socket({
	url:"/comet",
	headers: {
		"Accept": "application/json",
		"Content-Type": "application/json",
		"Client-Id": Math.random()
	}});  
function send(data){
	return socket.send(json.stringify(data));
}
socket.on("connect", function(){
	// now subscribe to all changes for MyTable
	send([{"to":"/MyTable/*", "method":"subscribe"}]);
});

Conclusion

Dojo’s socket API is a flexible simple module for connecting to a variety of servers and building powerful, efficient real-time applications without constraints. This adds to the array of awesome features in Dojo.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Getting right to the point, (The Official Point!), we are pleased to announce the expansion of SitePen Support to officially include more than fifteen popular open-source JavaScript toolkits!

Now supporting the following JavaScript toolkits:

In addition to toolkits, we will continue to support your custom JavaScript source code, as well as key underlying technologies and formats, including JSON, HTML5, WebSockets, SVG/Canvas, Mobile Web, Server-Side JavaScript, AMD, Node.js and many more.

Our expertise with Dojo and advanced JavaScript is relevant for a wide-range of desktop and mobile web application projects and our approach to SitePen Support has always been flexible with the priority being to improve our customers’ web apps. We strive to support our customers in every way possible and we continue to be Dojo experts. In addition, we’re now committed to providing your organization with the front-end development expertise that will optimize your application regardless of which toolkits and technologies your company is comfortable using. You have our word!

Learn More About SitePen Support or Contact Us to get started today!

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

ComposeJS is designed to work across multiple JavaScript platforms, client and server, including Node.js and browsers, and can be loaded directly as a script or as a module (with Dojo, RequireJS, etc.). ComposeJS has clean, simple syntax in all of these environments. Generating constructors (or classes) is easy:

MyClass = Compose(BaseClass, {
  someMethod: function(){
  }
});

This is just a quick example of using ComposeJS. The API is highly flexible and permutative, allowing for multiple mixins, doing constructor delegation, handling conflict resolution, and more. The ComposeJS documentation provides ample examples and explanation. It is easy to compose structures from multiple entities, for example:

MySubClass = MyClass.extend(SomeMixin, SomeOtherMixin, {...});

However, I want to focus on the design concepts that drove the development of ComposeJS.

Simple Design Aligned with Native JavaScript

ComposeJS is designed to have an extremely simple, consistent API. The ComposeJS function can take any number of arguments, and each are treated the same. Each argument can be an object or a function. Each object’s properties and methods are available on instances of the ComposeJS generated class, and each function acts as a constructor, called on a instantiation (with its prototype mixed in). These means that the API is very easy to remember. ComposeJS can also work with constructors created from any source. You can use ComposeJS to compose classes from native classes (like Error), ComposeJS classes, classes created from Dojo, MooTools, etc., or constructors created with plain JavaScript. Since ComposeJS is based on core JavaScript principles, ComposeJS is extremely flexible.

ComposeJS also includes helper functions and extensions to fulfill a wide variety of programming needs. Compose.create can be used to instantly create an object, providing quick object delegation. Compose.apply can be applied to an existing object, allowing you to copy and mix in properties from one object to another. ComposeJS also includes decorator construction that allows you to create your own extensions for ComposeJS for more sophisticated object composition techniques.

Design Influences

Let’s look at some of the primary computer science concepts that have been developed over the years to deal with object composition:

• Inheritance is the fundamental concept of inheriting functionality from other classes. Inheritance systems provide a means for overriding existing functionality. ComposeJS utilizes basic JavaScript (prototype-based) inheritance, making it easy to extend and override existing components.
• Multiple inheritance is an extension of inheritance to compose functionality from multiple classes. The original design has largely been superseded by mixins and traits due to the fragility of only being able to compose with hard references to super classes.
• Mixins are an improvement over multiple inheritance, allowing for more flexible composition of components, without relying on shared inheritance chains. ComposeJS utilizes the concept of mixins to combine functionality from different components.
• Traits address the issues of composition of mixins when different parts of mixins need to be individually selected, providing a much more robust solution in the face of conflicting methods.
• Aspects provide fine-grained composition of functionality for individual methods.

ComposeJS uses mixin-style ordered prioritization for default method resolution, but provides the conflict resolution capabilities of traits with method aliasing and exclusion control via the from() decorator. This allows you to enjoy the convenience of the mixins, plus the precise composition control of traits. Basically, you can assign a set of mixins and their order defines the priority of method override. When this is not fine-grained enough, you can explicitly rename methods and choose which method overrides others. When methods are in ambiguous placements in the hierarchy, they will resolve to a conflicted state to indicate that explicit selection is needed.

ComposeJS also strikes a pragmatic balance between the magically implicit C3 method resolution order (that few mortals can remember), and excessively explicit conflict resolution mandates of traits (where order is normally ignored). ComposeJS follows the simple rule that when explicit order has been supplied without conflicting order elsewhere, method resolution is determined, otherwise a conflict must be resolved.

ComposeJS also uses also provides aspect-oriented decorators to combine functionality of individual methods. This allows for super-call type functionality, but with the much more JavaScript oriented style of aspects, and the ease of using before, after, and around advice. This not only aligns more closely with idiomatic JavaScript, it is about twice as fast as traditional super-call approaches in JavaScript that search through the prototype chain for overridden methods. Aspect oriented decoration is also EcmaScript “strict mode” compatible (in fact ComposeJS opts for strict mode when available), whereas class constructors like dojo.declare are currently not compatible with strict mode.

Functional, Secure

ComposeJS is designed to follow principles of least authority. Creating new constructors or instances never modifies an existing object or the global scope (unless explicitly applied to an existing object using the “apply” API). ComposeJS creates new constructors and instances based on existing objects, preserving fundamental concepts of functional programming and avoiding sloppy side-effect based imperative programming styles. This also makes ComposeJS suitable for use in object capability security systems.

Installation

ComposeJS currently is a single file package. You can download the file from the github repository. ComposeJS is also available in the Dojo Foundation package repository, so you can also install it with CPM:

cpm install compose

Or you can install it for NodeJS:

npm install compose

Dojo and ComposeJS

ComposeJS is not Dojo specific, but can be used with Dojo classes. Because ComposeJS follows standard JavaScript prototype construction, ComposeJS can extend and mixin classes created with dojo.declare. Remember that dojo.declare‘s inherited() method won’t work from ComposeJS class methods, ComposeJS’ aspects should be used instead. ComposeJS is also being considered as a replacement for dojo.declare in Dojo 2.0.

Compose!

ComposeJS provides clean, simple, compact syntax for creating class-like constructors in JavaScript and is easy to start using. It builds on the best principles of composition and inheritance demonstrated in other languages, while maintaining a completely JavaScript oriented approach. ComposeJS has solid documentation, look through the explanations and examples and give it try!

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

npm install my-new-package

When you have updates for your package that you want to designate as a new version, simply tag it in Git. And that’s it! Next time your package is installed or upgraded the newest version will be there.

As mentioned in the introductory blog post about the repository, this entire toolset is also open source and interoperable. The package repository is based on the CommonJS package specification and registry specification that was developed and is used by Isaac Schlueter for NPM (Node package manager). NPM and the package registry specification was designed for multiple registries, and the package repository is fits into the distributed package registry model used by NPM. To see NPM in action working with the Dojo Foundation Packages, you can install the compose package, which is housed at the Dojo Foundation:

npm install compose

This repository is designed to for JavaScript packages for any platform. Dojo is using this repository for AMD-based browser packages with a browser-targeted package manager (CPM), but the repository can hold packages built for other systems, including Node. This is particularly useful for packages that can be used in the browser and on Node. Give it try with your next Node package for the easiest package maintenance imaginable!

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Module Compatibility

Dojo modules are now completely compatible with:

• RequireJS
• Nodules, a Persevere sub-project for Node.js package and module handling
• Backdraft Framework, the leading candidate for the future Dojo module loader
• other AMD-compatible module loaders

Flexibility, Performance, and Stack Traces

This refactoring gives Dojo excellent flexibility going forward, to support both legacy synchronous loading mechanisms, as well as new asynchronous script-tag based loading that provides significant performance boosts and debugging improvement (including real stack traces!).

Anonymous Modules

The AMD module format also has a significant advantage in that it allows for anonymous modules, which removes the tight coupling between code and module identity. With anonymous modules, identity is DRY, avoiding the duplication of information in filename and in code. This makes code much more portable, it can be moved to different directories or packages without altering the code itself. AMD also uses slash-delimited module identifiers rather than dot-delimited, which enables more direct mapping to URLs and allows for relative references to other modules, giving us portability of directory structures as well (a directory with a set of modules can be moved or renamed, and the relative ids would still be valid).

Backwards-Compatible

First, don’t worry, your current code based on dojo.provide/dojo.require will still work. By default, Dojo uses a backwards-compatible synchronous loader. You can continue to write:

// synchronously load dijit.Tree
dojo.require("dijit.Tree"); 
// dijit.Tree is now available
new dijit.Tree(...);

Using the New Module Format

While you can continue to use the legacy module format and the legacy module API is the supported format, it is possible to start using the new module format with proper precautions, allowing you to take advantage of asynchronous module loaders including RequireJS or Backdraft now for faster and better module loading.

However, first a word of warning. The refactor to AMD is a transitional step for Dojo, and user-authored AMD modules are not supported within Dojo’s loader/build system yet. The current build system in 1.6 has very limited support for the AMD format. If you write your modules in AMD format and you still use the default synchronous Dojo loader, make sure you follow the same style (including line breaks) as the Dojo modules to ensure proper builds. The build system was minimally altered to build Dojo’s modules.

There is no official support for building with AMD modules, but of course it works if you follow the module declaration style of Dojo’s modules. The build systems provided by RequireJS or Backdraft are specifically designed for AMD, and much more robust and capable for the new format.

Again, Dojo 1.6 doesn’t officially support user-based AMD modules or asynchronous loading, but various users have successfully tested this functionality, and this is the future of Dojo. With Dojo, we do not break forwards-compatibility in dot releases, e.g. 1.5 to 1.6. We’re currently adding functionality that will be the foundation for Dojo 2.0, without breaking our promise of forwards-compatibility.

Creating a module with this format is simple:

• define your dependencies in an array as the first argument
• and provide a callback function to execute your module once the dependencies are ready

define(["dijit/Tree"], function(Tree){
  // The dijit Tree is now available, and we can use the local variable Tree
  new Tree(...);
});

Now your module is ready for a Dojo async loader or RequireJS. You can still use this module with standard dojo.require() calls as well.

To summarize your options:

• Use the legacy format (dojo.require()/dojo.provide()) – These modules will only work with the Dojo sync loader, but they will build properly in the Dojo build system.
• Use the AMD format (define()) with Dojo sync loader – These modules will load properly and can be used with other module loaders. This is not officially supported yet and they will only build properly in the current Dojo 1.6 build system if you carefully follow the Dojo modules code style.
• Use the AMD format (define()) with RequireJS or Backdraft – Modules should load properly and can be used with other module loaders. They will build properly using their provided build system (however Dojo hasn’t been officially tested on other module loaders)

Referencing Modules

When using Dojo with the AMD format, there are also a few different ways to reference modules. First, the AMD recommended way of referencing modules is to declare them in the dependency list and then provide matching arguments, like we did above. However, this is not supported by the Dojo 1.6 build system. This module format is only for fully AMD-compliant loaders. An example:

define(["dojo/cookie", "dijit/Tree"], function(cookie, Tree){
  var cookieValue = cookie("cookieName"); // get a cookie
  new Tree(...); // create a dijit.Tree
});

This provides a big advantage over the nested-object-as-a-namespace approach to reference constructors and functions from modules since we no longer need to type out the namespace on every reference. We only need to give the full path “dojo/cookie” in the dependencies, and once it is aliased to an argument, we can directly reference by that argument/variable (cookie in this example). No longer does using Dojo mean typing “dojo.” until your ring finger develops carpal tunnel.

However, for those who need to use the Dojo build system, want to maintain current coding style, or want to migrate existing modules to the new format, we can also use “dojo” and “dijit” as dependencies themselves, for easy nested-object style referencing. Remember though, that we still need to list our dependencies, even if we don’t use the reference to them. The example above could be written:

define(["dojo", "dijit", "dojo/cookie", "dijit/Tree"], function(dojo, dijit){
  var cookieValue = dojo.cookie("cookieName"); // get a cookie
  new dijit.Tree(...); // create a dijit.Tree
});

Of course this is more verbose, but it makes migration much easier.

Another thing to be aware of with Dojo’s move to AMD is that this new format won’t work with previous versions of Dojo. Consequently, you can’t just copy Dojo 1.6 modules into an earlier Dojo release and expect it to work. In order to run 1.6 modules in an earlier version you would need to port them to the legacy dojo.require/dojo.provide API (and, of course, deal with any other dependencies from 1.6 that the module had).

Running Dojo on RequireJS

Dojo runs with RequireJS, using RequireJS as the module loader. To do this, first load RequireJS, then configure Dojo as a package:

And we can now load our application entry module, which can refer to “dojo” as a dependency:

Where my-module may look like:

define(["dojo"], function(dojo){
  dojo.query("some-query")...
  ...
});

There are a few things to be aware of when using RequireJS with Dojo:

• RequireJS is managing module loading, so Dojo cannot automatically determine when to parse your HTML for declarative components/widgets. You must manually call dojo.parser.parse() after all necessary modules are loaded and the page is ready.
• RequireJS is handling module loading, so you should also use RequireJS to do your builds.

The Future is Here. The Future is Near.

We will provide more information in the future on using the Backdraft loader, as we look to integrate it with Dojo.

We are also working on a package installer that will automatically generate your configuration setup.

We are looking forward to great improvements in module loading and interoperability with this new format. This will be a key part of our move towards autonomous packages as well. Look for more updates in the near future, and let us know if you would like to get involved.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Comet Server Choices

At the recent London Ajax JSMiniConf I presented a summary of the current state of Comet and Server-Side JavaScript tools.

London Ajax JSMiniConf: “A quick summary of Comet” by Dylan Schiemann from London Ajax User Group on Vimeo.

With so many great options to choose from, the choices can be overwhelming. Our current approach to Comet is as follows.

• cometD – We have invested resources in the Bayeux protocol, the DojoX cometD client, and the Python version of the cometD server. cometD is a popular choice because of its solid performance and the widespread support of the Bayeux protocol across many servers including Jetty (Java), DWR (Java), Erlycomet (Erlang), WebSphere (Java), WebLogic (Java), WebSync (IIS, .Net), and more.
• DWR – We have invested resources in DWR, helping bring Bayeux support to this popular framework for simple remoting between Java and JavaScript applications. DWR helps Java developers ease the transition to using JavaScript, in addition to providing real-time capabilities. In comparison to GWT, because DWR is a Java to JavaScript interface, you can code in JavaScript, the native language of the browser, and communicate with a Java back-end without the magic of translating/rewriting Java into JavaScript.
• Hookbox – A popular upstart Comet server written in Python that offers an extremely simple approach for getting started, including the Hosted Hookbox service. It includes support for Comet Session Protocol (CSP), and it extends the js.io microtoolkit for managing a wide variety of messaging protocols.
• Liberator and Lightstreamer – The two most popular commercial Comet servers on the market, Caplin Liberator and Lightstreamer have both improved dramatically over the past decade due to a healthy level of competition. Companies looking for a tried and true approach to the real-time web typically look at these options as well as Jetty and others. All of these options are great to work with, and each is optimized to handle performance in varying ways. While both of these platforms are proprietary and commercial, both offer limited-use free versions for commercial use as well. Watch their creators in action at the recent Comet panel, where I ask them which Comet servers they would suggest if their own product did not exist!
  
• Persevere and Tunguska – The majority of our non-Dojo research and development efforts have been focused on Persevere, including the new Tunguska sub-project. Tunguska is a set of tools for building real-time applications rather than a closed black box that can’t easily be integrated with, including connection and message queues, long-polling, message routing, and clustering. It runs on top of the popular Node.js or Narwhal+Jack, the two most popular server-side JavaScript engines on the market today. We are very focused on this work today because we’re strong believers in the value of server-side JavaScript
• XMPP – We receive a number of inquiries into XMPP server integration for chat applications. We developed the original dojox.xmpp chat client, and have assisted a number of organizations in making this work with their web applications.
• Others – We’re also fans of Socket.IO, Orbited, Meteor, APE, Google App Engine, Tornado and a number of other projects. We’re happy to help our clients use these servers should they be appropriate for their web application development needs.

With Dojo 1.6, we rely on our new dojox.socket implementation making it easy to work with a wide variety of Comet servers, including the ones listed above.

Bottom line: there is no shortage of excellent Comet servers, so pick one that meets the needs of your application and your development team.

SitePen Comet Services

SitePen offers assistance with your real-time Comet development needs, from consulting on architecture and server choices, to development of real-time applications, to support for your ongoing JavaScript, Ajax, and HTML5 application development needs when working with a Comet server. Contact SitePen for a free 30-minute consultation to learn more about how we can help your organization create extraordinary real-time web and mobile applications.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

RQL

RQL simply consists of named operators that take a set of arguments and syntactically follows the standard parenthesis’s-based call syntax used by most modern languages (JavaScript, Java, C, PHP, Python, etc) as prefix notation operators. Everything in RQL can be expressed as a set of these operators by nesting and sequencing them. Nothing needs to be altered to add new operators. The only additional syntax in RQL are comparisons that are simply sugar or shorthand for basic RQL operators, and thus do not introduce any parsing or execution complexities. For example, to express a query for all the resources where the rating equals 5 and the price is less than 10:

and(eq(rating,5),lt(price,10))

RQL is specifically designed for the web, using URI appropriate delimiters, and using URI character encoding for all text/string values. It provides a shorthand or sugar for certain operators that makes RQL a completely compatible superset of the default form encoding, application/x-www-urlencoded, as well as Feed Item Query Language (FIQL). The example above could alternately be written in RQL as:

rating=5&price=lt=10

This syntax makes it easy to use RQL with existing forms and query processors. The documentation in the RQL git repository contains many more details about the defined operators and syntax. Let’s look at more of the motivation for RQL.

What’s wrong with SQL?

SQL has long been the query language of choice for applications, but a couple factors have motivated alternate query languages. First, the NoSQL movement continues to gain ground, eroding the notion of using SQL as the basis for all database operations. Second, the explosion in Ajax applications has moved significant application logic to the browser, increasing the need for capable queries to be delivered from the browser to the server within URLs. SQL predates URLs by some 20 years, and does not fit well with URLs. SQL is not designed for document, graph, or object-oriented databases and is not only awkward within URLs, it is a terrible security issue to provide any pass-through of query strings directly to SQL (known as SQL injection). SQL is particularly hazardous since retrieval and modifying operations can all be combined in the same query. SQL’s syntax is also completely unlike that of modern programming languages. Trying to shore up SQL for URIs and modern databases with an SQL-derivative query language is an exercise in futility.

Again, RQL is designed to be URI friendly, leveraging URI encoding and designated delimiters for syntax that works perfectly in web requests. RQL also borrows from the familiar parenthesis-based call syntax of JavaScript, Python, C, C++, Java, PHP, etc. to give a highly composable and extensible query language.

What’s wrong with Map (and reduce) functions and key-value stores?

Using map and reduce functions for generating indexes is awesome! CouchDB’s use of mapping and reducing functions as the basis for highly scalable, incredibly flexible (Turing complete) indexed queries is brilliant, and is one of the main reasons for CouchDB’s enormous popularity. Map/Reduce functions are also an integral part of many other NoSQL databases like Riak, Hadoop, MongoDB, and more. Map/Reduce functions are somewhat low-level, however, and there are a lot of aspects to getting appropriate data out of databases besides just creating flexible indexes. Map/Reduce creates a great foundation to build on, but let’s look at some of the other useful tools for database querying. For many of these, SQL actually has constructs that we can learn from. It is quite beneficial to utilize a query language layer on top of a Map/Reduce layer.

SQL is a domain specific language (DSL). DSLs are powerful for improving our productivity by providing a syntax that is especially well suited for the task at hand. We have seen the power of DSLs in other areas of programming. For example, CSS selector querying has revolutionized how we retrieve DOM nodes in the browser. This functionality is the core of most modern JavaSscript libraries. Of course it is certainly still possible to retrieve nodes by purely programmatic APIs, but CSS selector querying makes life much easier. A query language plays the same role. While it is possible to query a database through programmatic means, and as I discussed in NoSQL architecture, is very important for maximizing control, modularity, and efficiency, this does not negate the remarkable benefit that can be afforded also being able to utilize a query language on top of a database.

RQL continues the tradition of providing a language specifically designed for the needs of querying data, but doing so with a much simpler, easy to use syntax. RQL is also highly extensible, making it extremely easy to utilize custom map-function-based indexes/views to compose and combine with other querying mechanisms. For example, we could have RQL translate queries to retrieve data from simple single-property indexes and from complex indexes:

price<1000&customProductEvaluationFunction(4)

SQL does a great job of handling massive permutations of queries and finding the most efficient usage of a fixed set of indexes to find results. With map-reduce alone, you create an index or view for every different type of query. With queries that can take many different forms, it can often be unfeasible to generate a mapping function for every conceivable permutation. This approach simply doesn't scale, developers often can't create large number of views and it wouldn't be efficient to keep a large number of views/indexes up to date. With SQL, the query execution engine can take queries in many forms, including multiple parameters, various constraining columns, and more, and find the most efficient execution path utilizing existing indexes. Query engine implementations have the freedom to make appropriate optimizations because the query language is decoupled from the indexes.

Unfettered query permutations is not without hazards. One of the advantages to key-value stores and mapping functions is the guarantee of O(log n) queries. SQL tends to make it far too easy to generate extremely expensive queries which may not appear to be problematic until a database grows large enough to cause problems. Because of this, RQL is designed in complement with a RQL templating form. This is essentially an application of URI templating with RQL, and allows one to define the set of acceptable RQL queries (without having to write out each individual form). See the RQL templates section below for more on this.

A semantically well-defined query language also serves to make querying more transparent. Interaction transparency is a key concept of REST, and allows intermediaries and components to participate in a meaningful way that can't be achieved with opaque queries. Frameworks can provide client-side querying of cached or replicated data, proxies can understand queries, and queries can be generated by reusable code that can be used across many applications.

What's wrong with JSONQuery?

With the increased popularity of JSON-based data representations, we have sought to provide a convenient syntax for querying JSON data by extending and improving JSONPath. This syntax is called JSONQuery. The JSONPath syntax that JSONQuery inherits is still not well-aligned with URL structures and is very difficult to extend due to the fact that each operator is based on a different syntax. Creating new operators thus requires modifications to the parsing engine.

RQL Templates

RQL templates provide a means for defining a query or a constrained set of queries and the variables that may be substituted into the query. When queries can be made from the web, we must deal with the challenge of untrusted users, and unmitigating querying capabilities typically makes a server highly vulnerable to overload and resource exhaustion. With RQL templates, we can specify a RQL template (or set) that we know can be efficiently processed by the server (typically O(log n) time). This is also one of the benefits of map-reduce functions, but RQL templates provide more flexibility, still allowing various permutations with adjustable constraint.

One of the key concepts about the Map/Reduce approach is the emphasis on utilizing information about expected query forms up front to generate customized indexes. Then most of the work only needs to be performed once per data change rather than doing large amounts of work for each query (and most applications read and query much more than write/change data). RQL templates not only serve to constrain queries, but RQL in template form can also serve to conveniently inform the creation of views/indexes and their map functions. One can auto-generate map functions and indexes based on RQL templates, fully leveraging the DSL approach. Here is an example of a template, that would indicate that based on the available queries, the price and rating properties should be indexed:

{&price,rating}

Because RQL is URI-based, templates can naturally be written using the standard URI templating syntax. For example, a simple RQL template might look like:

{&price,rating}&limit({count})

This indicates acceptable properties to search on (price and rating) and that the query must include a limit on the number of items returned (with the "count" variable).

While standard URI templating provides a good foundation for templating, RQL templates have some additional forms for greater expressibility. We can also use square brackets to indicate a fragment of a query that may or may not exist, or may occur a variable number of times. For example, we could indicate support for sorting, by allowing an optional sort operation:

{&price,rating}&limit({count})[&sort([+,-][price,rating])]?

With RQL templates, we can give a "menu" of possible queries that the server supports. This follows the fundamental hyperlink principle of REST, providing self-descriptive navigation to the user agent. Clients and users can easily discover what queries can be made against a server and the appropriate format.

RQL templates can also be used like parameterized prepared statements in SQL, where values can be provided outside the query. This can simplify query generation by automating the value encoding process which can be a source of vulnerability if done improperly (hence SQL injection is such a frequent vulnerability of web applications).

Finally, a set of RQL templates can also be used to generate appropriate indexes. Indexed properties can be selected or map (and reduce) functions can be determined from templates.

RQL templates are still a relatively new mechanism with RQL, and we will explore implementation possibilities in a later post.

Implementation

A JavaScript implementation of RQL is available on the RQL github project. This version runs a CommonJS module with async support and runs on NodeJS, Dojo 1.6, RequireJS, Narwhal and any other CommonJS platform (it is pure JavaScript), and is integrated into Persevere 2.0.

The most basic way to use the JavaScript implementation is to query a JavaScript array. To do this we prepare our query, and execute:

var query = require("rql/js-array").query;
// some sample data:
var products = [
  {price:14.99, rating: 5},
  {price:5.99, rating: 3}];

var underTen = query("price<10");
var productsUnderTen = underTen(products);
productsUnderTen.length -> 1

Using RQL with Persevere

RQL is the core query language for Persevere 2.0. Persevere runs on NodeJS and provides a JSON-oriented HTTP/REST interface to various data stores. This interface includes support for RQL. With Persevere, you can create a new data store (using the included data explorer) or by creating one programmatically. The Persevere example wiki includes a Page model as an example store. We can easily query the store with RQL-based URLs. For example, to find the first 10 pages that have a status of "published":

GET /Page/?status=published&limit(10)

In this example, we sort the pages by the author, and list the author and status, limiting to 10 items, starting at an offset of 10:

GET /Page/?sort(createdBy)&select(createdBy,status)&limit(10,10)

The composibility of RQL gives Persevere powerful web-based querying to data stores. Persevere uses the RQL parser module to parse the queries and deliver them to the underlying data store (converts to MongoDB queries, SQL queries, etc.).

The RQL implementation in Persevere also lets us generate queries using JavaScript chaining, allowing us to create queries in JavaScript with a similar look to URL-based queries. The previous two examples can be done in JavaScript. First we can filter and limit:

var Page = require("model/page").Page;
Page.query().eq("status","published").limit(10).forEach(function(page){
  // this is called for each item returned from the query
});

And we can sort, select, and limit:

var Page = require("model/page").Page;
Page.query().sort("createdBy").select("createdBy","status").limit(10,10).forEach(function(page){
  // process each item
});

The query is sent to the underlying data source (can be in memory, MongoDB, Redis, etc) once an array method is called. Array methods include forEach, map, filter, and other iterative array methods.

Using RQL with Dojo

We can also use RQL from Dojo. Download the RQL source files into your JavaScript directory and then you can make queries with RQL. A particularly powerful use case for RQL is as a query engine for Dojo object stores (the new store API introduced in 1.6). Replacing the query engine of an object store is as simple as setting the queryEngine to the RQL query executor:

define("my-module", ["dojo/store/Memory", "rql/js-array"],
function(Memory, jsArray){
  var memoryStore = new Memory({data:myData});
  memoryStore.queryEngine = jsArray.query;
  memoryStore.query("price<10&sort(rating)").forEach(function(product){
    // handle each product
  });
});

We can also utilize the JavaScript chaining API with the rql/query module in Dojo as well.

Adding Operators

RQL is designed to be extensible. The JavaScript makes it easy to add new operators. We can simply add a new operator to the operators object, exported from rql/js-array. Let's imagine we want to add a operator that finds all products that are on sale for at least a given percentage:

define("my-module", ["rql/js-array"],
function(jsArray){
  jsArray.operators.saleAt = function(percent){
    var result = [];
    for(var i = 0, length = this.length; i < length; i++){
      var item = this[i];
      if((item.regularPrice - item.salePrice) / item.salePrice * 100 > percent){
        result.push(item);
      } 
    }
    return result;
  };
  var productsOnSaleForMoreThan20PercentOff = jsArray.query("saleAt(20)")(products);
});

Future implementation work will include a tool for generating map and reduce functions based on RQL queries, and templating tools.

RQL: A Modern Query Language

RQL is designed for modern application development. It is built for the web, ready for NoSQL, and highly extensible with simple syntax. This is a query language for next generation database interaction.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Dojo 1.6 introduces a new API for Comet-style real-time communication based on the WebSocket API. WebSocket provides a bi-directional connection to servers that is ideal for pushing messages from a server to a client in real-time. Dojo’s new dojox.socket module provides access to this API with automated fallback to HTTP-based long-polling for browsers (or servers) that do not support the new WebSocket API. This allows you start using this API with Dojo now.

The dojox.socket module is designed to be simple, lightweight, and protocol agnostic. In the past Dojo has provided protocol specific modules like CometD and RestChannels, but there are numerous other Comet protocols out there, and dojox.socket provides the flexibility to work with virtually any of them, with a simple foundational interface. The dojox.socket module simply passes strings over the HTTP or WebSocket connection, making it compatible with any system.

The simplest way to start a dojox.socket is to simply call it with a URL path:

var socket = dojox.socket("/comet");

We can now listen for message events from the server:

socket.on("message", function(event){
  var data = event.data;
  // do something with the data from the server
});

Here we use the socket.on() event registration method (inspired by socket.io and NodeJS’s registration method) to listen to “message events” and retrieve data when they occur. This method is also aliased to the Dojo style socket.connect().

We can also use send() to send data to the server. If you have just started the connection, you should wait for the open to ensure the connection is ready to send data:

socket.on("open", function(event){
  socket.send("hi server");
});

Finally, we can listen for the connection being closed by the server or network by listening for the “close” event. And we can initiate the close of a connection from the client by calling socket.close().

The dojox.socket method can also be called with standard Dojo IO arguments to initiate the communication with the server. This makes it easy to provide any necessary headers for the requests. For example:

var socket = dojox.socket({
	url:"/comet",
	headers: {
		"Accept": "application/json",
		"Content-Type": "application/json"
	}});

This will automatically translate the relative URL path to a WebSocket URL (using ws:// scheme) or an HTTP URL depending on the browser capability.

For some applications, the server may only support HTTP/long-polling (without real WebSocket support). We can also explicitly create a long-poll based connection:

var socket = dojox.socket.LongPoll({
	url:"/comet",
	headers: {
		"Accept": "application/json",
		"Content-Type": "application/json"
	}});

We can also provide alternate transports in the socket arguments object. This would allow us to use dojo.io.script.get to connect to a server. However, a more robust solution is to use the dojox.io.xhrPlugins for cross-domain long-polling, which will work properly with dojox.socket.

Auto-Reconnect

In addition to dojox.socket, we have also added a dojox.socket.Reconnect module. This wraps a socket, adding auto-reconnection support. When a socket is closed by network or server problems, this module will automatically attempt to reconnect to the server on a periodic basis, with a back-off algorithm to minimize resource consumption. We can upgrade a socket to auto-reconnect by this simple code fragment:

socket = dojox.socket.Reconnect(socket);

Using Dojo WebSocket with Object Stores

One of the other big enhancements in Dojo 1.6 is the new Dojo object store API (supercedes the Dojo Data API), based on the HTML5 IndexedDB object store API. Dojo 1.6 comes with several store wrappers, and the Observable store provides notification events that work very well with Comet driven updates. Observable is a store wrapper. To use it, we first create a store, and then wrap it with Observable:

define("my-module", function(require){
  var JsonRest = require("dojo/store/JsonRest");
  var Observable = require("dojo/store/Observable");
  var store = new JsonRest({data:myData});
  store = Observable(store);
});

This store will now provide an observe() method on query results that widgets can use to react to changes in the data. We can notify the store of changes from the server by calling the notify() method on the store:

 
socket.on("message", function(event){
  var existingId = event.data.id;
  var object = event.data.object;
  store.notify(object, existingId);
});

We can signal a new object by calling store.notify() and omitting the id, and a deleted object by omitting the object (undefined). A changed/updated object should include both.

Handling Long-Polling from your Server

Long-polling style connection emulation can require some care on the server-side. For many applications, the server may have sufficient information from request cookies (or other ambient data) to determine what messages to send the browser. However, other applications may vary on what information should be sent to the browser during the life of the application. Different topics may be subscribed to and unsubscribed from. In these situations, the server may need to correlate different HTTP requests with a single connection and its associated state. While there are numerous protocols, one could do this very easily be defining a unique connection and adding that as a header for the socket (the headers are added to each request in the long-poll cycles). For example, we could do:

var socket = dojox.socket.LongPoll({
	url:"/comet",
	headers: {
		"Accept": "application/json",
		"Content-Type": "application/json",
		"Client-Id": Math.random()
	}});

In addition, dojox.socket includes a Pragma: long-poll to indicate the first request in a series of long-poll requests to help a server ensure that the connection setup and timeout is properly handled.

We can easily use dojox.socket with other protocols as well:

CometD

To initiate a Comet connection with a CometD server, we can do a CometD handshake, connection, and subscription:

var socket = dojox.socket("/cometd");
function send(data){
  return socket.send(dojo.toJson(data));
}
socket.on("connect", function(){
  // send a handshake
  send([
    {
       "channel": "/meta/handshake",
       "version": "1.0",
       "minimumVersion": "1.0beta",
       "supportedConnectionTypes": ["long-polling"] // or ["callback-polling"] for x-domain
     }
  ])
  socket.on("message", function(data){
    // wait for the response so we can connect with the provided client id
    data = dojo.fromJson(data);
    if(data.error){
      throw new Error(error);
    }
    // get the client id for all future messages
    clientId = data.clientId;
    // send a connect message
    send([
      {
         "channel": "/meta/connect",
         "clientId": clientId,
         "connectionType": "long-polling"
       },
       {  // also send a subscription message
         "channel": "/meta/subscribe",
         "clientId": clientId,
         "subscription": "/foo/**"
       }
    ]);
    socket.on("message", function(){
      // handle messages from the server
    });
  });
});

Socket.IO

Socket.IO provides a lower-level interface like dojox.socket, providing simple text-based message passing. Here is an example of how to connect to a Socket.IO server:

var args, ws = typeof WebSocket != "undefined";
var socket = dojox.socket(args = {
  url: ws ? "/socket.io/websocket" : "/socket.io/xhr-polling",
  headers:{
    "Content-Type":"application/x-www-urlencoded"
  },
  transport: function(args, message){
    args.content = message; // use URL-encoding to send the message instead of a raw body
    dojo.xhrPost(args);
  };
});
var sessionId;
socket.on("message", function(){
  if (!sessionId){
    sessionId = message;
    args.url += '/' + sessionId;
  }else if(message.substr(0, 3) == '~h~'){
    // a heartbeat
  }
});

Comet Session Protocol

And here is an example of connecting to a Comet Session Protocol server (the following example was tested with Orbited, but could work with Hookbox, APE, and others):

var args, socket = dojox.socket(args = {
  url: "/csp/handshake"
});
function send(data){
  return socket.send(dojo.toJson(data));
}
var sessionId = Math.random().toString().substring(2);
socket.on("connect", function(){
  send({session:sessionId});
  socket.on("message", function(){
    args.url = "/csp/comet";
    send({session:sessionId});
  });
});

Tunguska

Tunguska provides a Comet-based interface for subscribing to data changes. This is a very simple protocol which allows us to communicate with a Tunguska server:

var socket = dojox.socket({
	url:"/comet",
	headers: {
		"Accept": "application/json",
		"Content-Type": "application/json",
		"Client-Id": Math.random()
	}});  
function send(data){
  return socket.send(dojo.toJson(data));
}
socket.on("connect", function(){
  // now subscribe to all changes for MyTable
  send([{"to":"/MyTable/*", "method":"subscribe"}]);
});

Conclusion

Dojo’s socket API is a flexible simple module for connecting to a variety of servers and building powerful, efficient real-time applications without constraints. This adds to the array of awesome new features and improvements in Dojo 1.6.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

var assert = require("assert");
tests = {
  testSomething: function(){
    assert.eq(3, 3);
  }
}
require("patr/runner").run(tests);

Creating asynchronous tests is easy as well. Inspired by Dojo’s DOH test runner for returning a promise from tests, we don’t need a new interface to indicate when an asynchronous test is completed. The promise’s completion or failure indicates the completion or failure of the test. This allows us to elegantly pipe asynchronous operations through to the test runner. Here is an example of an asynchronous test:

var fs = require("promised-io/fs");
require("patr/runner").run({
    testFile: function(){
        return fs.readFile("my-file.txt")  // asynchronously read the file
            .then(function(contents){ // next action in promise flow
                // make an assertion
                assert(contents.toString(), "expected contents of the file");
            });
    };
});

Now this test will asynchronously read the file, make an assertion, and when completed the returned promise will notify the test runner that the test is complete (or failed) and continue on to the next test.

Tests can be organized into groups by simply creating nested objects:

require("patr/runner").run({
    testGroupA: {
      testFile: function(){
        ..
      },
      ..
    },
    testGroupB: {
      testAnother: function(){
      ....
    }
});

(Also, remember all your test properties must start with “test”.)

The recommended approach for creating test modules is to use the exports object to hold tests. This makes it very easy to run a test module directly or as part of a group. For example:

//something.js:
var assert = require("assert");
exports.testSomething: function(){
    assert.eq(3, 3);
};

if (require.main === module)
    require("patr/runner").run(exports);

Now we can directly run this module to run these tests. We can also include this module in another set of tests:

//all-tests.js:
exports.testSomething = require("./something");
exports.testAnotherGroupOf = require("./more-tests");
if(require.main === module){
    require("patr/runner").run(exports);
}

We now have each of these tests within a sub group for easy execution of groups of tests at any level in the hierarchy (all the tests or individual groups).

Performance Testing

Patr is also great for performance testing. You can set an iterations flag with in a test group or from the command line to run through the test a specified number of iterations and give performance statistics.

Summary

Patr is a lightweight test runner that makes it extremely simple to create test suites and hierarchies with idiomatic JavaScript and asynchronous tests with promises. Patr is built on Promised-IO, so it is ready for cross-platform execution, including Node, Narwhal, and within the web browser.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Promised-IO

Promised-IO utilizes promises as an abstraction for I/O operations on top of Node, Narwhal/Rhino, and the browser (where possible). This serves two purposes. First, this package provides the benefits of promise usage: clean separation of concerns and proper encapsulation of eventual values. Second, Promised-IO provides a consistent normalized interface for I/O that will work on multiple platforms without sacrificing any of the advantages of asynchronous I/O, making it easy to build modules that can be used by developers on many platforms.

Usage Principles

One of the characteristics of a good developer is understanding when to apply different abstractions (rather than trying to force a single approach on every situation). Asynchronous code is often more complicated than synchronous code, but can provide big benefits to performance and scalability. In the application of asynchronous I/O there are several different approaches, and each has an appropriate usage.

Synchronous (don’t use asynchronous)

Of course synchronous calls are usually simpler (no addition of callbacks), and for parts of applications that aren’t “hot” parts of concurrent code paths, and aren’t performance sensitive, this can be the smart choice. For the startup process for servers, long period scheduled tasks, and infrequently executed code, the simplicity and code manageability of using synchronous code can outweigh any performance benefit of asynchronous code.

(There are actually times when synchronous performs better than asynchronous. Asynchronous operations have the additional overhead of handling the low-level callback, then adding the JavaScript callback to the event queue, and finally pulling events of the event loop. Synchronous operations do not have this overhead. For example, for file operations that hit the OS cache, the synchronous operation can often be about twice as fast (half as much CPU time) as the asynchronous counterpart. Of course this is only advantageous for cached data.

Synchronous operations can have a much larger performance advantage when there isn’t to-the-core support for asynchronicity. Fortunately, most of the I/O operations in NodeJS leverage libeio and low level async Posix operations for good performance, but for actions that are natively synchronous, some libraries use libeio’s eio_custom function to make it asynchronous. This function can be very expensive and should only be used for operations that will take a significant amount of time (more than about 0.1ms). I have observed synchronous operations executing hundreds of times faster than the asynchronous counterparts when eio_custom is used.

All this being said, most of the time asynchronous is a good choice for code that needs to scale, but it is important to know when and when not to use it.)

Event Registration

This is the typical way that one is notified of events in the browser and in NodeJS. You register for an event and are notified of the event anytime after the registration (no notification of past events). Event registration is the most generic and low-level asynchronous API. NodeJS uses this model exclusively because it is a low-level platform for I/O. For some situations (see below) it is appropriate to build abstractions on top of this API. However, event registration is the right choice if multiple events may take place and you only want future events.

Promises

An asynchronous abstraction designed specifically to model an action that has a single eventual completion (successful or failing). Promises act much like a value (returned from the result of a function call or computation), except that the result is asynchronous, and may or may not be immediately available. Promises are more specific than event registration, providing encapsulation (that can easily be passed around) of an eventual value and decoupling the receiver from the timing of the underlying events that fulfill the action (whether they be past or future). Promises are the right choice for asynchronous actions that have a single point of eventual completion.

Streams

Streams are an asynchronous abstraction for the progressive flow of sequential data. Streams are important for allowing a sender to feed data to a receiver without requiring large scale buffering at either end. Promised-IO implements streams with “lazy arrays”, an array-like interface that follows the standard API of JavaScript Arrays. This greatly improves the modularity of streams since they can be used with code that expects standard arrays. Lazy arrays are indeed lazy, following the functional programming principle of lazy evaluation as well. Iterative methods including map, some, filter, and every are purely functional and only iterate through the stream as needed, avoiding any unnecessary buffering or computations. Promised-IO’s streams also utilize promises to signal the eventual completion of a stream (for the end of a file or end of a HTTP response, for example), for consistency. For situations where you want to model sequential data, streams/lazy arrays are appropriate.

Promised-IO Modules

There are several key modules available in Promised-IO for access to promise-based I/O.

fs

The “fs” module implements key parts of the CommonJS file system API (the entire API will eventually be supported). It also provides Node’s file system API where each asynchronous function returns a promise rather than taking callback parameters (fortunately these two APIs can easily coexist). This gives you access to a number of the convenience functions from CommonJS’s API (like makeTree()) and gives normalizable access to asynchronous functions. This module is obviously not available for browser usage. Here is an example of using readFile() (an asynchronous function from Node’s API):

var fs = require("promised-io/fs");
return fs.readFile("my-file.txt").then(function(contents){
   // once it is loaded, we can do something with the contents
});

One of the most important functions in the “fs” module is the open() function which can be used in several powerful ways. The open function returns a file descriptor for use with Node’s functions that take a descriptor (the read, write, and close functions). While the open function opens the file asynchronously, the returned object can be directly passed to read, write, and close functions (they will execute once the file is open):

var fs = require("promised-io/fs");
var myFile = fs.open("my-file.txt", "r");
fs.read(myFile, buffer, offset, length, position).then(function(){
  // buffer should now be filled
});

The returned object is also a promise for the completion of the open operation:

var when = require("promised-io/promise").when;
var myFile = when(fs.open("my-file.txt", "r"), onSuccess, onFailure);

The object returned from open also has methods (based on asynchronous versions of the CommonJS filesystem API) for writing and closing:

var myFile = fs.open("my-file.txt", "a");
myFile.write("some data").then(myFile.close);

And finally, the returned file object can be treated like a JavaScript array for streamed reading (lazy arrays):

var myFile = fs.open("my-file.txt", "r");
myFile.forEach(function(block){
    // called for each block of data
}).then(myFile.close); // The forEach returns promise for the completion of the reading

The some() JavaScript array method can be particularly useful if you might not need to traverse the whole file:

var myFile = fs.open("my-file.txt", "r");
myFile.some(function(block){
    // if we are searching for something, we can exit be returning true at any time
});

One other useful aspect of using these lazy arrays is that we can return the file object directly as the body of JSGI responses and it will automatically be piped to the client.

function SomeJSGIApp(request){
  return {
    status: 200,
    headers: {"content-type":"text/plain"},
    body: fs.open("my-file.txt", "r");
  };
}

http-client

An HTTP client that follows the JSGI API. Originally developed for handling incoming HTTP requests (for web servers) and returning responses, http-client effectively uses the JSGI API for the reverse situation, allowing for the construction of JSGI requests and receiving JSGI responses. This module allows for some extra convenience properties and defaults to make it easy to create a request. You can simply provide a “url” property to indicate the target, and then all other properties are optional. For example:

var request = require("promised-io/http-client").request;
request({
  url:"http://somesite.com/data",
  method: "GET", // optional
  headers: {} // also optional
}).then(function(response){
  response.status -> http status code
  response.headers -> http response headers
  response.body -> A lazy array stream representing the body of the response
});

This module can be used on the server and browser.

delay

This module provides promise-based scheduling and timing similar to that of setTimeout and setInterval, but utilizing promises. The two exported functions are delay(ms) and schedule(ms). The delay() function returns a promise that is fulfilled after the specified amount of time. The schedule() function returns a lazy array that iterates periodically by the given amount of time. For example:

var delay = require("promised-io/delay").delay;
delay(2000).then(function(){
  // and two seconds later this executes
});

This module can also can be used on the server and browser.

process

This module implements the CommonJS “system” module API and provides access to a print() function and process arguments.

Summary

Promised-IO provides a robust, cross-platform I/O system based on the proven design principles of promises for efficient, portable, asynchronous JavaScript.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!