Getting right to the point, (The Official Point!), we are pleased to announce the expansion of SitePen Support to officially include more than fifteen popular open-source JavaScript toolkits!

Now supporting the following JavaScript toolkits:

In addition to toolkits, we will continue to support your custom JavaScript source code, as well as key underlying technologies and formats, including JSON, HTML5, WebSockets, SVG/Canvas, Mobile Web, Server-Side JavaScript, AMD, Node.js and many more.

Our expertise with Dojo and advanced JavaScript is relevant for a wide-range of desktop and mobile web application projects and our approach to SitePen Support has always been flexible with the priority being to improve our customers’ web apps. We strive to support our customers in every way possible and we continue to be Dojo experts. In addition, we’re now committed to providing your organization with the front-end development expertise that will optimize your application regardless of which toolkits and technologies your company is comfortable using. You have our word!

Learn More About SitePen Support or Contact Us to get started today!

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

define(, , );

This simple API can be used in a variety of different ways for different situations.

Anonymous Modules

The recommended general purpose way of writing your own modules with this API is to omit the first argument, and just include the dependencies and the module factory. A simple module can be written:

define(["math"], function(math){
  return {
    addTen: function(x){
      return math.add(x, 10);
    }
  };
});

The first argument defines the set of dependencies (as module names) for this module. The modules exported values are passed in as the arguments to the callback function once the dependent modules have been loaded.

When the module name (of the current module) is omitted, as in the example, the module is anonymous. This is a powerful way to write modules since the module source code is not coupled to any identity. The module can easily be moved to a different location without requiring any code change. This technique follows basic DRY principles, avoiding duplicate information about the identity of a module (the filename/path information does not need to be duplicated in code). This not only makes writing modules easier, but gives greater flexibility in module reuse.

Let’s look at how we load this module from a web page. We will assume the above module was put in a file called adder.js. Using RequireJS, we can load our module like this:

Once the initial entry require is called, RequireJS takes care of downloading all the dependencies of adder. We can use this same require call to load any of the module forms described below as well.

There are some rules for anonymous module usage. There may only be one anonymous module per file, and the anonymous modules must be loaded through the loader (you can just include a <script> pointing to the module).

Data Wrapping: The New JSON-P

Modules that only provide data, or dependency free methods can simply provide an object as the only argument to define():

define({
  name:"some data"
});

This is similar to JSON-P, but actually provides a significant advantage over JSON-P since it can be written in a static file without requiring the dynamic callback generation. This makes this format cacheable and CDN friendly.

CommonJS module wrapping

CommonJS modules can easily be used with the asynchronous module loader by simply wrapping them with a define call. Here you can also omit the first and second parameter, and the module’s require calls will be scanned to determine the appropriate dependencies. For example:

define(function(require, exports){
// standard CommonJS module:
  var math = require("math");
  exports.addTen = function(x){
    return math.add(x, 10);
  };
});

It is important to note that this form requires the module loader to inspect the function for require calls. The require calls must be written in the form of require("...") in order for this analysis to work properly. There are also certain browsers where this will simply not work (some versions of Opera mobile and PS3). This may not be an issue at all if your modules will be run through a build process before deployment. However, you can also wrap CommonJS modules, and manually specify dependencies. The specification allows us to also reference CommonJS variables, so we can include the standard “require” and “exports” variables:

define(["require", "exports", "math"], function(require, exports){
// standard CommonJS module:
  var math = require("math");
  exports.addTen = function(x){
    return math.add(x, 10);
  };
});

Full Module Definitions

A full module definition includes the module name, dependencies, and the factory. This form has the advantage that the module can be included in another file or it can be directly loaded with a script tag. This is the format generated by the build tools so that multiple dependencies can be combined in a single file. An example of this format:

define("adder", ["math"], function(math){
  return {
    addTen: function(x){
      return math.add(x, 10);
    }
  };
});

Finally, the last permutation of these arguments is to include the module id, but omit the dependencies list. This can be used when you want to indicate the module id (for use with direct script tag), but there are no dependencies. With no dependency array, the arguments default to “require”, “exports”, and “module”. We could alternately create our adder module:

define("adder", function(require, exports){
  exports.addTen = function(x){
      return x + 10;
  };
});

Again all these module forms can be handled by RequireJS and can be loaded by listing them as dependencies in another module or directly loading them with a require() call.

This simple API provides flexibility to declare modules for a variety of situations, from anonymous modules that can easily be moved around, to “built” modules that are ready to be used from script tags. This API is implemented by RequireJS and Dojo, as well Nodules for Node. You can also use Transporter to do on-the-fly concatenation of AMD modules and automatic wrapping of CommonJS modules that are delivered in this format.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

var assert = require("assert");
tests = {
  testSomething: function(){
    assert.eq(3, 3);
  }
}
require("patr/runner").run(tests);

Creating asynchronous tests is easy as well. Inspired by Dojo’s DOH test runner for returning a promise from tests, we don’t need a new interface to indicate when an asynchronous test is completed. The promise’s completion or failure indicates the completion or failure of the test. This allows us to elegantly pipe asynchronous operations through to the test runner. Here is an example of an asynchronous test:

var fs = require("promised-io/fs");
require("patr/runner").run({
    testFile: function(){
        return fs.readFile("my-file.txt")  // asynchronously read the file
            .then(function(contents){ // next action in promise flow
                // make an assertion
                assert(contents.toString(), "expected contents of the file");
            });
    };
});

Now this test will asynchronously read the file, make an assertion, and when completed the returned promise will notify the test runner that the test is complete (or failed) and continue on to the next test.

Tests can be organized into groups by simply creating nested objects:

require("patr/runner").run({
    testGroupA: {
      testFile: function(){
        ..
      },
      ..
    },
    testGroupB: {
      testAnother: function(){
      ....
    }
});

(Also, remember all your test properties must start with “test”.)

The recommended approach for creating test modules is to use the exports object to hold tests. This makes it very easy to run a test module directly or as part of a group. For example:

//something.js:
var assert = require("assert");
exports.testSomething: function(){
    assert.eq(3, 3);
};

if (require.main === module)
    require("patr/runner").run(exports);

Now we can directly run this module to run these tests. We can also include this module in another set of tests:

//all-tests.js:
exports.testSomething = require("./something");
exports.testAnotherGroupOf = require("./more-tests");
if(require.main === module){
    require("patr/runner").run(exports);
}

We now have each of these tests within a sub group for easy execution of groups of tests at any level in the hierarchy (all the tests or individual groups).

Performance Testing

Patr is also great for performance testing. You can set an iterations flag with in a test group or from the command line to run through the test a specified number of iterations and give performance statistics.

Summary

Patr is a lightweight test runner that makes it extremely simple to create test suites and hierarchies with idiomatic JavaScript and asynchronous tests with promises. Patr is built on Promised-IO, so it is ready for cross-platform execution, including Node, Narwhal, and within the web browser.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Promised-IO

Promised-IO utilizes promises as an abstraction for I/O operations on top of Node, Narwhal/Rhino, and the browser (where possible). This serves two purposes. First, this package provides the benefits of promise usage: clean separation of concerns and proper encapsulation of eventual values. Second, Promised-IO provides a consistent normalized interface for I/O that will work on multiple platforms without sacrificing any of the advantages of asynchronous I/O, making it easy to build modules that can be used by developers on many platforms.

Usage Principles

One of the characteristics of a good developer is understanding when to apply different abstractions (rather than trying to force a single approach on every situation). Asynchronous code is often more complicated than synchronous code, but can provide big benefits to performance and scalability. In the application of asynchronous I/O there are several different approaches, and each has an appropriate usage.

Synchronous (don’t use asynchronous)

Of course synchronous calls are usually simpler (no addition of callbacks), and for parts of applications that aren’t “hot” parts of concurrent code paths, and aren’t performance sensitive, this can be the smart choice. For the startup process for servers, long period scheduled tasks, and infrequently executed code, the simplicity and code manageability of using synchronous code can outweigh any performance benefit of asynchronous code.

(There are actually times when synchronous performs better than asynchronous. Asynchronous operations have the additional overhead of handling the low-level callback, then adding the JavaScript callback to the event queue, and finally pulling events of the event loop. Synchronous operations do not have this overhead. For example, for file operations that hit the OS cache, the synchronous operation can often be about twice as fast (half as much CPU time) as the asynchronous counterpart. Of course this is only advantageous for cached data.

Synchronous operations can have a much larger performance advantage when there isn’t to-the-core support for asynchronicity. Fortunately, most of the I/O operations in NodeJS leverage libeio and low level async Posix operations for good performance, but for actions that are natively synchronous, some libraries use libeio’s eio_custom function to make it asynchronous. This function can be very expensive and should only be used for operations that will take a significant amount of time (more than about 0.1ms). I have observed synchronous operations executing hundreds of times faster than the asynchronous counterparts when eio_custom is used.

All this being said, most of the time asynchronous is a good choice for code that needs to scale, but it is important to know when and when not to use it.)

Event Registration

This is the typical way that one is notified of events in the browser and in NodeJS. You register for an event and are notified of the event anytime after the registration (no notification of past events). Event registration is the most generic and low-level asynchronous API. NodeJS uses this model exclusively because it is a low-level platform for I/O. For some situations (see below) it is appropriate to build abstractions on top of this API. However, event registration is the right choice if multiple events may take place and you only want future events.

Promises

An asynchronous abstraction designed specifically to model an action that has a single eventual completion (successful or failing). Promises act much like a value (returned from the result of a function call or computation), except that the result is asynchronous, and may or may not be immediately available. Promises are more specific than event registration, providing encapsulation (that can easily be passed around) of an eventual value and decoupling the receiver from the timing of the underlying events that fulfill the action (whether they be past or future). Promises are the right choice for asynchronous actions that have a single point of eventual completion.

Streams

Streams are an asynchronous abstraction for the progressive flow of sequential data. Streams are important for allowing a sender to feed data to a receiver without requiring large scale buffering at either end. Promised-IO implements streams with “lazy arrays”, an array-like interface that follows the standard API of JavaScript Arrays. This greatly improves the modularity of streams since they can be used with code that expects standard arrays. Lazy arrays are indeed lazy, following the functional programming principle of lazy evaluation as well. Iterative methods including map, some, filter, and every are purely functional and only iterate through the stream as needed, avoiding any unnecessary buffering or computations. Promised-IO’s streams also utilize promises to signal the eventual completion of a stream (for the end of a file or end of a HTTP response, for example), for consistency. For situations where you want to model sequential data, streams/lazy arrays are appropriate.

Promised-IO Modules

There are several key modules available in Promised-IO for access to promise-based I/O.

fs

The “fs” module implements key parts of the CommonJS file system API (the entire API will eventually be supported). It also provides Node’s file system API where each asynchronous function returns a promise rather than taking callback parameters (fortunately these two APIs can easily coexist). This gives you access to a number of the convenience functions from CommonJS’s API (like makeTree()) and gives normalizable access to asynchronous functions. This module is obviously not available for browser usage. Here is an example of using readFile() (an asynchronous function from Node’s API):

var fs = require("promised-io/fs");
return fs.readFile("my-file.txt").then(function(contents){
   // once it is loaded, we can do something with the contents
});

One of the most important functions in the “fs” module is the open() function which can be used in several powerful ways. The open function returns a file descriptor for use with Node’s functions that take a descriptor (the read, write, and close functions). While the open function opens the file asynchronously, the returned object can be directly passed to read, write, and close functions (they will execute once the file is open):

var fs = require("promised-io/fs");
var myFile = fs.open("my-file.txt", "r");
fs.read(myFile, buffer, offset, length, position).then(function(){
  // buffer should now be filled
});

The returned object is also a promise for the completion of the open operation:

var when = require("promised-io/promise").when;
var myFile = when(fs.open("my-file.txt", "r"), onSuccess, onFailure);

The object returned from open also has methods (based on asynchronous versions of the CommonJS filesystem API) for writing and closing:

var myFile = fs.open("my-file.txt", "a");
myFile.write("some data").then(myFile.close);

And finally, the returned file object can be treated like a JavaScript array for streamed reading (lazy arrays):

var myFile = fs.open("my-file.txt", "r");
myFile.forEach(function(block){
    // called for each block of data
}).then(myFile.close); // The forEach returns promise for the completion of the reading

The some() JavaScript array method can be particularly useful if you might not need to traverse the whole file:

var myFile = fs.open("my-file.txt", "r");
myFile.some(function(block){
    // if we are searching for something, we can exit be returning true at any time
});

One other useful aspect of using these lazy arrays is that we can return the file object directly as the body of JSGI responses and it will automatically be piped to the client.

function SomeJSGIApp(request){
  return {
    status: 200,
    headers: {"content-type":"text/plain"},
    body: fs.open("my-file.txt", "r");
  };
}

http-client

An HTTP client that follows the JSGI API. Originally developed for handling incoming HTTP requests (for web servers) and returning responses, http-client effectively uses the JSGI API for the reverse situation, allowing for the construction of JSGI requests and receiving JSGI responses. This module allows for some extra convenience properties and defaults to make it easy to create a request. You can simply provide a “url” property to indicate the target, and then all other properties are optional. For example:

var request = require("promised-io/http-client").request;
request({
  url:"http://somesite.com/data",
  method: "GET", // optional
  headers: {} // also optional
}).then(function(response){
  response.status -> http status code
  response.headers -> http response headers
  response.body -> A lazy array stream representing the body of the response
});

This module can be used on the server and browser.

delay

This module provides promise-based scheduling and timing similar to that of setTimeout and setInterval, but utilizing promises. The two exported functions are delay(ms) and schedule(ms). The delay() function returns a promise that is fulfilled after the specified amount of time. The schedule() function returns a lazy array that iterates periodically by the given amount of time. For example:

var delay = require("promised-io/delay").delay;
delay(2000).then(function(){
  // and two seconds later this executes
});

This module can also can be used on the server and browser.

process

This module implements the CommonJS “system” module API and provides access to a print() function and process arguments.

Summary

Promised-IO provides a robust, cross-platform I/O system based on the proven design principles of promises for efficient, portable, asynchronous JavaScript.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

However, building a real-life real-time application involves more than just a platform that gives you asynchronous communication, there are a number of other important techniques to understand. We will look at these techniques and introduce project Tunguska that provides some helpful tools to assist in building applications. While a number of Comet projects out there attempt to provide a black box solution to Comet, Tunguska recognizes that most real-time applications involve deep integration into the application and its security, messaging, and data structures. Consequently Tunguska is a set of tools for building real-time applications rather than a closed black box that can’t easily be integrated with. Let’s look at some of these tools.

Connections/Message Queues

The most fundamental concept of building real-time applications is providing the ability to asynchronously deliver messages that occur at any time, to the browser. With new technologies like WebSockets, this is relatively straightforward. You can create a connection to the server and then the server is free to send messages to the browser at any time. However, WebSocket capable browsers are still nowhere near widely distributed enough to rely on (and the WebSocket protocol is still in development, a recent revision rendered the first version incompatible, and more changes are likely to occur due to the immature design).

The most widely used technique for Comet is known as long-polling. With long-polling an HTTP request is made to the server which waits until it has a message to send. The message is sent to the browser as the response, and upon receiving the response the client makes another request to the server. However, the complication of this strategy is that the server does not have a continuous, uninterrupted stream to send messages to the client. There is a time gap between each message/response being sent from the server and the next long-polling request from the browser reaching the server.

However, we obviously don’t want to miss the messages that were generated during these time gaps. Therefore we need to provide a virtual connection by creating a queue where messages can be stored during these time gaps between the response and request. These message queues allow us to effectively emulate a continuous connection, whereby messages can be delivered at any time to the virtual connection. If an active long-polling response is waiting the server can immediately deliver the message, otherwise the message can go into a queue to wait for the next request from the browser to send a message.

Tunguska provides this type of virtual connection or message queueing with its queue module, and message queues can be accessed with the getQueue(clientId) function. Queues are identified by a client id. This function will return a queue object for a given client id passed as the argument. The queue object extends JavaScript’s Array (providing array functions for accessing queued events), provides a send() for delivering events, a message event, and a close event (accessible via addListener() or observe()) as the key functionality.

var getQueue = require("tunguska/queue").getQueue;
// get the virtual connection for this request
var queue = getQueue("32a3fa5");
queue.addListener("message", function(message){
   // this will be fired by the send call below
});
// put a message in the queue identified by this request.
queue.send("Hello");

The queue module does more than just provide a queue object, it also handles timeouts of non-active connections so as to avoid unbounded memory usage, or memory leakage due to accumulation of unused connection objects. It also handles distributed message queues (across multiple processes/machines) as we will see later.

Tunguska also provides a “jsgi/comet” module for the higher HTTP level interaction. A message queue, which acts as a virtual connection to a client, can be obtained from a request object (JSGI or Node) passed into its getClientConnection(request) function. This function will look for the “Client-Id” header to use as the client id for connection identification. If a connection exists with the given client id, it will be returned, otherwise a new connection object/message queue will be created and returned.

The connection object is based on the WebSocket API. Sending a message to the client is as simple as calling the send(message) function. The connection object is also an array. New messages are pushed onto the array and any listeners are notified.

var comet = require("tunguska/jsgi/comet");
// get the virtual connection for this request
var connection = comet.getClientConnection(request);
// put a message in the queue identified by this request.
connection.send({from:"Kris", body:"hi"});

Long-polling handler

A long-polling request handler can now listen for messages by initially checking for any messages in the queue for instant delivery and if no messages are ready to deliver to the client, it can listen for message events to trigger delivery. A JSGI middleware appliance is provided with the jsgi/comet module to do just that. We can easily insert this appliance into a middleware stack:

cometApp = comet.Broadcaster(nextApp);

The optional nextApp parameter can be used to define another middleware appliance that can handle any needed subscriptions and/or other wiring to event triggers to relay messages to the virtual connection object. For example:

cometApp = comet.Broadcaster(function(request){
  var connection = comet.getClientConnection(request);
  // user function for registering a user to get user events
  listenForMessagesForUser(
    request.remoteUser, // if we have an authenticated user
    function(message){
      // this can be called when a message needs to be sent to the user
      connection.send(JSON.stringify(message));
    });
});

Message Routing

One of the key aspects of real-time applications is message routing. When an action or event occurs, the message needs to be routed to the appropriate party. In a chat application, when someone sends a chat message, we need to route and send the message to the recipient. A frequently used mechanism for this is a publish/subscribe hub. A hub consists of “channels” or “topics”, and messages can be published to channels on the hub, and subscribers can listen for messages on these channels. This paradigm decouples sending messages and listening for messages. Message publishers do not need to be concerned with who is subscribed and subscribers do not need to be concerned with the details of who and how messages are published.

Pub/Sub Hub

There are certainly alternate approaches to topic-based pub/sub hubs for message routing. Rather than subscribing to a particular named topic, subscribers could provide message filter functions that filter through all messages for appropriate messages of interests. However, this approach does not scale well. As more diverse topics and messages are being sent through a system, a filter often begins to process more non-matching functions and performance suffers. On the other hand, pub/sub hubs scale well. A normal hash-table based hub maintains constant time message routing regardless of the number of topics. This highly scalable approach fits well with the Node philosophy of making it easy to do the right thing (in terms of scalability) and hard to do the wrong thing. By using the named topics, it is very easy to create highly scalable message routing systems.

Creating a very simple pub/sub hub in JavaScript is actually extremely easy. JavaScript’s hash-based objects make a great core of a hub. You can simply create properties on a “hub” object with names that correspond to topics and values that are arrays of listeners. However, more interesting features than simple topic based message routing are often desirable. Tunguska provides a “hub” module with a rich set of publish/subscribe capabilities.

The Tunguska hub supports globbing or wildcard-based subscriptions for listening for a set of topics (rather than only being able to register for a single topic at once).

var hub = require("tunguska/hub");
hub.subscribe("foo/*", function(event){
  // will be fired by the publish below
});
hub.publish("foo/bar", "hi");

It allows for “typed” messages to allow for filtering/routing by type as well as topic (essentially allowing a hub to have a two-dimensional namespace). This hub uses “typed” messages for broadcasting subscription events, sending out a “monitored” event when a topic starts to have subscribers or ceases to have any subscribers.

hub.subscribe("foo/*", "monitored", function(event){
  // will be fired by the subscribe action below
});
hub.subscribe("foo/bar", function(){...});

Tunguska’s hub also supports echo suppression, allowing subscribers and publishers to indicate a client identifier so that published messages do not echo back to the client. These components are important for various applications and are essential for Tunguska’s distributed messaging system, where hubs can be linked together in a cluster.

var clientHub = hub.fromClient("42a92f3");
clientHub.subscribe("foo/bar", function(event){
  // will be *not* fired by the publish below
});
clientHub.publish("foo/bar", "for everyone else");

Clustering

Building Comet applications on a single Node process (single threaded) can be deceptively easy. But, scaling Comet applications to multiple concurrent processes is where the bulk of complexity usually arises. Inter-process communication, non-deterministic request delegation, message framing, and true parallel concurrency can combine to create a formidable challenge compared to the rather simple single-process application development. Fortunately, Tunguska provides clustering support for linking distributed pub/sub hubs together, allowing applications to maintain a simple hub-based message routing system without having to worry about the inter-process interaction complexity.

The core component for building Tunguska clusters is the “connector” module. The “connector” module provides a means of using connections to other processes to coherently link hubs together. These connectors will listen for subscriptions and send subscription requests to other processes, allowing for collection of all messages for a given topic from all processes. A connector can be created by passing it a connection object (that follows the WebSocket API), and the connector will handle the messaging concerns for propagating subscriptions and message routing.

WebSocket connection objects can easily be utilized with actual WebSocket TCP connections to other machines. For inter-process communication on a single machine, multi-node provides easy access to a WebSocket-based connection to the sibling HTTP serving processes. Connecting a set of processes with Tunguska with multi-node is very simple:

var multiNode = require("multi-node/multi-node"),
    Connector = require("tunguska/connector").Connector;
// start the multiple processes
var nodes = multiNode.listen({port: 80, nodes: 4}, serverObject);

// add a listener for each connection to the other sibling process
nodes.addListener("node", function(stream){
  // create a new connector using the framed WS stream
  Connector("local-workers", multiNode.frameStream(stream));
});

Jack/JSGI Support on Other Platforms

In addition to Node, Tunguska can run on any other JSGI-compliant server including Jack 0.3, and should be easily adaptable to RingoJS as well. The main complication with different platforms is creating connectors with alternate forms of cross-process or cross-thread communication. Jack 0.3 utilizes threads for concurrency that implement the WebWorker API and includes special events for connecting to other workers for fully connected workers (achieving fully connected workers, where every worker can talk to every other worker is problematic with the standard worker API since it relies on the less ubiquitous SharedWorker API and requires a high level of name coordination). To use the Tunguska connectors with Jack, you utilize the “jack-connector” module, which provides “worker” events that return connection streams that can be used with the connectors:

require("tunguska/jack-connector").addListener("worker", function(connection){
  Connector("local-workers", connection);
});

Tunguska provides a solid set of tools for building scalable, non-blocking, real-time applications with the flexibility and modularity needed to integrate with essential application logic including custom serialization, authorization/access-control, and middleware routing to truly leverage the asynchronous I/O capabilities of Node. Next up, we will look at how Tunguska, which is a sub-project of Persevere, fits into the greater Persevere ecosystem for data integration with real-time data views, and RESTful content negotiation.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!