Promised-IO

Promised-IO utilizes promises as an abstraction for I/O operations on top of Node, Narwhal/Rhino, and the browser (where possible). This serves two purposes. First, this package provides the benefits of promise usage: clean separation of concerns and proper encapsulation of eventual values. Second, Promised-IO provides a consistent normalized interface for I/O that will work on multiple platforms without sacrificing any of the advantages of asynchronous I/O, making it easy to build modules that can be used by developers on many platforms.

Usage Principles

One of the characteristics of a good developer is understanding when to apply different abstractions (rather than trying to force a single approach on every situation). Asynchronous code is often more complicated than synchronous code, but can provide big benefits to performance and scalability. In the application of asynchronous I/O there are several different approaches, and each has an appropriate usage.

Synchronous (don’t use asynchronous)

Of course synchronous calls are usually simpler (no addition of callbacks), and for parts of applications that aren’t “hot” parts of concurrent code paths, and aren’t performance sensitive, this can be the smart choice. For the startup process for servers, long period scheduled tasks, and infrequently executed code, the simplicity and code manageability of using synchronous code can outweigh any performance benefit of asynchronous code.

(There are actually times when synchronous performs better than asynchronous. Asynchronous operations have the additional overhead of handling the low-level callback, then adding the JavaScript callback to the event queue, and finally pulling events of the event loop. Synchronous operations do not have this overhead. For example, for file operations that hit the OS cache, the synchronous operation can often be about twice as fast (half as much CPU time) as the asynchronous counterpart. Of course this is only advantageous for cached data.

Synchronous operations can have a much larger performance advantage when there isn’t to-the-core support for asynchronicity. Fortunately, most of the I/O operations in NodeJS leverage libeio and low level async Posix operations for good performance, but for actions that are natively synchronous, some libraries use libeio’s eio_custom function to make it asynchronous. This function can be very expensive and should only be used for operations that will take a significant amount of time (more than about 0.1ms). I have observed synchronous operations executing hundreds of times faster than the asynchronous counterparts when eio_custom is used.

All this being said, most of the time asynchronous is a good choice for code that needs to scale, but it is important to know when and when not to use it.)

Event Registration

This is the typical way that one is notified of events in the browser and in NodeJS. You register for an event and are notified of the event anytime after the registration (no notification of past events). Event registration is the most generic and low-level asynchronous API. NodeJS uses this model exclusively because it is a low-level platform for I/O. For some situations (see below) it is appropriate to build abstractions on top of this API. However, event registration is the right choice if multiple events may take place and you only want future events.

Promises

An asynchronous abstraction designed specifically to model an action that has a single eventual completion (successful or failing). Promises act much like a value (returned from the result of a function call or computation), except that the result is asynchronous, and may or may not be immediately available. Promises are more specific than event registration, providing encapsulation (that can easily be passed around) of an eventual value and decoupling the receiver from the timing of the underlying events that fulfill the action (whether they be past or future). Promises are the right choice for asynchronous actions that have a single point of eventual completion.

Streams

Streams are an asynchronous abstraction for the progressive flow of sequential data. Streams are important for allowing a sender to feed data to a receiver without requiring large scale buffering at either end. Promised-IO implements streams with “lazy arrays”, an array-like interface that follows the standard API of JavaScript Arrays. This greatly improves the modularity of streams since they can be used with code that expects standard arrays. Lazy arrays are indeed lazy, following the functional programming principle of lazy evaluation as well. Iterative methods including map, some, filter, and every are purely functional and only iterate through the stream as needed, avoiding any unnecessary buffering or computations. Promised-IO’s streams also utilize promises to signal the eventual completion of a stream (for the end of a file or end of a HTTP response, for example), for consistency. For situations where you want to model sequential data, streams/lazy arrays are appropriate.

Promised-IO Modules

There are several key modules available in Promised-IO for access to promise-based I/O.

fs

The “fs” module implements key parts of the CommonJS file system API (the entire API will eventually be supported). It also provides Node’s file system API where each asynchronous function returns a promise rather than taking callback parameters (fortunately these two APIs can easily coexist). This gives you access to a number of the convenience functions from CommonJS’s API (like makeTree()) and gives normalizable access to asynchronous functions. This module is obviously not available for browser usage. Here is an example of using readFile() (an asynchronous function from Node’s API):

var fs = require("promised-io/fs");
return fs.readFile("my-file.txt").then(function(contents){
   // once it is loaded, we can do something with the contents
});

One of the most important functions in the “fs” module is the open() function which can be used in several powerful ways. The open function returns a file descriptor for use with Node’s functions that take a descriptor (the read, write, and close functions). While the open function opens the file asynchronously, the returned object can be directly passed to read, write, and close functions (they will execute once the file is open):

var fs = require("promised-io/fs");
var myFile = fs.open("my-file.txt", "r");
fs.read(myFile, buffer, offset, length, position).then(function(){
  // buffer should now be filled
});

The returned object is also a promise for the completion of the open operation:

var when = require("promised-io/promise").when;
var myFile = when(fs.open("my-file.txt", "r"), onSuccess, onFailure);

The object returned from open also has methods (based on asynchronous versions of the CommonJS filesystem API) for writing and closing:

var myFile = fs.open("my-file.txt", "a");
myFile.write("some data").then(myFile.close);

And finally, the returned file object can be treated like a JavaScript array for streamed reading (lazy arrays):

var myFile = fs.open("my-file.txt", "r");
myFile.forEach(function(block){
    // called for each block of data
}).then(myFile.close); // The forEach returns promise for the completion of the reading

The some() JavaScript array method can be particularly useful if you might not need to traverse the whole file:

var myFile = fs.open("my-file.txt", "r");
myFile.some(function(block){
    // if we are searching for something, we can exit be returning true at any time
});

One other useful aspect of using these lazy arrays is that we can return the file object directly as the body of JSGI responses and it will automatically be piped to the client.

function SomeJSGIApp(request){
  return {
    status: 200,
    headers: {"content-type":"text/plain"},
    body: fs.open("my-file.txt", "r");
  };
}

http-client

An HTTP client that follows the JSGI API. Originally developed for handling incoming HTTP requests (for web servers) and returning responses, http-client effectively uses the JSGI API for the reverse situation, allowing for the construction of JSGI requests and receiving JSGI responses. This module allows for some extra convenience properties and defaults to make it easy to create a request. You can simply provide a “url” property to indicate the target, and then all other properties are optional. For example:

var request = require("promised-io/http-client").request;
request({
  url:"http://somesite.com/data",
  method: "GET", // optional
  headers: {} // also optional
}).then(function(response){
  response.status -> http status code
  response.headers -> http response headers
  response.body -> A lazy array stream representing the body of the response
});

This module can be used on the server and browser.

delay

This module provides promise-based scheduling and timing similar to that of setTimeout and setInterval, but utilizing promises. The two exported functions are delay(ms) and schedule(ms). The delay() function returns a promise that is fulfilled after the specified amount of time. The schedule() function returns a lazy array that iterates periodically by the given amount of time. For example:

var delay = require("promised-io/delay").delay;
delay(2000).then(function(){
  // and two seconds later this executes
});

This module can also can be used on the server and browser.

process

This module implements the CommonJS “system” module API and provides access to a print() function and process arguments.

Summary

Promised-IO provides a robust, cross-platform I/O system based on the proven design principles of promises for efficient, portable, asynchronous JavaScript.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!

Pintura is not a traditional MVC web server framework, which often conflate presentation and interaction concerns across the client and server, but rather follows the REST prescription of maintaining simple storage and serialization oriented server also known as thin server architecture or SOFEA. Pintura is designed to cleanly separate the concerns of presentation and interaction on the client, and storage and model logic concerns on the server. This design fits perfectly with comprehensive JavaScript frameworks like Dojo, General Interface, Cappuccino, YUI, and ExtJS that provide client-side MVC. In particular, Dojo has excellent support for standards-based JSON REST interaction that matches perfectly with this server-side framework.

The core design principles behind Pintura:

• Standards should be leveraged as much as possible to maximize interoperability with other projects, and facilitate freedom in composing web applications with the tools of choice. Pintura is based on RFC 2616 (HTTP/REST), JSON Schema, CommonJS, JSGI, JSON-RPC, and other interoperable standards.
• Presentation and data model logic should be properly separated such that clients can provide the user interface and servers can handle storage and application modeling logic. One should be able to develop multiple web clients for a given application data server.
• Storage and data processing concerns should be separated, with pluggable storage options that can easily be switched and combined (such MongoDB plus a MySQL database).
• The best option choice between configuration and convention is neither. Integration between different entities without dependence upon certain programming patterns (convention or configuration) is optimal.
• End-to-end consistency in data (JSON/JavaScript) handling provides the smoothest communication and interaction from client to server and back.
• The most common cases, like standard CRUD style applications, should be very easy, while still making it possible to build more unusual or sophisticated applications.
• A framework should be simple enough to read the source and easily extend or modify. Applications and frameworks should be organized into layers of functionality.
• Utilize object-capability design for security to allow for flexible security design while protecting against confused deputy style vulnerabilities.
• Full support throughout for promises, providing asynchronous functionality (used extensively by Node), while preserving clean encapsulated interfaces.

Improvements over Persevere 1.0

The server-side JavaScript community has dramatically changed since Persevere development started. There have been numerous exciting new server-side projects that have started like node.js, flusspferd, and Narwhal, utilizing the new generation of advanced high-performance JavaScript engines like V8, TraceMonkey, and JavaScriptCore. Persevere’s main focus is on enabling efficient client server interaction and for well-structured web application development, and matching this functionality with powerful new platforms is a great fit.

In many ways, Pintura represents a simplification of Persevere. With an entirely JavaScript-based modular design, it is very easy to dig in and understand how it works, and the complicated Java-JavaScript bridges have been eliminated. The central request handling mechanism is composed of about 13 middleware modules that can easily be rearranged and used on their own. From the ground up, Pintura has a simple, minimal, direct approach to providing REST functionality.

There are numerous other improvements over Persevere 1.0. Pintura introduces a new security system that is based on the object capability model and is therefore much more flexible and powerful than the previous purely role-based mechanism. Much scientific research has pointed to the superiority of capability based design and how it avoids dangerous confused deputy issues. Pintura utilizes “facets” (also known as proxies) to provide fine-grained access control to different stored objects. Each facet defines what actions are available for objects, with the flexibility of programmatic guard functions. In the end, security is essentially as simple as defining what facets are available for given users, and then defining what actions the facets allow.

Pintura also has moved to implementing the W3C proposed standard Indexed Database APIs object store interface for all data stores. Pintura is based around the Perstore persistence library, which implements this simple API consisting of a REST-style interface. The stores expose get, put, and delete methods and is used at multiple levels in the framework for simplicity and consistency, including for facets, model classes, as well as lower level object stores. Furthermore, additional functionality is easily applied to stores with provided store wrappers that add support for inheritance, caching, replication, event notification and more. Not only that, but with the interface likely to be implemented in browsers soon, this API will provide true end-to-end consistency.

Pintura has also embraced a much more asynchronous oriented design. JavaScript in the browser uses shared-nothing event loop concurrency, and server side projects like Node and Narwhal are following suit. This not only improves client-server consistency, but there is compelling evidence that suggests that event-based design performs and scales much better than traditional thread-based web servers. One of the major challenges of event-based systems is the encapsulation of functionality when dealing with callbacks, but promises are used extensively to greatly simplify working with asynchronous code.

Pintura also leverages the link relation definitions specified by the JSON schema specification that is now a submitted Internet-Draft. Link relations superset the functionality of JSON referencing, and allow for much cleaner, more flexible referencing syntax. This further advances the flexibility of a loosely coupled hyperlink-driven REST architecture.

A core feature of Pintura is now content negotiation, following the REST architectural design of permitting multiple representations of a resource. Resources effectively are persistable JavaScript objects, and representations allows these objects to be serialized and deserialized in multiple formats. Pintura has excellent RFC 2616 conformance to content negotiation and provides a number of out of the box media type handlers, such JSON, JavaScript, CSV, and more, while making it easy to create new custom media type handlers.

Finally, Pintura uses greatly simplified querying syntax. It continues to provide powerful URL-based querying, but the format is simplified to work easily with standard HTML form URL-encoding, aligns with the feed item query language, and has an easy to use and extensible call syntax. Pintura provides a parsing mechanism to help developers implement their own custom query handlers.

Role in Persevere 2.0

Persevere 2.0 is not a single monolithic project, but will be a distribution of several web stack configurations composed of various components including Pintura, Perstore, and the store explorer. Pintura and Perstore are still alpha stage software, but the Persevere 2.0 system is shaping up quickly.

The Next Web Stack

Pintura brings together the cumulative advances of modern JavaScript and web architecture design patterns, including object capability based security, event-driven flow, immutable promises, JSON Schema, REST style, and thin server architecture. Coupled with the powerful new CommonJS platforms and remarkably fast JavaScript engines, this framework fulfills a critical role in the next generation web application stack, and is built to adapt to your JS platform of choice. The next article in this series will explore how to get started with Pintura.

SitePen offers beginner, intermediate, and advanced Dojo Toolkit workshops to make your development team as skilled and efficient as possible when creating dynamic, responsive web applications. Sign up today!