Optimized Navigation: Isomorphic Resources

By on February 27, 2015 8:14 am

When building web applications, developers often face the dilemma of whether to use traditional server-side rendered pages, or client-centric applications that rely on Ajax/JSON data requests, with client-side rendering of data. These two approaches also tend to dictate navigation (among other things), leading to traditional page navigation or a single-page application (perhaps with hash-based navigation). With the traditional approach, content is delivered in HTML form, relying on browser-driven page loads for different views. However, modern applications with more sophisticated user interfaces, often use a single-page style, controlling events, view updates, and relying on Ajax-style communication with the server to load additional content, typically in structured object format, like JSON. The traditional approach naturally works well with search engines, and browser navigation and can be quickly rendered by the browser. Single-page applications facilitate well structured user interfaces, with clean separation of presentation and data. But is there a way to combine the best of both worlds?

Some applications have attempted to combine these approaches by rendering some content on the server to HTML, and some on the client-side. Others have attempted to render the same code on both the server and client, sometimes called “isomorphic JavaScript.” But in this post, we will demonstrate a strategy for delivering content in a simple object-oriented HTML format, that minimizes server complexity, maximizes loading performance and scalability, and still preserves the consolidation of primary UI logic on the client, for easy management of user interaction and state. We will also look at combining this strategy with modern browser techniques, leveraging HTML5 microdata and the page history API. This will enable fast, searchable pages while still retaining the architecture of single-page applications with a single clean presentation codebase based on client-side rendering of structured data.

A Comparison

To approach this strategy, let’s review the benefits of each the two main approaches. First, traditional pages:

  • Browsers can immediately begin rendering, without waiting for JavaScript to load and execute.
  • Page navigation naturally works, with easy bookmarking and back/forward support.
  • Pages can easily be indexed by search engines.

And advantages of single-page applications:

  • Rendering processing is distributed to clients, reducing server workload, and improving scalability
  • UI code is easier to write since it is on one machine, making it easier to create data models, preserve state, etc.
  • Page navigation can be quicker, since only the raw content needs to be retrieved, without any page reloading or rerendering.
  • Content can be transferred in object format, retaining the same conceptual form we generally use in programming.

From this list of advantages, let’s assimilate a strategy that can combine these benefits:

  • Deliver content in HTML for initial page load
  • Deliver content in JSON for later transitions, keeping in sync with page history
  • Deliver content with minimal presentation/rendering overhead, deferring more complex presentation additions to the client


The last point is key to understanding this approach. Rather than treating “rendering” as a single, monolithic transform to the final HTML/DOM structure, we recognize that translation of content into the user interface can be done in granular steps, with the transform of the core content to HTML, as a basic first step, and perhaps the only step that might need to be done on the server. While we won’t explore every layer that you could employ in an application, you can consider a visualization of how an application rendering could be broken down.

Consistency of Object Structure

So how can we deliver on this strategy? A key feature of the single-page architecture is the use of object structured content that can be rendered through JavaScript views without relying on page reloads. There are a plethora of rendering technologies available, some relying on templating, others on programmatic DOM creation. ReactJS even extends the JavaScript language to provide reactive capabilities for rendering DOM elements with in-language syntax.

However, to achieve true page-driven content, we need to deliver the data for a page as HTML. We will do this in a way to retain our structured data directly in the page. Generally, HTML can be an ambiguous and cumbersome source of data, but we can easily define a simple structure for HTML that conveys the natural object structures that we use to store data as well as programmatically interact with data.

This approach of using different formats (HTML and JSON) that have clear, minimal, and unambiguous transformations to object structures, could be called Isomorphic Resources. The use of these different isomorphic formats can then be optimized for initial (HTML) and later (JSON) usage. Not only is this an alternative to isomorphic JavaScript, but the well-defined mapping of transformations back and forth between different representations actually precisely matches the mathematical definition of isomorphic. Using multiple formats for a resource is also a best practice of REST architecture.

One approach we can use for defining object structures in HTML is to use the microdata standard, which provides a predictable structure for data such that it can easily be converted to JavaScript objects that can be rendered. Not only can microdata be clearly converted to JavaScript structures, but it can also be clearly interpreted and indexed by search engines, and with even more precision and clarity than standard HTML pages.

Again, traditionally, with Ajax applications, JSON is the preferred data format. It is succinct, and can easily and quickly be parsed into JavaScript object structures. And we can still use JSON as we retrieve data dynamically from the page. But, for our initial page load, we will embed the initial data directly in the HTML using microdata semantics, as an alternate format for our data.

Example

Let’s look at an example to see how we can do this in practice. Imagine we are building an application (or at least a part of an application) that displays a list of different products. We will use a dgrid to display the list of products, which is a distinctly sophisticated and commonly used client-side widget. Again, as with single page architecture, once a page is loaded, we typically will want to use JSON to transfer the data. We may setup a RESTful service that provides information on a product by id, responding with JSON:

Request:

GET /Product/?category=food
Accept: application/json

Response:

[
    {"id":"oj", "name":"Orange Juice", "price": 3.49, "quantity": 65,
            "category": "food"},
    {"id": "bread", "name":"Bread", "price": 2.99, "quantity": 32,
            "category": "food"}
]

This will work great for our in-page Ajax requests. But we will also setup our server to provide this same data in HTML form, so that we can load a page directly with this data embedded in it. There are numerous ways you could indicate which format to use for a particular product. But, if we want to really adhere to REST/HTTP semantics, we should use the Accept header (although we could certainly use an alternate means of indicating this). We can then make a request to the same resource and get the data back in HTML form. Our strategy here is again to return the bare minimum needed to convey the data, and provide a reasonable presentation/styling of that data while we wait for the JavaScript to load:

Request:

GET /Product/?category=food

Response:

<!DOCTYPE html>
<html>
    <head>
        <link rel="stylesheet" type="text/css" href="ui.css">
    </head>
    <body>
        <table class="page-content">
            <tr itemscope itemid="oj">
                <td itemprop="name">Orange Juice
                <td itemprop="price">3.49
                <td itemprop="quantity">65
                <td itemprop="category">food
            </tr>
            <tr itemscope itemid="bread">
                <td itemprop="name">Bread
                <td itemprop="price">2.99
                <td itemprop="quantity">32
                <td itemprop="category">food
            </tr>
        </table>
        <script src="../dojo/dojo.js" async
                data-dojo-config="deps: ['app/main'], async: true">
        </script>
    </body>
</html>

It should be clear here how this approach differs from server-side rendering of the user interface. This HTML contains almost no presentation information itself, it is direct, simple output of the data, just in a different format. The presentation of this data is still primarily the responsibility of the client JavaScript code. We do make use of some simple HTML semantics, so we can style this data for reading while the further user interface controls load, but the primary presentation is cleanly encapsulated, and is not spread across server code.

Parsing the Data

Next, we need to be able to actually use this data. With JSON, we simply can pass the data to JSON.parse to parse. With our HTML-based object structure, parsing the data is a little more involved, but certainly not excessively difficult.

Let’s create our top level module, my-app/main, that will parse the page for content, creating an object structure:

define([
    'dojo/_base/declare',
    'dojo/dom',
    'dojo/query',
    'dgrid/OnDemandGrid',
    'dstore/Rest',
    'dstore/Cache',
    'dstore/Memory',
    'dojo/domReady!'
], function (
    declare,
    dom,
    query,
    Grid,
    Rest,
    Cache,
    Memory) {

    function parseObjects(rootElement) {
        var data = [];
        query('[itemscope]', rootElement).forEach(function (itemNode) {
            // process each node with an itemscope, indicating a
            // corresponding object representation, and set the id
            var item = {id: itemNode.getAttribute('itemid')};
            query('*', itemNode).forEach(function (propertyNode) {
                var propertyName = propertyNode.getAttribute('itemprop');
                item[propertyName] = propertyNode.firstChild.nodeValue;
            });
            // add to our array of objects
            data.push(item);
        });
        return data;
    }
    var pageContentElement = dom.byId('page-content');

    var pageData = parseObjects(pageContentElement);
    ...

Data Store

Next, we will setup a data store to provide this data to our user interface component (the dgrid in this case). The store will provide a key piece of functionality for smoothly and consistently switching between initial page-provided data, and subsequent AJAX-requested data. Using dstore, We will setup a Cache store that uses the page data as the cache, in conjunction with a Rest store for later interactions with the server:

// create cached REST store
var RestCache = declare([Rest, Cache]);
// setup the caching store based on the in-page data
var cachingStore = new Memory({
    data: pageData
});
// create our product store with the Rest store and
// the prepopulated cache data from the page.
var productStore = new RestCache({
    cachingStore: cachingStore
});

In this example, we are constructing our data based on the idea that we will be navigating to different “views” of the data, corresponding to different filters applied to the data. For example, we might be loading the page with ?category=food to find all the food items.

// now we create a collection representing the
// current view of the data, a filtered set of product
// items by category
function getCategoryView (category) {
    return productStore.filter({
        category: category
    });
}
var initialCategory = location.search.match(/category=([^&]*)/)[1];
var initialCategoryView = getCategoryView(initialCategory);

// now we declare that this collection can be satisfied from
// data in our cache (the in-page data), we don't need to
// make any request for this data from the server
initialCategoryView.isValidFetchCache = initialCategoryView.allLoaded = true;

User Interface (Grid)

Next we create the user interface component, the dgrid, to display the content with all the behavioral capabilities and functionality that dgrid provides. Now that we have parsed the data from the page content element, we can clear out that element and use it to house the grid:

var pageContentElement = dom.byId('page-content');

// clear out the prior content, we have it in object form now,
// and we will render the grid in its place.
pageContentElement.innerHTML = '';

// create the grid now
var grid = new Grid({
    collection: initialCategoryView,
    columns: {
        name: 'Name',
        price: 'Price',
        quantity: 'Quantity'
    }
}, pageContentElement);

Typically with a dgrid, we will want to style columns in the grid. When we do this we can apply the same styling by including the microdata attributes selectors in our column rules. This will allow us to style the content that appears before our code is actually loaded, and have the same styling applied before and after the grid is instantiated. Again, this helps us to maximize the performance of showing content, in a readable form, before the page is a fully loaded, for a smooth progressive load. For example, we could define the name column to have a width of 50% and bold text, using the dgrid and microdata selectors:

.field-name, /* the dgrid column selector */
.[itemprop=name] { /* microdata selector */
    width: 200px;
    font-weight: bold;
}

At this point, we could now create additional components and controls that are used to interact with the main content. This would include controls for adding, modifying content, or performing other actions (for example, purchasing items). This might also include inputs for performing additional querying or filtering. It could include extra functionality such as real-time chat or status information as well. These are all user interface components that can and should be described outside the content itself.

Navigation

We have now loaded our initial content and rendered it. Next, we need to handle navigation. One of the key benefits of traditional pages is that navigation naturally fits with the browser model of pages, allowing for easy back, forward and bookmark actions. If we are going to provide the best of both approaches, we need to preserve this functionality. In Ajax applications, it is common to use hash based navigation, and if we need to support older browsers, we most likely need to support this as a fallback. However, we can and should leverage the newer HTML5 history APIs to preserve navigation functionality where possible, though it does require support from the server-side to achieve. This new history API allow us to specify real server URLs (not just hashes) to associate with each change in data views in our application.

State Navigation

Traditionally, in-page navigation also relies on hash-based navigation, since it does involve any server coordination. But, fortunately, if we are building on the isomorphic resource approach, where different resources correspond to different views, these URLs can conveniently and correctly map to real server URLs that may be delivered in JSON for an in-page transition, or in HTML for a new page transition.

Whenever the user has initiated an action that should lead to presenting a new set of data or view, we go through a navigation process, so that the navigation goes through the browser’s history. Dojo provides an extensible router component. By default dojo/router will use hash based navigation, but we can readily extend it to implement navigation based on the HTML5 history API. We do this by listening for popstate events, and triggering navigation through the pushState (or replaceState) functions. One additional concern with real paths, is that they may be relative, so we need to handle relativizing URLs.

//HistoryRouter.js:
define(['dojo/router/RouterBase', 'dojo/_base/declare', 'dojo/on'], function(RouterBase, declare, on){
    return declare([RouterBase], {
        startup: function(){
            if (this._started) { return; }
            this.inherited(arguments);
            var self = this;

            this._handlePathChange(this._computePath());
            on(window, 'popstate', function () {
                self._handlePathChange(self._computePath());
            });
        },

        basePath: '',

        _computePath: function (relativePath) {
            // compute full path based on relative path
            var url = relativePath ?
                new URL(relativePath, location) : location;
            return (url.pathname + url.search).slice(this.basePath.length);
        },
        
        go: function(path, replace){
            var applyChange;
            if(typeof path !== 'string'){
                return false;
            }

            path = this._computePath(path);
            applyChange = this._handlePathChange(path);
            if (applyChange) {
                var state = replace ? 'replaceState' : 'pushState';
                history[state]({}, '', path);
            }
            return applyChange;
        }
    });
});

Register Navigation Handling

Once we have the routing infrastructure in place, we can register our response to navigation events. From the beginning, we determined the current category from the URL, and use that to select a filtered collection of data from the store, to display in the grid. We can reuse this functionality to determine the current collection and send that to the grid. This can then be implemented in our navigation handler that we register. We will create a new instance of our custom router, and register our handler:

var router = new HistoryRouter({
    // use my current path as the base path, everything after 
    // the base path is handled by the router paths
    basePath: location.pathname
});
router.register('\\?category=:category', function (event) {
    // respond to navigating to a new category
    var category = event.params.category;
    // use the initial category view if we are in the initial category
    grid.set('collection', initialCategory === category ?
        initialCategoryView : getCategoryView(category));
    // update the document title
    document.title = category + ' for sale';
});
router.startup();

If we are navigating to a new category, this will automatically trigger a new request to the server for data. Because we did not mark non-initial categories as cached, it won’t go to the pre-filled memory cache as the initial category view did, but rather go to the master Rest store, which will fulfill a request for data through an HTTP GET request. This will trigger a request with the same URL structure as our main page, but will include an Accept: application/json header to indicate to the server that a JSON response is expected.

In this example, we are only registering a handler for navigating to different filtered views of data to show in our grid. Naturally, an application may have other types of views, perhaps a detailed view of an individual item or product, or an ordering form. We could certainly create alternate handlers for these different views.

This router callback function is basically a reactive function, causing the grid to react and sync with collection changes. This conceptually can be used with any type of component or rendering technology that can react to underlying data changes.

Also remember that if you are supporting legacy browsers, you will still want to fallback to hash-based navigation for these users.

Triggering Navigation

Finally, we need to actually provide the user interface to trigger navigation. There are numerous options for this. In navigation that is guiding a user to filter grid results, we may want to employ a form with inputs for choosing different search constraints. In this example, we are currently simply filtering on the category, and we could use a simple drop-down to choose a category. With each of these, we could respond to a new filtering requests by calling router.go(queryString) with the new query to send to the server. The router would then handle changing the URL through a pushState call, then handle the change, by updating the current filtered collection used by the grid, which would ultimately result in a new XHR HTTP request to retrieve the new results in JSON form.

These types of filtering form controls could all be rendered purely in our client-side code. Here is an example of how we could create a simple HTML dropdown (<select>), which would respond to changes by triggering navigation, using put-selector and dojo/on:

define(['put/selector/put', 'dojo/on'],
    function (put, on) {
        var categorySelect = put('select', [
            put('option[value=food]', 'Food'),
            put('option[value=clothing]', 'Clothing'),
            put('option[value=auto]', 'Auto Parts'),
        ]);
        on(categorySelect, 'change', function () {
            router.go('?category=' + categorySelect.value);
        });
    });

As you can see, we respond to the user selecting a category by calling router.go to navigate to the new category to display. Rather than using a native HTML select, we could alternately do this with a Dijit component, like the Select or FilteringSelect.

Now we may want to have the list of categories be provided from a server-side database, rather than hard-coding the list in the UI code. On the server-side, it may be convenient to actually dynamically retrieve the list of categories with something like an SQL DISTINCT call (for example SELECT DISTINCT category FROM Product). We could include this in the HTML page, and retrieve the data as an array, or as a pre-populated store as we did early, and use this to render our select element or widget. This leads us to the next improvement we could make to fully integrate the benefits of traditional navigation into a single-page architecture.

Search Engine Visibility

By using the new pushState API in combination with a router, we have successfully provided idiomatic navigation, and by embedding our content in simple HTML form, we have succeeded in making our content easily visible, for quick pre-JavaScript rendering (or even without JavaScript), and for search engine accessibility. However, there is a still missing piece for full search engine indexing. While we have provided users with a means of navigation, we haven’t made this navigation visible in the HTML. Consequently, search engines can not follow links to additional content that we may have available, and wish to have indexed.

Briefly, we can correct this by adding hyperlinks in our HTML representation of our content. The idea of adding links to resources is a central to HTML and a key part of the REST architecture (called Hypermedia as the Engine of Application State or HATEOAS).

We can accomplish this by following the same pattern of microdata usage. We could generate a set of HTML elements that represent our available categories using standard anchor elements. We can make use of server-provided data, and annotate that data with anchor tags:

    

We can then read the data from our HTML page using the same function as we read the main page content:

    var categories = parseObjects(dom.byId('categories'));

We can then empty this <div> and use the generated array to drive our preferred user interface control (a <select> drop-down or otherwise).

We could also use the same strategy to add hyperlinks to the individual items in the main page content. This could be useful if we are to provide navigation to product details, and we want this navigation to be visible to the search engines for indexing.

Demo

Putting together these examples, we have a demo where you can see the code in action. This is a fairly minimal demonstration, to clearly show the techniques in this post. Again, this is a simple demonstration, and is missing support for older browsers, but this support could easily be added with a fallback to hash-based navigation. Also, this doesn’t have any server-side coordination. The demo page is assumed to be the response for the food category. Other page navigation is emulated through a mocked JSON HTTP service.

Finally, this demo also includes an added one second delay before rendering, to make the rendering progress clearly visible. You should be able to easily see the initial rendering of the content of the page, prior to any JavaScript rendering.

Isomorphic resources demo.

Screenshots

Before JavaScript rendering:

navigation-pre

After JavaScript rendering:

navigation-post

On the Server

The server-side of this type of application could be implemented with virtually any server technology. We have minimized server processing to little more than delivery of content; the main complexity of the user interface is pushed to client-side code. However, we will look briefly at how you could implement this with Persevere. The Persevere documentation includes examples of how you can easily setup a data model, and expose the data through JSON. And fortunately, Persevere already includes support for handling content negotiation, so that if we register an HTML media handler, it will automatically be selected by the Accept header indication. Our main addition we need to provide is the HTML media handler, that will serialize our data as HTML with the appropriate microdata annotations and included metadata (like navigation options).

Conclusion

This post brings together several different technologies and ideas to define a strategy for creating web applications with modern client-centric architecture, yet still preserving the benefits of traditional pages, in ways that hash-based approaches have never achieved. These new technologies are welcome additions, and fit together elegantly to enable a new class of seamless applications. If you are interested in creating advanced applications that can fully leverage these practices and technologies, we would love to help. Contact us for a free 30 minute consultation to discuss your application and how we can assist your organization.

Comments

  • W.S. Hager

    Firstly, it’s a nice theory, but I don’t think you’d use it in practice. Secondly, the demo is slow to parse and won’t reload with another selection.

    I usually feed static HTML to search engines (or visitors that can’t or won’t view the javascript application) and redirect other clients to the javascript application. The idea is the same (isomorphic resources) but without the extra load of downloading and parsing HTML in your app.

    I suspect you wrote this in response to single-page application haters. I have those too. Oh well…

  • I would say this topic is early research, rather than production-ready, but we like to talk about ideas we’re thinking about. Isomorphic JavaScript in general is a big benefit of React, though the approach outlined in this post is a bit different.

  • W.S. Hager

    I’m not familiar with React’s approach. Could you elaborate please?

  • You can run it on the server-side against a virtual DOM, return static JS on the first request and then run JS after that. It’s sort of like using _AttachMixin when working with Dijit, but it’s a bit easier to get working than it is with Dijit today.

  • W.S. Hager

    It’d be great if this tech could be catered without the inevitable React lock-in. I do dislike redundancy, but perhaps that’s just the way it is with this approach.

  • There are some smaller libraries out there. We’re looking at options for making stuff like this work with Dojo 2 and Mayhem

  • W.S. Hager

    The main issues I have with parsing DOM after the page loads are the clunky visual effect and the redundant markup sent over the wire. IMO complex JSON-based widgets are best rendered from an xhr store, while requests to simpler (templated) JSON or static HTML content could be hidden and pulled into the app structure instead of using Ajax.

    How does this approach relate to declarative widgets or web components?

  • Today it’s possible, though not ideal from an API or code perspective, with Dijit’s _AttachMixin, to pull in widgets from the DOM structure. Hopefully we’ll do better with Dojo 2 in this regard.

  • W.S. Hager

    Cool, I’m curious to hear how you would go about it in this day and age.

  • Philippe Soares

    I like the idea, but given the dgrid example, given that we would be feeding the first page of data to the cache store, how would the grid know the total number of items of the “real” collection ? As far as I know this is given to the grid through the Content-Range header coming from an ajax response, right ?

  • Kris Zyp

    Philippe, dstore’s Cache store does not yet support caching partial (paged) results for later partial requests out of the box. This would need to be implemented by defining the total count somewhere in the HTML, overriding the fetchRange method, and retrieving the partial results and total count there. This could certainly still be done following this pattern, it is just a little more involved than I wanted to get into in a blog post.

  • Kris Zyp

    Just ICYMI, the parsing on this page only takes about 2ms according to my profiler. However, the page has an intentional mocked one second delay, so you can clearly see the loading progress and simulate how the main content is visible prior to any JS execution.

  • Kris Zyp

    Actually part of the goal of this approach is to avoid the clunky rendering and/or redundancy associated with the normal parsing DOM approach. A typical declarative dijit page loads with a bunch of half-baked widgets, which are nearly impossible to provide reasonable styling until they instantiated. Or with isomorphic rendering you end up with redundant markup as you said. With this approach, the server avoids the redundancy of sending presentational markup, but just sends a minimal “content”, and with corresponding minimal styling for the content subset of the page, and then the visual controls are not sent half-baked (or redundantly server baked), but are fully instantiated/rendered on the client side as you would do with a normal single page application.

    Again, this demo allows you to see the process. The initial content is simple and easily styled, without any widgets (or placeholders for widget) included, and then after the one second delay, all the widgets are created (client-side) and rendered for the next painted frame.