The Tech of SitePen Support

By on August 19, 2008 12:01 am

SitePen’s Support service is built using a variety of interesting techniques and technologies. Read on to see how we built a system that treats the web browser as a real client tier and bridges the worlds of JavaScript, Python and PHP seamlessly to provide a great experience for our customers.

Starting with the User

Even though this article is about the technology used to implement SitePen’s Support service, it doesn’t make any sense to talk about technology without talking about the user experience. The technology is there to do something. But, what?

With the SitePen Support service we provide support for the Dojo Toolkit, DWR and Cometd open source projects. We’re out to provide customers with the help they need when they need it. At the highest level, we needed to:

  1. Collect support requests from our customers
  2. Act on them
  3. Keep the customer informed
  4. Follow along with the terms of our support contracts

Without those things, we wouldn’t have a service at all. In addition, there are other requirements for making it the kind of service we’d be proud of:

  1. The customer user interface should be very responsive
  2. The user interface should be less like a content-oriented web site and more like an application
  3. People in a company should be able to work together easily (data should be shared)
  4. Customers should be able to bring themselves up-to-date on what’s happening in their account at any time
  5. Email is still a super convenient user interface

Note that our goals were all about providing a great support service, and not about creating software. If there was off-the-shelf software that would do all of the above for us, at the level of quality we expected, we would certainly have used it. But, there wasn’t. Which brings us to…

The Tech of SitePen Support

To realize those goals for the service, we employed a bunch of different tools and techniques:

  • Dojo runs the client-side
  • The client drives the whole interaction
  • The browser speaks JSON-RPC with the server for most operations
  • The server is built on Python’s WSGI standard
  • Client-driven apps have very little “obscurity” to try to hide behind, so we needed to be sure we followed best practices
  • Off-the-shelf help desk software (HelpSpot) handles part of the work for us

The Client-side Runs the App

A typical web app today looks something like this:

Traditional Web 2.0 App Development
Typical modern web app model

Most interactions are decided by the server. The client makes a request, the server gathers data and uses some sort of template engine to format that data and present it. That cycle is repeated over and over again, with the server always deciding what comes next and how the next bit will be displayed. Many apps today add some Ajax to that (that little JavaScript box at the top of the diagram), but very often the formatting of data is handled by the server and the client just uses .innerHTML to drop the fresh content in place.

For our support application, the model looks more like:

Rich Client Application Model
One model for rich client web apps

With this setup, the server is not responsible for the presentation layer at all. The server sends up static HTML files, and the browser does all of the work in displaying the data to the user. This approach was the topic of my PyCon 2008 talk: Rich UI Webapps with TurboGears 2 and Dojo.

If plain HTML is like the modern day equivalent of a green screen terminal, this design approach helps make your browser usable as a real thick client.

Dojo Runs the Client

The entire user interaction in the Support application is driven by the JavaScript client-side code. Dojo is a natural fit for this style of working, with its built in module system, RPC support, interfaces and powerful Dijits.

We set up a very simple “PageModule” system where we use Dojo’s dynamic loading to load a new JavaScript module from the server and then call “initPage” on the code in that module. That will load up any HTML it needs to display and initialize everything for the user. A simple call to spsupport.core.loadContent is all we need to do to move the user onto the next piece of functionality:

loadContent: function(moduleName, params) {
	// Loads a new content section (if necessary).
	// A content section is defined by a Dojo module that
	// has an "initPage" function in it. The module is loaded
	// if need be and then initPage is called.
	if(spsupport.core.currentPage && spsupport.core.currentPage.cleanupPage){
	var module = dojo.getObject(moduleName);
	spsupport.core.currentPage = module;
	if(module && module.initPage){
		// initPage is like _Widget startup(), though you can safely
		// call it often. each "content" resource should implement it's
		// own _started mechanism in initPage, and treat initPage() as
		// if it were a "selectChild()" call
		var thedot	= moduleName.lastIndexOf(".");
		var packageName = moduleName.substring(0, thedot);
		var moduleRemainder = moduleName.substring(thedot + 1) + ".js";
			url: dojo.moduleUrl(packageName, moduleRemainder).uri,
			preventCache: true,
			load: function(js) {	
				if(js && js.initPage){ js.initPage(params); }

With this kind of modular architecture, we could have a giant application in which everything gets loaded on demand. Combining this setup with Dojo’s build system gives us quite a bit of control over exactly when things load, allowing us to balance initial load time with interactive responsiveness. Any significant “single page” application will need this. The Support application is by no measure a “giant” application, but using good application design techniques like this allows us to add whatever features we need to the application without impacting its load time or responsiveness.

URL Dispatch: Not Just for Servers Anymore

Support Login
Support Signup
The page content is decided by the JavaScript in the client, not by the server.

URL dispatch is one of the core features of a server-side web framework. It turns out that putting the client in charge moved some of the burden of URL dispatch to the client! When you hit the front page of the Support site, the JavaScript figures out where you really want to go:

  • /: send the user over to the support page on SitePen’s main site
  • /?login: give the user a chance to login
  • /?signup-(someplan): give the user a signup page with a plan selected

When working with server-side frameworks, you get used to URLs being divided up by slashes and everything after the ? denoting extra query parameters. With the client in control, the slashes tell the server what static file to serve up, and everything after the ? tells the client what to display. Given that we only have three different possibilities there, we didn’t have to get fancy with our URL dispatch. You certainly could write a client-side framework that has many of the same features you get from a server-side framework, if that’s what your application needs. The support application only needed to interpret a very small number of URLs.

Full-stack Framework? Not Anymore!

Our support application doesn’t use server-side templates for the user interface and doesn’t really do URL dispatch. The “full-stack” web frameworks in use today (Rails, Django, TurboGears, CakePHP, Grails, to name a few) are basically defined by their URL dispatch, templates and database support. Given that we didn’t need two of the three of those, we could go a lot simpler on the server than a full-stack framework.

Our server-side code is written in Python. Many components and libraries are now built around the Web Server Gateway Interface (WSGI) specification, making it easier than ever to put a collection of webapp components together. WSGI works well enough to have spawned a version in Erlang.

In our main web stack, we gathered up the following components:

  • CherryPy’s WSGI server, used both in development on its own and in production behind Apache
  • Luke Arno’s Static WSGI app, which serves up static files. This is used primarily in development, and Apache serves up our static files in production
  • Mikeal Rogers’ wsgi_jsonrpc for responding to the RPC requests from the client
  • Ian Bicking’s WebOb for the small number of dynamic operations that couldn’t be JSON-RPC
  • Mike Bayer’s SQLAlchemy for database mapping


In the past, I’ve often used plain old HTTP requests returning JSON results as a convenient and simple mechanism for requesting data and actions on the server. In fact, that was the approach I took in my PyCon talk. For the Support project, we decided to use JSON-RPC instead because it’s a little bit cleaner. In Dojo, making a JSON-RPC request is just like making a function call that returns a Deferred. So, the syntax for using our server-side API was very straightforward. Even better, parameters and return values automatically came across as the correct types (strings, numbers, arrays, etc). The server-side is also simplified, because it does not need to do much in the way of URL dispatch (JSON-RPC POSTs to a single URL) and the server doesn’t need to worry about converting incoming values from strings.

wsgi_jsonrpc is quite easy to work with. We subclassed it to handle our authentication easily and added a neat bit where making a GET request to the JSON-RPC URL would return the service description. That little change made it easy to wire up Dojo on the client:

	url: "/jsonrpc",
	handleAs: 'json',
	load: function(response){
		spsupport.service = new dojo.rpc.JsonService({
			serviceType: 'JSON-RPC',
			serviceURL: "/jsonrpc",
			timeout: 6000,
			methods : response['procs']

This small snippet will synchronously load the RPC service description. It works synchronously because the UI can’t do much until it can make RPC calls for data. Then, it pulls the function names out of the response to create the JsonService. From that point onward, we just make calls to spsupport.service.function_name whenever we need to call the server.

Each available RPC call is simply a Python function in a module that has some extra metadata attached to it. For example, the request_details function is used to look up a support request by ID and return the detailed information about the request:

@params(dict(name='id', type='str'))
def request_details(user, id):
    """Returns the detailed information for a request.
    * id: the request ID
    * object with the detailed information

The @returns decorator is used to mark the function as one that should be available via JSON-RPC, and to also make note of the return type. The @params decorator, combined with the return type listed in @returns, are used when generating the service description for the client. @auth tells our JSONRPCDispatcher subclass that this function requires authentication. Whenever the @auth decorator is present, the first parameter passed to the function is always the user object. The function itself can then perform additional checks. For example, request_details makes sure that the request is from the same organization as the user.

This design makes it very easy for us to write automated tests for the server side code. I’m personally a fan of test driven development in general, and in the next section we’ll see why automated tests are particularly important for this kind of application.

All Out in the Open

When you provide a rich user interface, particularly using Open Web technologies, you cannot count on security through obscurity. You have to assume that people will study your code and learn about all of your “hidden” URLs that make up sensitive APIs. You would never guess that Gmail doesn’t have a public API, given the number of add-ons people have made for it.

Never trust the client code and requests coming from the client. When creating new, authenticated APIs on the server, the first unit test I write is one that ensures that unauthorized users are given the boot. When working with WSGI and WebOb, writing tests that can run without a server is quite easy:

def test_bad_credentials():
    req = Request.blank('/lookup')
    req.headers['Authorization'] = 'Basic HiThere=='
    res = req.get_response(requests.lookup_customer)
    print res
    assert res.status == '401 Unauthorized'

webob.Request.blank gives you a new Request object that is properly populated to look like a real request. You can then make changes from there to set up your test conditions. In the example above, I’m passing along bad authentication information. At the end, I assert that the result of sending bad authentication information is the expected 401 response.

As easy as testing is using WebOb, unit testing the JSON-RPC calls is even more straightforward. We just call the function directly as we would any Python function that we are unit testing. This kind of application setup makes server-side testing a breeze.

Of course, there’s a lot more work required to ensure that you’re handling data securely than just authenticating the users. We confirm that the user is authorized to access the data they are trying to access (with unit tests, of course). Using SQLAlchemy, we are not vulnerable to SQL injection attacks. We also run all of the requests over SSL to make sure that customer data is not grabbed off of possibly insecure networks.

HelpSpot: the Extra Layer in our Stack

Once a user is logged in, they’re taken to the /dashboard/ page where they can review their requests and account information and create new requests. The support dashboard is built around everything I’ve discussed so far. Each “page” in the dashboard is a separate module loaded via the loadContent call, and those modules make JSON-RPC requests to retrieve and update data on the server.

The Support Dashboard
When you first log in, you can see the recent support request activity.

Account information
All of the details of your current support plan are available on one screen.

The server-side software that we wrote is responsible for keeping track of support plans and gathering up support requests for a given organization so that they can be displayed together. The support requests themselves with their complete histories are all tracked by HelpSpot. Within SitePen, we use HelpSpot’s user interface to update requests, and HelpSpot manages all email interaction. This saved us a good deal of implementation work.

HelpSpot is written in PHP, so we can’t directly call its functions from our Python-based server. One reason we chose HelpSpot is that it offers a solid web API of its own. We make simple HTTP requests to HelpSpot and it returns JSON formatted data. All of those requests are between our support application and HelpSpot. There are some instances where we needed to look up or update data in bulk, and HelpSpot did not have APIs specifically for that. Luckily, HelpSpot’s database schema is nicely designed and easy to understand, so there are instances where we also collect data directly from HelpSpot’s database.

Putting it all together

Bringing new developers up to speed on our support project is simple, because we use zc.buildout. zc.buildout creates a sandbox on the developer’s system with all of the pieces they need to work on the project.

Once we’re ready to deploy, we use a Paver pavement file to describe how to package up the software. Our pavement runs the Dojo build system to combine and shrink the JavaScript files and then bundles everything up into an egg file. The pavement also includes a task that will upload the egg to the server. Using zc.buildout also works great at deployment time, because we just have to run “bin/buildout” in the server’s deployment directory.

Creating the Desired User Experience

I think that the tools and techniques we used in building our support application were nifty and different from how most people are building webapps today. Our approach was all driven by a desire to provide a great user experience that meets our four original goals:

  1. Collect up support requests from our customers
  2. Act on them
  3. Keep the customer informed
  4. Follow along with the terms of our support contracts

The right tools helped us to reach these goals without a giant development budget.

Next month, I’ll be writing about the processes and tools we use to manage the support service within SitePen and ensure that we’re always on top of our customers’ needs.