An HTTP Proxy server relays requests between the HTTP client (e.g. your browser) and the server—whether it be out on the web, intranet or localhost. When it’s under your control, the proxy is a great place to inspect and debug client-server interactions over HTTP, log and report, tune and tamper with the requests the client makes, and the responses the server(s) produce. In this article I’ll show how to use Charles (one such proxy tool) to help solve a range of common web, and especially RIA, development problems.
The remainder of this article will feature screenshots and descriptions of how to use Charles—a shareware, cross-platform desktop application—but I’ll pause here to mention that it’s only one of many similar tools that fulfill the same or similar functions. You might know or care to look up: Fiddler, Squid, Firefox plugins like Firebug, Poster, RestTest as well as any number of solutions of varying quality in your programming language of choice. Onwards…
Let’s start simple. A question I’ve seen a few times on the #dojo IRC channel is “why do I get a ‘dojo not defined..’ error in the console?” Why is dojo not defined? Chances are it didn’t load, and chances are it didn’t load because the script path was wrong. You want to rule this out before you dig deeper, and you can check this lots of ways: Firebug’s net tab, your access logs. I like to use Charles to flag 404s:
Each request shows up as a row in the Sequence view, with the response status code and other particulars easily visible.
A variant on this issue is “why do I get responseXML is not defined” or “why is my callback being passed null?” This is just one of the side-effects you can encounter if your content is not being served with the expected mime-type. This too is made readily apparent:
Both the icon in the left column of the grid row, and the General tab show this response was delivered with Content-type: text/plain.
Note, because Charles can act as a system HTTP proxy, you can inspect requests from any browser, or in fact, any application that makes HTTP requests.
Here we can see the initial request to itunes.com/version, which returns a redirect, and the next request is answered with an xml document describing the most recent version of the software.
Accurate web mirroring
Part 2 of the web mirroring task is measuring page weight.
Client-Server Inspection and Live Coding
We’ve all lived through the challenging situation where reproducing a particular problem requires stepping several clicks—or worse, forms—deep into an application. I use Charles’ request and response inspection, tampering and repeat requests features to speed up development and troubleshooting in this situation. When you get to the screen or state you’re interested in, you can examine each request and server response in detail: view image, XML, JSON, hex and text responses in the response viewer, or peruse the raw response. At any time, you can edit and replay a request – which allows for a workflow where you can iterate on the server-side code and re-request it from the client. You skip the tedium and noise surrounding the response output, but can remain confident that all session, user-agent and other awkward variables that constitute the client environment are accurate.
In a similar manner, examining JSON and other response from Ajax-style back-channel requests is convenient and easy. JSON can be viewed in its original (likely condensed) format, or a expandable tree format, as can XML.
The JSON tree view shortcuts the mental or actual formatting necessary to review a typical JSON server response—which typically removes newlines and indentation for reduced size on the wire.
Mapping and Rewriting
Many proxy applications offer a way to configure a mapping of urls from one point to another – so a request to sitepen.com/xyz would actually (transparently) send back the response from sitepen.somecache.com/xyz. More powerful still is the ability to create rules that rewrite requests and responses in a fairly arbitrary way to tune, re-jigger or completely mangle a given site/page/resource. In a development context I’ve found this useful for adjusting paths and/or requesting files on my own server in place of those held on the target server. Here’s one example that might resonate:
In this screenshot, I’m using the Remote Mapping editor to ensure requests for the dojo.js on sitepen.com actually get fulfilled with the response from /js/dojo/release/dojo/dojo/dojo.js.uncompressed.js. I could just as easily map to a path on any server.
Using Remote Mapping, you can enable sensible error logging output with meaningful line numbers, practical stepwise debugging and other niceties normally confounded by source code compression and obfuscation – with a simple rule. Actually you can go further: as the files now reside on your own server (and if they didn’t, you can quickly mirror them so they do), you can inject your own code to log, skip branches and generally rummage in code which you may not have easy access to modify. Mapping to localhost URLs works fine, but for this particular use case you might also take a look at the Map Local dialog, which allows URL to local file mappings.
I hope this has given you some ideas on how you could be using an HTTP proxy. I’ve concentrated on Charles because I think it is a great tool, and being Java-based isn’t going to leave half of the readership nonplussed. Fiddler also offers great options, in particular its scripting hooks and the extra response viewing options, to view as chunked/encoded or not. There are numerous features in Charles that I’ve not covered here, including convenient charting and summaries, up/download throttling, reverse-proxying and more. Charles is just one tool in SitePen’s arsenal for performance testing, profiling, and optimization services. I know I’m not the only one working in this way. So, please comment with your thoughts and remember to support Charles and shareware in general!