Queued: API Challenges

By on March 26, 2009 12:06 am

SitePen’s new Queued application works very well with the Netflix API, but the smoothness of this functionality was the result of a lot of research, and trial and error. In fact, this experience led me to propose that future project timelines should budget extra time when working with an unfamiliar API—and even more time when that API is brand new and untested. Netflix released one of the more exciting APIs in recent months and SitePen began to work with it right away. The Netflix team did great work on their API and they were also very helpful with us when we had questions or there was a bug on their end. I can imagine the challenges of setting up a (Netflix) REST API with an existing system and a large and complex library of items was not simple. Integration with the Netflix API presented its own set of challenges to us.


The first step in communicating with Netflix is the Open Authentication handshake. For details on how OAuth works see the OAuth in DojoX blog post by Tom Trenka.

A lot of credit must be given to Joseph Smarr who was an early adopter of the Netflix API, and Eran Hammer-Lahav, a contributor to OAuth who has some excellent articles on the subject. This gave us a lot of information to get started. Our first attempt at the handshake was using the oauth-as3 library. This is an ActionScript version of an OAuth library which I thought would be a prefect fit, considering we were using AIR and would have to deal with cross-domain issues.

Unfortunately, we had problems with the AS3 library. It lacked documentation and the source was referencing code that was missing. I managed to find the missing code, but then ran into a strange problem. One line of code wouldn’t compile with Flex MXMLC because it was requesting a Flash library only found in the Flash IDE. I was concerned because I wanted this to be available in Deft, and therefore not use the IDE—but switched over anyway as we needed to get it to work. Unfortunately, that was not the only dependency. There was also code that demanded something specific from the Flex library. I did manage to get the handshake to work, but never could get subsequent calls to succeed.

My friend and colleague Tom Trenka came to the rescue. Referencing OAuth libraries from other languages, he wrote an implementation of dojox.io.OAuth—and it worked the first time. But what about the cross-domain concerns? It turns out that it was never an issue. AIR doesn’t have the domain restrictions like a browser, since it’s a desktop application with a sandbox model.

While the handshake was done and on the road, OAuth continued to be a bumpy ride. We ran across every HTTP error code possible, most often 401s including:

  • Invalid signature
  • Access token is invalid
  • Expired or bad request token

These error messages are not very helpful and in fact are often misleading. If you have your handshake working, chances are high that you don’t have to revisit it. You usually need to tweak and refine your GET request. DojoX OAuth handled a lot of things for us, like the mandatory alphabetical order of the parameters. In all but a few cases we used GET (POST was used for saving data), but also added a parameter which tells Netflix when it is not really a GET Request: GET method="PUT"

Other things to look for include how the call is assembled. If you have one parameter incorrect or missing, the authentication errors, so you get an error like one of the ones listed above, not something specific like “Missing the position parameter”. We also found success by double encoding the parameters.

User Queue and Movie Ratings

Initially, retrieving the user’s queue of discs worked without a hitch. The code was fairly simple:

dojo.xhrGet(dojox.io.OAuth.sign("GET", {
	url: "http://api.netflix.com/users/" + USERID + "/queues/disc/available",
	method:"GET", // or POST, DELETE, PUT
	load: function(xml, ioArgs){
		// parse xml

The example shown returns a decent amount of data and returns even more with an expand option. But regardless of the parameters, the ratings were not included and required a separate call. Having the user-reviewed or Netflix-suggested ratings next to the movie was a requirement and we had to have them.

The ratings fetch used the same format as the example code above, with the URL for each movie appended. That’s right, the requirement is not an ID or a GUID—it’s a fully qualified URL:


It was quite evident that the GET URL was going to be huge, and sure enough, I started receiving errors after about 50 movies (my personal queue had 85 movies and that’s not very many). What’s funny is the documentation allows us to fetch up to 500 items at a time. In spite of evidence to the contrary on the forums we could not get this call to work with POST. Compounding this difficulty was other problems using PUT, which led us to believe our problems were related to a bug in WebKit.

Not to be deterred, I authored code that would send the GET requests in chunks small enough to not throw errors. But that led us into another problem: an undocumented limit of four Netflix API calls per second. I further refined the code to work as queues that were called at time intervals which kept us within the API call limits.


Search is actually still a bit of a problem for us, as Netflix doesn’t return the same results with the API as it does with a traditional search on their web site. The results are movies that they recommend based on the keyword —not titles that closely match it. Another search problem was related to the Top Movies sections. We retrieved these lists from a public RSS, not the authenticated API. As a result, the data was different and didn’t include the GUIDs needed to fetch additional information on a movie. This required a second call for every movie in these sections.

What is a GUID anyway?

The first element in the returned XML is the id which looks like this:


We started the application using, in this case, 70070417 as the item id. As it turns out, the id is not necessarily unique. We had to retrofit our code to handle what we called a guid, that came from this element:

This not only gave us a definitely unique id, it gave us more information on the “movie”, or in this case, a TV show — which also may be a TV show, a TV season, or a TV series.

Hey, is that a Movie or Not?

During the making of Queued, we had been calling queued items “movies”, even though the item was possibly a TV show. This proved to be short-sighted, because as I indicated above, there are movies, discs, shows, series, and seasons. Late in the game, we ended up changing all of our code to call these items “titles”. We couldn’t even really call them items, because that’s what the encapsulated data coming from Netflix was called. The nomenclature is in general somewhat jumbled. In addition to a title, synopsis, and other descriptive movie properties, a Netflix item contained things like position for the position in your queue, and shipped for the date is was sent to you. Semantically, this is not correct, as some of these properties belong to the user and some to the movie.

We changed the data to an item which contained this personal information, and within that personal information was an object called title, which contained only information relative to the “movie”.

item = {
	id : http://api.netflix.com/users/USERID/queues/disc/saved/70084781,
	position : 2,
	shipped : 10/08/08,
	watched : NULL,
	estimatedArrival : 10/08/08,
	returned : 10/16/08,
	viewed : NULL,
	format : Blu-ray,
	instant : NULL
	title: {
		guid : http://api.netflix.com/catalog/titles/movies/70084781,
		type : movie,
		title : Over Her Dead Body,
		art : {
			small : http://nflximg.com/us/boxshots/tiny/70084781.jpg,
			medium : http://nflximg.com/us/boxshots/small/70084781.jpg,
			large : http://nflximg.com/us/boxshots/large/70084781.jpg
		releaseYear : 2008,
		runTime : 95,
		rating : PG-13,
		ratings : {
			average : 3.3,
			predicted : 3.1,
			user : 4.0
		categories : [
			0 : Romance,
			1 : Romantic Comedies,
			2 : Blu-ray
		formats : {
			Blu-ray : 05/05/2008,
			DVD : 05/05/2008

Separating this contextual data was necessary in order for Queued’s offline caching to work. If a movie is retrieved from a different source other than your queue, say from a search, or Top Movies, that information is cached in the offline storage and reused in all the different areas of the application. The cached movie data needed to be interoperable between Top Movies, search, and user queue.


The Netflix API is a boon to the mashup world. It’s still pretty new however, and may need a little massaging. In the meantime I hope our efforts on Queued will make things easier for other developers to create other extraordinary applications with this great API.