Amazon S3 + Dojo

By on July 3, 2008 12:03 am

Dojo‘s improved RPC and new REST services can be used with a wide array of web services. One particular use is Amazon’s Simple Storage Service (S3). Dojo can connect to Amazon S3’s with a trivial proxy and be used as a normal RPC service. Not only that, Amazon S3 is a REST service and therefore it can be used as a data store with Dojo’s new REST implementation of dojo.data, JsonRestStore. You can read and write to your S3 database using the convenient Dojo Data API, and use the data store in Dojo widgets.

s3store.png

Getting Started

To get started with Dojo + S3:

  • First, you must be signed up to use Amazon S3.
  • Dojo uses a simple proxy to communicate with S3 (doing so directly from the browser would be insecure). The proxy PHP script, proxy.example-php, located in /dojox/rpc/s3/, must be renamed to proxy.php to be used.
  • The proxy requires PHP 5 (other languages hopefully will be added) and the following modules:
  • Next you must enter your Amazon access key and secret access key into the proxy.php file on line 3 and 4:
    $accessKey = "access key";
    $secretAccessKey = "secret access key";

Now we can start using Dojo services to connect to S3. You use the Dojo RPC service with the “PROXIED-PATH” envelope:

dojo.require("dojox.rpc.Service");
dojo.require("dojox.rpc.ProxiedPath");
var s3Buckets = new dojox.rpc.Service({
	target:"http://s3.amazonaws.com/",
	proxyUrl:"../s3/proxy.php", // the path to the proxy
	transport:"REST",
	envelope:"PROXIED-PATH",
	contentType:"application/json",
	services:{
		myBucket:{ // name of the service (usually the bucket)
			target:"myBucket", // enter you bucket here
			parameters:[{type:"string"}]
		}
	}
});

To use the S3 as a Dojo data store you can use the S3Store module. First setup an RPC service as shown above and then pass the RPC service to the S3JsonRestStore:

dojo.require("dojox.data.S3Store");
// and create a store for it
s3Store = new dojox.data.S3Store({service:s3Buckets.myBucket});

You can then use the s3Store as a normal Read/Write Dojo Data store. And since the S3Store is an extension of the JsonRestStore, you can use all the additional features of that store including fast, compact syntax, referencing, and offline capabilities are coming soon. With the S3Store you can build sophisticated apps with very thin server architecture; UI logic can be handled on the browser, and database storage can be delegated to Amazon’s S3. This is just one of the many new features available with Dojo 1.2.

Comments

  • Is it possible to use Google Gears and/or a very lightweight server as the proxy and to maintain security?

  • Sorry, I didn’t type that right….

    Is it possible to use Google Gears (Dojo off-line), GreaseMonkey (to inject JavaScript data without putting it on the server), and/or a very lightweight server *running locally* as the proxy and to still maintain security?

    This would be great for delivering rich applications via Amazon S3, without having a dedicated CPU.

  • I don’t see how Gears, GreaseMonkey, or a local proxy would enable you to create a secure client-side only application. As with application (web-based or natively executing), client side code can always be reverse engineered. Any controlled access action/resource must have authorization from a server in order to maintain security.

    Another issue with using S3 directly from a browser (without a server side proxy) is there is no way to send the HTTP requests (and receive the responses) from the browser because of some-origin restrictions.

    Do you have something else in mind?

  • If the accessKey and secretAccessKey are stored locally, what does it matter if the application is reverse engineered? My concept is that you make those user inputs such that they could use their own S3 storage, providing persistent, backed-up, almost limitless, and available-world-wide storage. If the desire is to use the S3 storage of the application author, then putting those keys on the web application author’s host is the obvious way to go.

  • >>> Another issue with using S3 directly from a browser (without a server side proxy) is there is no way to send the HTTP requests (and receive the responses) from the browser because of some-origin restrictions.

    You can use the S3 POST API from browser without any hassle, atleast for uploads.

    >> database storage can be delegated to Amazon’s S3.

    How do you handle the 5 GB limit ?

  • >> This would be great for delivering rich applications via Amazon S3, without having a dedicated CPU.

    If you are connected to S3, I see no reason why apps should be offline. Why cant you be directly connected ?

  • @Jadon: I was definitely assuming that the application author’s S3 account would be the one be used. It would seem strange to me to build an application that requires users to go through the process of setting an S3 account, this seems like a process for devs to go through.

  • Pingback: Release Notes zum Dojo Toolkit V1.2 - dojotoolkit-forum.de()

  • mauro

    I think it would be great to have JsonRestStore to support Google AppEngine
    I found many projects aiming at giving a REST interface to GAE Datastore:
    http://code.google.com/p/app3/
    http://code.google.com/p/python-rest/
    http://code.google.com/p/gae-json-rest/

    also there is a project to give S3 api’s to GAE
    http://code.google.com/p/baltic-avenue/

    Have you got any plan in this sense?

  • @mauro: I certainly hope to have JsonRestStore be compatible with as many servers that provide REST, and in theory if they properly implement HTTP according to the spec, it should work with little effort. I can’t test every server that claims RESTfulness, but I know the gae-json-rest project is by Peter Svenson, and I think he is specifically intending that implementation to work with Dojo’s JsonRestStore. If you test JsonRestStore on any other RESTful servers I would be glad to help if you find any problems/incompatibilities, I would love to see it tested against as many servers as possible.

  • mauro

    Thanks Kriss…
    I see that all the google data services are based on the AtomPublishing protocol, supporting this potocol in full CRUD mode would make dojo compatible with many services and also with GAE…
    do you know if someone is already working on it? I could try something myself…

  • Is there any tutorial to use dojo to send object using S3 POST API?

    It is not possible to use dojox.data.S3Store without proxy, isn’t?

  • Hon Chong Chang

    Recently Amazon enabled / allowed CORS on their S3; which makes this use case immensely useful.

    I’m currently trying to get S3Store working again, to go from a S3 website (through CORS handshake) to a RESTFUL S3 bucket. There’s some difficulty trying to get this to work (the docs on this is less than ideal). Issues with create, etc; I have a ticket open on this with sitepen support at this point.

    Very exciting stuff…

    For those that wonder how you could keep a pure client side logic SECURE, the data could be bounced to a secure gateway which actually verifies the user’s identity via some kind of auth-token and then bounce a redirect back with the signed amazon URL — thereby keeping the actual amazon secret access key secret.