Touching and Gesturing on the iPhone

By on July 10, 2008 11:28 pm

Everyone who owns an iPhone (or who has been holding out for an iPhone 3G) is bound to be excited about a lot of the new things the device can finally do, particularly the introduction of third-party applications. But those of us in the web development community have been itching for something further still: good web applications on the iPhone. This means we need a suitable replacement for mouse events. And boy did we get them! Though at first the APIs seem a little sketchy, once you’ve learned them you should be able to do amazing things in your application.

I’ll start with how to set up the iPhone console, since I found it invaluable while testing. Under Settings > Safari > Developer, you can turn it on or off. Simple log, error, and warn functions are provided (as part of the console object), all of which accept a single object.

My quest to understand the API led me to this Apple Developer Connection page that, while providing pretty thorough documentation about what’s available, left me a little confused about the details. Also, if you aren’t a member of ADC, trying to follow this link will leave you even more confused.

Clearing it Up

Apple introduced two new ideas with this API: touches and gestures. Touches are important for keeping track of how many fingers are on the screen, where they are, and what they’re doing. Gestures are important for determining what the user is doing when they have two fingers on the screen and are either pinching, pushing, or rotating them.

Touches

When you put a finger down on the screen, it kicks off the lifecycle of touch events. Each time a new finger touches the screen, a new touchstart event happens. As each finger lifts up, a touchend event happen. If, after touching the screen, you move any of your fingers around, touchmove events happen.

We have the following touch events:

  • touchstart: Happens every time a finger is placed on the screen
  • touchend: Happens every time a finger is removed from the screen
  • touchmove: Happens as a finger already placed on the screen is moved across the screen
  • touchcancel: The system can cancel events, but I’m not sure how this can happen. I thought it might happen when you receive something like an SMS during a drag, but I tested that with no success
node.ontouchstart = function(evt){
  console.log(evt.pageX + "/" + evt.pageY);
  // OH NO! These values are blank, this must be a bug
}

My first mistake was monitoring these events and trying to get location information from the events (pageX, pageY, etc). After consulting the ADC documentation again, I learned about three event lists that come attached to the object. But I wasn’t sure what they did, so I went back to testing, logging, and experimenting.

It helped when I figured out the problem the Apple developers were trying to solve. With a mouse, you really only have one point of contact: through the cursor. With your hand, you can keep two fingers held down on the left of the screen while you keep tapping the right side of the screen.

Our event object has a list, and this list contains information for every finger that’s currently touching the screen. It also contains two other lists, one which contains only the information for fingers that originated from the same node, and one which contains only the information for fingers that are associated with the current event. These lists are available to every touch event.

We have the following lists:

  • touches: A list of information for every finger currently touching the screen
  • targetTouches: Like touches, but is filtered to only the information for finger touches that started out within the same node
  • changedTouches: A list of information for every finger involved in the event (see below)

To better understand what might be in these lists, let’s go over some examples quickly

  • When I put a finger down, all three lists will have the same information. It will be in changedTouches because putting the finger down is what caused the event
  • When I put a second finger down, touches will have two items, one for each finger. targetTouches will have two items only if the finger was placed in the same node as the first finger. changedTouches will have the information related to the second finger, because it’s what caused the event
  • If I put two fingers down at exactly the same time, it’s possible to have two items in changedTouches, one for each finger
  • If I move my fingers, the only list that will change is changedTouches and will contain information related to as many fingers as have moved (at least one).
  • When I lift a finger, it will be removed from touches, targetTouches and will appear in changedTouches since it’s what caused the event
  • Removing my last finger will leave touches and targetTouches empty, and changedTouches will contain information for the last finger

Using these lists, I can keep very close tabs on what the user is doing. Imagine creating a(nother) Super Mario clone in JavaScript. I’d be able to tell what direction the user currently has his or her thumb on, while also being able to watch for when the user wants to jump or shoot a fireball.

I’ve been saying that these lists contain information about the fingers touching the screen. These objects are very similar to what you’d normally see in an event object passed to an event handler A limited set of properties are available in these objects. Following is the full list of properties for these objects:

  • clientX: X coordinate of touch relative to the viewport (excludes scroll offset)
  • clientY: Y coordinate of touch relative to the viewport (excludes scroll offset)
  • screenX: Relative to the screen
  • screenY: Relative to the screen
  • pageX: Relative to the full page (includes scrolling)
  • pageY: Relative to the full page (includes scrolling)
  • target: Node the touch event originated from
  • identifier: An identifying number, unique to each touch event

For those of you coming from the normal web design world, in a normal mousemove event, the node passed in the target attribute is usually what the mouse is currently over. But in all iPhone touch events, the target is a reference to the originating node.

One of the annoyances of writing web applications for the iPhone has been that even if you set a viewport for your application, dragging your finger around will move the page around. Fortunately, the touchmove‘s event object has a preventDefault() function (a standard DOM event function) that will make the page stay absolutely still while you move your finger around.

Drag and Drop with the Touch API

We don’t have to worry about keeping track of down/up events as we do with mousemove since the only way touchmove is triggered is after touchstart.

node.ontouchmove = function(e){
  if(e.touches.length == 1){ // Only deal with one finger
    var touch = e.touches[0]; // Get the information for finger #1
    var node = touch.target; // Find the node the drag started from
    node.style.position = "absolute";
    node.style.left = touch.pageX + "px";
    node.style.top = touch.pageY + "px";
  }
}

Gestures

This was much easier to figure out than the touch API. A gesture event occurs any time two fingers are touching the screen. If either finger lands in the node you’ve connected any of the gesture handlers (gesturestart, gesturechange, gestureend) to, you’ll start receiving the corresponding events.

scale and rotation are the two important keys of this event object. While scale gives you the multiplier the user has pinched or pushed in the gesture (relative to 1), rotation gives you the amount in degrees the user has rotated their fingers.

Resizing and Rotating with the Gestures API

We’ll be using WebKit’s transform property to rotate the node.

var width = 100, height = 200, rotation = ;

node.ongesturechange = function(e){
  var node = e.target;
  // scale and rotation are relative values,
  // so we wait to change our variables until the gesture ends
  node.style.width = (width * e.scale) + "px";
  node.style.height = (height * e.scale) + "px";
  node.style.webkitTransform = "rotate(" + ((rotation + e.rotation) % 360) + "deg)";
}

node.ongestureend = function(e){
  // Update the values for the next time a gesture happens
  width *= e.scale;
  height *= e.scale;
  rotation = (rotation + e.rotation) % 360;
}

Conflicts

Some readers might have noticed that a gesture is just a prettier way of looking at touch events. It’s completely true, and if you don’t handle things properly, you can end up with some odd behavior. Remember to keep track of what’s currently happening in a page, as you’ll probably want to let one of these two operations “win” when they come in conflict.

In Action

I put together a quick demo:

This is a simple application that showcases the incredible flexibility and power of these APIs. It’s a simple gray square that can have its colors and borders restyled, can be dragged around, and can be resized and rotated.

Load http://tinyurl.com/sp-iphone up on your iPhone and try the following:

  • Keep a finger over one of the colored squares, and put another finger on one of the border squares
  • Try the same thing using two colored squares or two border squares
  • Use one finger to drag the square around the page
  • Pinch and rotate the square
  • Start dragging the square, but put another finger down and turn it into a pinch and rotate. Lift one of your fingers back up, and resume dragging the square around

Can I Do X?

I’m not sure what sort of APIs we’ll be able to build on top of what Apple has provided for us. What I do know is that Apple has given us a very well thought out API.

mousedown and mouseup are events we can easily emulate with this new API. mousemove is a beast. First of all, we only get touch events after the finger has made contact (the equivalent of mousedown) while we get mousemove events regardless of whether the button is down or not. Also, preventing the page from jumping around isn’t something we can automate. Attach a handler to the document and the user wouldn’t be able to scroll at all!

Which brings us to DnD in general. Even though DnD only cares about mousemove in the context of the mouse button being down (the way that touchmove works), we don’t have any way to tell what node the user’s finger is over at the end of the drag (since target refers to the originating node). If a DnD system is to be used, it would have to be for registered drop targets who are aware of their position and size on the page.

Comments

  • Chris

    Cool demo Neil. Thanks for the article.
    Are you using the Canvas renderer with gfx in this demo or is it also supporting svg now?

  • This is all just plain old HTML and CSS with the addition of WebKit’s transform property

  • Pingback: Ajaxian » iPhone Web Goodies: Drag and Drop with Touch, Resize and Rotate with Gestures()

  • Looks like Apple left out Safari full screen mode. They alluded to it a few months back with a meta tag:

    Gestures combined with SQLLite and Full Screen would have allowed us to create web apps very close to natives. I hope it shows up in a later release.

  • Pingback: iPhone Microsites - iPhone Web Development - Gesturing and Touches()

  • Pingback: tlrobinson.net / blog » Blog Archive » Multitouch JavaScript “Virtual Light Table” on iPhone v2.0()

  • Wow, this looks amazing. Will be interesting to see how far one can take this. A version of CanvasPaint that actually works in mobile safari, anyone?

  • Tom

    I think this stuff is still under NDA, or have you heard otherwise?

  • @Tom: with the iPhone 2.0 software live for a few days now, this is publicly discoverable knowledge by introspecting on the DOM using standard JavaScript techniques.

  • Pingback: coderkind.com » Blog Archive » iPhone touching and gesturing()

  • Tom

    @Dylan: good point.

  • Greetings,

    I have written a jQuery plugin that can use the built in Touch methods including ontouchstart, ontouchend, ontouchmove, ongesturestart, ongesturechange and ongestureend.

    You may specify what options you want available for the specified target(s), including: animate, dragx, dragy, rotate, resort and scale

    LINK: http://plugins.jquery.com/project/touch

    DEMO: http://www.manifestinteractive.com/iphone/touch/

    – Peter Schmalfeldt

  • Pingback: Multi-Touch versus the Mouse()

  • Pingback: Blog What I Made » Multimap and OSM Maps on iPhone()

  • gdb

    You can get a touchCancel if you press the lock button while touching the web page.

  • Pingback: RossBoucher.Com » Funk Rock » Blog Archive » iPhone Touch Events in JavaScript()

  • Cool! I only recently found out about the new features, and this article was the inspiration for my iPhone drawing webapp BriteBoard: http://briteboard.thetalogik.com

  • Renato

    Hi! I’m new in the world of java and programming so my skills are not so good yet.
    So sorry for maybe the stupid question but I need a little help.
    In a html page for iPhone I want to make run a java function only when I tap with two finger on the screen. Is it possible? How can I handle that?
    Thank you guys!
    Rex

  • Cool article Neil!
    I wrote an tutorial on how to make drag and drop using iPhone’s Javascript API after reading this article: http://www.nextroot.com/post/2008/09/20/Drag-and-Drop-using-iPhonee28099s-JavaScript.aspx

  • Why does the scale transform sometimes not take effect until after I’ve removed my fingers? I saw this in the video first, then when I tried it on my iPod Touch.

  • Pingback: Touch and OpenLayers at GeoSpiel()

  • Hawkman

    This is cool. Touchmove is driving me nuts at the moment – it only fires after a tap-and-hold unless you preventDefault, which kills scrolling!

  • That’s really cool man. I’m learning with you code :)

  • Pingback: jQuery Touch - Desenvolvimento Web para iPhone - pomoTI()

  • Excellent tutorial. Hopefully I can find the Apple API to the Itouch for web development on the ADC site, * although alot of shuffling through garbage has found me no where *.

    Much Appreciated,

    Aric Holland
    On Demand Programmers Of America
    http://www.odpoa.com

  • Jean

    About the ontouch cancel event… Did you try sliding your finger to the mobile safari toolbar?

  • Excellent post, and undoubtedly will be very helpful. Thanks!

    On the last DnD comment: Couldn’t you use touchmove in combination with mouseUp (instead of touchend)? The mouseUp event still fires as expected in the old way when you release, giving you the new coordinates and/or target.

  • I tried the demo in Android’s ADP 1.5 which supports touch(begin|move|end|cancel), though I couldn’t get much out the demo.

  • By evaluating the accelerometer data (x,y,z) of the device you can define even more gestures. With the following JavaScript code for example you can trigger the event “Pull” and “HoldAndPull” (holding the iPhone vertical for one second and then pull):
    kTolerance = 30; // degrees
    kToleranceSin = Math.sin(kTolerance*Math.PI/180);
    kToleranceCos = Math.cos(kTolerance*Math.PI/180);
    kToleranceSin1 = -1*Math.sin(kTolerance*Math.PI/180);
    kToleranceCos1 = -1*Math.cos(kTolerance*Math.PI/180);
    kPullPower = 1.2;
    aXLow = 0;
    aYLow = 0;
    aZLow = 0;
    stateVertical = false;
    date = new Date();
    now = date.getTime();
    stateVerticalEnd = now;
    kHoldTime = 1000;
    kMaxDelay = 270;
    verticalTime = 0;
    kEventDisplayTime = 700;

    var watchAccel = function() {
    var absX = Math.abs(a.x);
    var absY = Math.abs(a.y);
    var absZ = Math.abs(a.z);
    var kFilteringFactor = 0.1;
    aXLow = (a.x * kFilteringFactor) + (aXLow * (1.0 – kFilteringFactor));
    aYLow = (a.y * kFilteringFactor) + (aYLow * (1.0 – kFilteringFactor));
    aZLow = (a.z * kFilteringFactor) + (aZLow * (1.0 – kFilteringFactor));
    var absXLow = Math.abs(aXLow);
    var absYLow = Math.abs(aYLow);
    var absZLow = Math.abs(aZLow);

    if (absXLow < kToleranceSin && aYLow < kToleranceCos1 && absZLow 0) {
    verticalEnd = a.timestamp;
    verticalStart = 0;
    stateVertical = false;
    }
    if (a.z > kPullPower) {
    browserEvent = ‘PullVertical’;
    eventTime = a.timestamp;
    if (verticalEnd > (a.timestamp – kMaxDelay) && verticalTime > kHoldTime) {
    // we don’t need this anymore, do we?
    verticalEnd = 0;
    browserEvent = ‘HoldAndPullVertical’;
    }
    }
    if (eventTime < (a.timestamp – kEventDisplayTime)) browserEvent = ”;
    document.getElementById(‘state’).innerHTML = ((stateVertical)?’vertical’: ”);
    document.getElementById(‘event’).innerHTML = browserEvent;
    };
    }

  • Pingback: Expand Your Development Skills With Creative Tech Projects | Inspiration | Smashing Magazine()

  • Lee

    I tested the example application and found an annoying thing. Whenever I delay some time placing my finger on the square, iPhone’s copy function is triggered. Does anyone know how to disable that?

  • rahulvyas

    could you make this for iphone in a native app.

  • @Lee : It’s really easy to disable, but you do it with CSS instead of javascript like this:

    * { -webkit-user-select: none; }

    this disables user select (for copy / paste) on all elements in the html page.

  • Pingback: Touch screen Devices and the Web. « Creative Woot()

  • From my testing, changedTouches will contain only a subset of targetTouches.

    So for example, if we add listeners for touchstart/move/end on an element and:
    1) touch starts outside of element
    2) touch moves to the element
    3) touch moves within the element

    Now the touch is moving within the element, but is still not in changedTouches, because it’s not in targetTouches. It is, however, in event.touches.

  • Martin

    Is there a possibility to use this width Safari on Windows 7 on a multi touch screen?

  • Lukas

    Anyone else having problems with this on the iPad? I can’t get the touchend event to work properly; if two fingers touch the screen and I release one, “touches” contains zero elements, and “changedTouches” contains both. Which effectively means that I don’t know which touch has ended. Any ideas?

  • Tim

    Great article. I’m having the same problem as Lukas on the iPad, touchend reports all touches to have ended, and when moving the remaining fingers, the touches are reported by touchestart as new, though with the same identifier as before. So the only way to know which touch has ended (if you have more than one), is to check the following touchstart/touchmove events. Of course if the user has very steady fingers, there won’t be a touchstart/touchmove events immediately.

  • Angel

    Can you use ongesturechange or ongesturestart to scroll a div?

  • @Angel, For the moment, I recommend checking out TouchScroll, http://uxebu.com/blog/2010/04/27/touchscroll-a-scrolling-layer-for-webkit-mobile/ . Dojo 1.6 will include this capability as well.

  • @Tim(from July 23),
    Same problem. touchEnd event seems worthless. Delivers an array of changedTouches but no touches. Also found that event occasionally scrambles touches array, that is, touch1 at element 0 with identifier 12345678 may eventually end up at element 1. Frustrating!

    Firefox 4 seems to have a better handle on implementing multi-touch exposed to JavaScript.

  • I find it really surprising that the touch event doesn’t support offsetX and offsetY that allow you to get the touch position relative to the containing element. The mousemove event in Safari on my desktop supports these properties.

    I had to use:

    var myElement = document.getElementById(“TheElementID”);

    var t = e.touches[0];

    var tx = t.pageX – myElement.offsetLeft;
    var ty = t.pageY – myElement.offsetTop;

    Hope that helps someone else. Took me ages to figure it out.

  • P.S. Is there an easier way?

  • Jelly

    One thing you actually got wrong there is in the sequence of touches. When you place a finger on the screen, and then place another one, removing either one will trigger the touchstart event of BOTH fingers.

  • Pat

    One thing I’ve encountered is that the “click” (legacy) event on iPad fires after a duration, but it fires from the screen position to whatever is then underneath, meaning it is possible to have a touchend event move one node out of the way, and swap another node beneath that position and have it catch the click event. It appears the click event fires after a duration rather than immediately after touchend.

  • Jelly

    Sorry, when I say touchstart, I actually mean touchend. Removing one finger will trigger touchend for all fingers on the screen.

  • Pingback: OCTO talks ! » Ce que jQuery Mobile nous apprend sur le Web Mobile()

  • I tested the example application and found an annoying thing. Whenever I delay some time placing my finger on the square, iPhone’s copy function is triggered. Does anyone know how to disable that?

  • Now the touch is moving within the element, but is still not in changedTouches, because it’s not in targetTouches. It is, however, in event.touches.

  • John Eke

    First of all, thanks a lot for this it really saved me a lot of time getting to UNDERSTAND the TouchAPI.

    I just wanted to mention that the touchcancel event easy to simulate. If you recieve a js alert during a drag, touchCancel gets fired :)

  • iforce2d

    …just wanted to mention another way to get touchesCancelled, it seems that 5 touches is the maximum handled, so when you put the sixth finger on, something gets cancelled – I would expect the first touch gets cancelled but I couldn’t be bothered checking…! :p

  • Why http://tinyurl.com/sp-iphone doesn’t work on my iPod Touch?

  • Excellent article!
    I’m just wondering why “flick” and “swipe” is not part of gesture event which requires 2 fingers. IMHO, “flick” and “swipe” is also application level gesture.

  • Pingback: JavaScript Touch and Gesture Events iPhone and Android « Back To The Code()

  • Pingback: Multitouch in the browser – iPhone only! » Attrakt()

  • Andreas Schneider

    Hi,
    great article. Do you have any ideas about touch events when voice over is active?

    Bests

  • Where do I start?
    What do I do?
    Where do I get Api?
    How do I install it?
    Where is complete example of how this works?
    How do I start?

  • amruta

    What does node stand for in this??

  • Seb

    ‘node’ is the html element that is being touched.

    I’m trying to implement this in to my mouse drag/drop function.

    Is their a value that can be used to test if a touch screen exists?

  • vivid

    Hi, I’m having difficulty working out when a single finger is lifted to fire an event.

    It seems you have done this, but I can’t work out how.

    Could anyone explain how I can achieve this?

    vivid.

  • Pingback: Touching and Gesturing on iPhone, Android, and More()

  • Cindy

    Thanks for the good explanation. I wonder if you have any ideas why I cannot receive these events in live event handling. My entire page is created live. Currently I have to use document.ontouchX and sort out the element that was touched. It would be cleaner to select, with jQuery of course, on my element’s class.

  • Ray Bellis

    You get “touchcancel” if you remove a finger whilst leaving another finger still touching.

  • thx for this great introduction to ipad events :)

  • Ramesh Boosa

    touchcancel: it works when remove finger from screen then click on view in that time it hapen and image is going to be remove from the view

  • Heh, I think he gave examples. The API is built right into the web browser, so just open up your notepad and start typing ;)