[webkit-dev] Considering a Touchhover

[Charles Pritchard]

Checking in on this old issue.

We have drawFocusRing(element) in the Canvas 2d spec for keyboard 
accessibility,
we're looking to add setClickableRegion(element) for pointer 
event/spatial accessibility.

Primarily, for apps like VoiceOver on Mobile Safari, and ZoomText on 
windows, we need
the a11y tree to stash bounding box information for interactive elements 
in the shadow dom,
and it needs to be done in a way that's efficient. That information is 
shared by drawFocusRing,
only applies when an element has active focus. setClickableRegion would 
apply
to an element as long as it is in the canvas shadow dom; using an empty 
path could also
reset the bounding information.

As part of encouraging Canvas developers (and as requested by several 
canvas developers)
to use the method, we've discussed using the method on the UA side as 
well, to capture
and forward pointer events. I've found some precedent for delegating 
click events/using an onclick handler,
though most developers would like to see all pointer events bubble up 
from the canvas shadow dom.

The highest priority is to fill in the bounding box information for the 
accessibility tree;
and when the shadow dom in webkit is activated, this should be a trivial 
task. This is not
"retained mode" canvas, as the scripting environment has no access to 
that information, it
simply passes that info to the accessibility API.

Links galore:

Precedent for ".click()", the pointer-event analogue similar to ".focus()":
https://dev.mozilla.jp/localmdc/localmdc_6373.html#Using_hidden_file_input_elements_using_the_click%28%29_method

I believe click() is passed to shadow dom elements in IE9 with focus() 
events, as part of an obligation to the ARIA specs.


Proposals for something like setClickableRegion:

Charles Pritchard (me):
http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0187.html

Jonas Sickling:
http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0195.html

Richard Schwerdtfeger:
http://lists.w3.org/Archives/Public/public-canvas-api/2011JanMar/0081.html


More abstract:

David Singer:
http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0179.html

Edward O'Connor:
http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0024.html


More concrete:

WebIDL demonstrating Edward's proposal, as well as options for 
Richard/Jonas proposals:
(search for Supplemental)
http://lists.w3.org/Archives/Public/public-canvas-api/2011AprJun/0047.html

Richard's earlier proposal:
http://lists.w3.org/Archives/Public/public-canvas-api/2011JanMar/0090.html
http://lists.w3.org/Archives/Public/public-canvas-api/2011JanMar/att-0090/Canvas2DClickableRegion.html#focus-management

CanvasRenderingContext2D implementations:
http://trac.webkit.org/browser/trunk/Source/WebCore/html/canvas/CanvasRenderingContext2D.cpp
https://hg.mozilla.org/mozilla-central/file/ec809c159ad2/content/canvas/src/nsCanvasRenderingContext2D.cpp


Please let me know if there's anything I can clear up about this issue. 
The list of links and discussion is rather lengthy,
but the issue itself is quite simple.

My original thread was about adding a touchhover event, which would 
allow the scripting environment
to manage hit testing for VoiceOver on Mobile Safari. Though it's a 
relatively easy to implement, it does not solve
the underlying issue that VoiceOver and other ATs would do better to 
have bounding box information ahead of time.

At some point, touch screens may benefit from a touchhover event; 
tablets are capable of penpressure and penhover events,
though they simply pass them as mouse events (see Wacom's Bamboo as an 
example) with a serial interface to pressure.


-Charles

On 3/21/2011 12:06 PM, James Craig wrote:
> The canvas "sub-DOM" proposal should allow some of this, but AFAIK, it is not implemented in WebKit yet. I believe we were the first to propose it, though to date I think it is only implemented in IE9. I have not checked IE to verify support, but they have claimed support in IE9. Basically the ARIA-enhanced subtree is accessible to the keyboard and assistive technology, and the DOM structure mimics the UI visible in the canvas. It requires a focus model change to WebKit because the canvas subtree was not originally intended to be accessible in any modality.
>
> http://lists.w3.org/Archives/Public/public-canvas-api/2009OctDec/0026.html
>
> I'm not sure I agree with the need for a touchhover event, but I'd be interested to hear how you think it should work.
>
> James
>
>
> On Mar 11, 2011, at 3:30 PM, Chris Fleizach wrote:
>
>> There's no established API to handle this, but we are working on a W3C proposal
>>
>> http://lists.w3.org/Archives/Public/wai-xtech/2010Aug/att-0079/UserInterfaceIndependence.html
>>
>> to address this.
>>
>> In the meantime, VoiceOver on iOS will call .focus() every time it "hovers" on an item, so you can use that monitor where VO is at any moment.
>>
>> If that doesn't work with<canvas>  tags please file bugs at bugs.webkit.org and CC me
>>
>> On Mar 11, 2011, at 9:43 AM, Charles Pritchard wrote:
>>
>>> Recently, while working on code review / accessibility for a large canvas application, I found that the iOS VoiceOver AT
>>> does not play well with the html canvas element. It can work, but it's not a natural transition from html to
>>> canvas (with "buttons" in it). Using transparent [button] tags work, but it is rather awkward.
>>>
>>> It occurred to me that, when VoiceOver is on, on these mobile devices, that a new "touch" type is active.
>>> For the time being, I'm proposing calling it "touchhover".
>>>
>>> Currently, VoiceOver AT captures touch events, waiting until the user has set virtual focus to an element,
>>> then tapped twice, to set event focus on that element. It works quite well for html elements.
>>>
>>> My thinking is that this state, where it's not passing touchstart/touchmove/touchend events, is
>>> very much like the traditional "hover" events we've used with mice.
>>>
>>> If touchhover events were passed through Mobile Safari, I'd be able to use hit detection on canvas
>>> elements, and set the canvas element to the appropriate title matching whatever the user has focus upon.
>>>
>>> Does this make sense? Should I provide more examples? The purpose here is to enable UI elements
>>> in Canvas to work appropriately with touch-based AT, such as the iOS VoiceOver extension.
>>>
>>>
>>> -Charles
>>>
>>>
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev