[webkit-dev] Feature announcement: canvas pixel access API additions for high-resolution backing stores

[Charles Pritchard]

On 4/16/2012 3:34 PM, Maciej Stachowiak wrote:
>
> On Apr 16, 2012, at 1:44 PM, Darin Fisher wrote:
>
>> On Mon, Apr 16, 2012 at 1:42 PM, Oliver Hunt <oliver at apple.com 
>> <mailto:oliver at apple.com>> wrote:
>>
>>
>>     On Apr 16, 2012, at 1:00 PM, Darin Fisher <darin at chromium.org
>>     <mailto:darin at chromium.org>> wrote:
>>
>>>     On Mon, Apr 16, 2012 at 12:55 PM, Maciej Stachowiak
>>>     <mjs at apple.com <mailto:mjs at apple.com>> wrote:
>>>
>>>
>>>         On Apr 16, 2012, at 10:52 AM, Darin Fisher wrote:
>>>
>>>>         Could this be an opportunity to design an asynchronous API
>>>>         for fetching the pixel buffer?  It seems like many
>>>>         implementations are GPU backed now, and fetching the pixel
>>>>         buffer is an expensive (blocking) operation.  Had you
>>>>         considered making such a change?
>>>
>>>         Adding async support was suggested on the whatwg thread
>>>         about this. I think async support is useful, but should not
>>>         be tied to high DPI support. In particular, we shouldn't, in
>>>         my opinion, require authors to rewrite their existing sync
>>>         code to an async model just to properly support higher
>>>         resolutions.
>>>
>>>         In addition, the whatwg thread revealed that there are many
>>>         hidden complexities in the design of get/putImageData, in
>>>         particular designing how they work in the face of sychronous
>>>         drawing operations to the same canvas. The HiDPI problem is
>>>         much more straightforward and does not need to be gated on
>>>         resolving the async design issues.
>>>
>>>
>>>     I think you are giving up a good opportunity.  The barriers to
>>>     an async API are more readily overcome when there are extra
>>>     benefits to developers.  HiDPI could be a great way to attract
>>>     developers to a better API.
>>>
>>>     I've addressed those other concerns on the WhatWG thread.
>>
>>     No, gating HiDPI on rewriting your code into a more complex, and
>>     generally slower model seems like a great way to encourage
>>     developers to ignore high dpi.
>>
>>     --Oliver
>>
>>
>> I'm not sure why developers would choose to ignore HiDPI.  It seems 
>> like it helps their apps/pages look better!
>>
>> You really feel like you need to "kowtow" to developers to get them 
>> to adopt HiDPI?
>
> Different developers will have different priorities. HD image data and 
> async readback both have potential benefits in image quality and 
> nonblocking responsiveness respectively. Here is an example of an 
> application using getImageData which would clearly benefit from HD, 
> but it's not obvious that async would be useful:
>
> http://mudcu.be/sketchpad/

Developers are rarely targeting existing HiDPI with iOS via 
window.devicePixelRatio. They have learned though, word spreads around.
That's just an issue of project and community maturity. Canvas on iOS 
requires a lot of advanced techniques for good operation.

I was heavily involved in Sketchpad as well as a Canvas-based image 
editor, Darkroom.
http://www.youtube.com/watch?v=4MrEtsplano#t=2m

In a subsequent version we did enable arbitrary blend modes in drawing. 
Michael put together several dozen modes.
Blocking was and is a big issue. Especially with high resolution devices 
like Wacom pen-tablets.

If there's any blocking due to heavy computations, we end up losing a 
lot of precision of movement and pen pressure.
Some drawing applications have a quick render, which is then followed up 
by a high quality composite (running on pixels).

The actual getImageData call is not a big deal, but regardless, we 
already face the need of an async loop, of possibly dropping frames and 
of keeping various state data around.
In the sketchpad project as you are looking at it, it is not using 
getImageData. It's using a lot of drawImage calls.


> Here is another where HD helps but benefits of async are unclear, 
> since it does a pixel read-write-modify cycle:
>
> http://nerget.com/pressure/pressure.html
>
> Tying HD to async may hurt the adoption of both by requiring 
> developers to take two different code change hits when they only care 
> about one. In particular, the async change is likely to be more 
> invasive to code structure.  If developers are discouraged, they may 
> end up using neither. Thus, in my opinion, these two enhancements to 
> ImageData should stand and fall on their own merits.
>

That project has one getImageData call, it's kept in memory and it's 
initialized from a blank canvas.
It would only need an additional line of code (and argument) to support 
async getHD. It's not a separate code path.

I am trying to say this: an async getImageDataHD will not "hurt" 
developers nor discourage them.

The fluid dynamics demo would benefit from using requestAnimationFrame 
instead of setInterval, and using
webkitPostMessage to do the heavy lifting in the displayDensity method.

It could benefit from multiple-cores (in theory) by using several buffers.

I think there are excellent points to discuss; points that have been 
discussed in this thread. There were calls for benchmarking, weighing up 
the technical overhead.

I want to make it clear that developers would not be harmed by having 
getHD be asynchronous.
I'm happy to write examples, to point out existing code, to point out 
the ease in which it could be patched, and to point out
how existing code may benefit from additional async hooks and other 
advanced Canvas techniques. All of that is relevant to that one issue.


-Charles
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120416/e921784e/attachment.html>