[webkit-dev] Status of multithreaded image decoding

[James Robinson]

On Thu, Aug 9, 2012 at 5:10 PM, Alpha Lam <hclam at chromium.org> wrote:

> Hi everyone!
>
> A few weeks ago some folks from company100.net have started a thread of
> multithreaded (parallel) image decoding in WebKit. We have worked together
> since then to get a better idea how to complete this feature. I would like
> to report on the progress and our plan.
>
> (The goal for Chromium is slightly different but it will reuse most of the
> architecture discussed here.)
>
> Master bug: https://bugs.webkit.org/show_bug.cgi?id=90375
>
> In WebKit today image rendering is progressive, meaning that image is
> rendered as bytes are received from the network. This is done through
> ImageObserver<http://trac.webkit.org/browser/trunk/Source/WebCore/platform/graphics/ImageObserver.h>
>  and CachedImageClient<http://trac.webkit.org/browser/trunk/Source/WebCore/loader/cache/CachedImageClient.h> interfaces.
> Multithreaded image decoding uses the same notification architecture,
> clients of BitmapImage are notified when a new region is decoded and
> available for painting.
>
> Today image decoding happens synchronously when a BitmapImage is drawn
> into a GraphicsContext. We plan to use the draw operation as a trigger to
> start image decoding asynchronously. Which means the first draw of
> BitmapImage will get a transparent image, subsequent draws will have the
> most recently decoded bitmap.
>
>
I don't think this approach will work terribly well (at least not by
itself).  It seems to require that we flash at least one frame without the
image data even when we really do have it available.  For instance, imagine
a page with a number of small image resources (small enough that they all
load completely in roughly the same instant), or cached image resources,
that impact the layout of the page.  Today, when the images load we will
redo the layout to accomodate the resources size, then at paint time decode
the images and render them.  The user never sees an intermediate state
unless the resources isn't fully loaded at paint time.  With this proposal,
the user would always see a flash where the images occupy layout space but
are not actually rendered.  I think that will be an unacceptably bad user
experience.

To do this, I think you either need deeper integration with the raster
system to make sure that it actually renders the images on the first paint
(perhaps by deferring the actual raster ops to give the decoder some time),
or kicking off the decode steps sooner and adding synchronization at paint
time to make sure we actually see the pixels.

I'm really happy someone is looking into these difficult issues.

- James

This architecture will take several steps:
>
> https://bugs.webkit.org/show_bug.cgi?id=93467
>
> This modifies ImageSource to be asynchronous. ImageSource is used as the
> public interface for decoding images. By making this interface asynchronous
> individual port can implement parallel image decoding or a similar
> architecture. This change is ready for review. I would like to get more
> feedback on the interface since this touches all ports.
>
> https://bugs.webkit.org/show_bug.cgi?id=93590
>
> Progressive painting of an image may not be possible everywhere. For
> example <canvas> and accelerated-composited <img> requires synchronous
> decoding. We plan to keep synchronous decoding the default case. This
> change identifies code location where asynchronous decoding is possible and
> tell BitmapImage asynchronous image decoding is requested.
>
> After these two changes we will be able to implement multithreaded image
> decoder inside BitmapImage and ImageSource. I will report on the progress
> once we get to this point.
>
> Alpha
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120810/f2918f9c/attachment.html>