[webkit-dev] Status of multithreaded image decoding

[James Robinson]

On Fri, Aug 17, 2012 at 1:54 AM, Dong Seong Hwang
<luxtella at company100.net>wrote:

> 2012/8/14 Dong Seong Hwang <luxtella at company100.net>
> > Trigger image decoding early in layout and scroll will certainly
> > relieve flashing though it can’t completely remove the problem. We
> > will apply this optimization to parallel image decoder.
> >
>
> I changed parallel image decoder to trigger decoding on layout as
> Alpha suggested. Unfortunately, it does not seem to relieve flashing.
> I tested the change on Tumblr site, but the difference was visually
> indistinguishable.
>

By itself it won't help - you also need to synchronize at raster time (i.e.
block if the image isn't available).  Once the encoded bits are downloaded
off the wire and we've fire the load event on the image, we have to block
the next raster on having the decoded pixels available.

I think the way to look at this is running a race.  Your goal is to have a
non-main thread run a race - decode the images - and try to complete faster
than the main thread would have.  The finish line is the point at which the
raster engine needs access to the decoded pixels for a given image.  At
that point, you have to block the rasterization process until the pixels
are ready.  The start line is sometime after the bits are downloaded off of
the network and before the raster engine needs it.

If you think of the current code in that model, the start line is when we
first attempt to paint an image that isn't in the decode cache and the
finish line is when we hand it to the GraphicsContext.  Those happen to be
a straight call through and there's no useful main thread work to be done
in that time so we decode on the main thread.  The only way to speed this
up with threads would be to use multiple threads to decode a given image or
to split the decode and resize operations across threads.  Useful, but
there's limited utility.

The real trick is to move either the start or the finish line.  Starting
the decode on layout is one way to move the start line forward and provide
more time for the thread to get the image ready before paint time.  You
could even consider starting sooner - such as on network load - but the
tradeoff there is the sooner you start decoding the less information you
will have about how the image will actually be used.  You might start
decoding an image that will never be painted, or decode at the wrong scale
or with the wrong clip.  The closer you get to the actual paint time the
more accurate your information will be.  For instance after layout you can
have a pretty good guess at whether an image will be in the viewport and
the scale, although the layout might change again before painting.  This
indicates that you may want some sort of decoding queue with decoding jobs
that can be added, modified, or removed after being entered into the queue.

The other line to move is the finish line.  You could imagine that instead
of passing GraphicsContext commands directly to the raster backend
(CoreGraphics/cairo/skia/etc) you instead buffer the commands into queue
and play them later.  Then you could start the decoding process when you
hit the WebCore paint call for the image and only block when passing the
buffered commands into the actual backend.  Buffering the GraphicsContext
commands could allow for a number of other optimizations (collapsing draws,
dropping fully obscured draw calls, etc) The raster backend may even be
capable of taking a promise for decoded data instead of the data itself
which would let it push the finish line further back.  This last option is
what we're pursuing in skia as part of other optimizations, but might be
useful for other backends.

I think you have a lot of options here.  I want to emphasize that I think
the user experience is the most important thing to keep in mind with all of
this work.  Displaying the page faster is a great enhancement to a user's
happiness and improving our image decoding can definitely get us some big
wins.  I think any successful project here will have to be invisible to the
end user - we can't have pages flashing in unpredictably.

- James


> However, theoretically there must be some improvements. I want to
> discuss more about it while parallel image decoder is reviewed.

_______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120820/380f8611/attachment.html>