[webkit-dev] Status of multithreaded image decoding

[Dong Seong Hwang]

Thanks for raising this issue. It's a very valid issue. This is a
topic I want to discuss more broadly.

There is no behavior change in rendering small images. Parallel image
decoders decode only large images (over 300x300). Although parallel
image decoders change the rendering behavior of large images, I'm not
sure if this change is unacceptably bad for user experience.

After applying parallel image decoders, all render objects are
rendered quickly because we do not wait for large images to be
decoded. Large images are rendered later when decoding is complete. I
think parallel image decoder is an optimization similar to incremental
image rendering in that both optimizations render contents
progressively.

Large images can certainly cause flashing, but I could hardly
recognize flashing in most sites. My view might be biased, so I
captured video clips in order to listen other folks' opinions.

I tested on two sites.
1. tumblr : http://www.dorothybrowser.com/test/webkitTest/imgdecode/tumblr.html
 There are many large images whose size are about 400x350. I used a
local copy of the original site.
2. 35 images : http://www.dorothybrowser.com/test/webkitTest/imgdecode/bgimage-png/test.html
 This is a very artificial site, but it gives us an insight on the
difference between the reference WebKit and the threaded WebKit.

The test machine is Intel® Xeon(R) CPU X5650 @ 2.67GHz × 6.

In all movie clips, I loaded the site twice. We can see incremental
image rendering until loading is done in the first trial, and then we
can see the different behavior between the referenced webkit and the
threaded webkit in the second trial, which keeps all cached images.

Case 1) Load the tumblr site.
1. serial image decoder
http://youtu.be/HXspMXNt-bI

2. parallel image decoder
http://youtu.be/mJ4zO7pzX0Q
In the secode trial, I can hardly recognize flashing of large images.


Case 2) Load the 35 large images site
1. serial image decoder
http://youtu.be/yyUouFcH6T0
In the second trial, we experienced about 1 sec hang-up becuae UI
hangs up until serial image decoder decodes 35 large images.
In the third trial, I pushed the menu button as soon as I clicked 35
images site link. We can see the menu list after about 0.5 sec because
of UI hang-up.

2. parallel image decoder
http://youtu.be/2GFmCy1L1Ug
In the second trial, we can see the intermediate state for 0.3 sec
even if the resources are fully loaded. But UI hang-up disappeared.


2012/8/11 Alpha Lam <hclam at chromium.org>:
> This is a very valid concern. The question you raised is one topic I want to
> discuss more broadly.
>
> Chromium has a separate rasterization stage so there is some time for
> decoders to work, synchronization can happen in rasterization stage. This
> doesn't mean Chromium won't benefit if decoding starts before draw time.
>
> For other ports without rasterization stage this approach alone would have
> the problem you mentioned. Just like you suggested decoding have to start
> before paint time and then synchronize at paint time. One approach is to
> start asynchronous decoding during layout time. When an image is downloaded
> metadata is decoded synchronously and triggers a layout when intrinsic
> dimension is available. This could be the time to check with current
> viewport and start asynchronous decoding. For image used as background of a
> RenderBox we can also start decoding if the rectangle of the RenderBox
> intersect with viewport.
>
> Alpha
>
> 2012/8/10 James Robinson <jamesr at google.com>
>>
>>
>>
>> On Thu, Aug 9, 2012 at 5:10 PM, Alpha Lam <hclam at chromium.org> wrote:
>>>
>>> Hi everyone!
>>>
>>> A few weeks ago some folks from company100.net have started a thread of
>>> multithreaded (parallel) image decoding in WebKit. We have worked together
>>> since then to get a better idea how to complete this feature. I would like
>>> to report on the progress and our plan.
>>>
>>> (The goal for Chromium is slightly different but it will reuse most of
>>> the architecture discussed here.)
>>>
>>> Master bug: https://bugs.webkit.org/show_bug.cgi?id=90375
>>>
>>> In WebKit today image rendering is progressive, meaning that image is
>>> rendered as bytes are received from the network. This is done through
>>> ImageObserver and CachedImageClient interfaces. Multithreaded image decoding
>>> uses the same notification architecture, clients of BitmapImage are notified
>>> when a new region is decoded and available for painting.
>>>
>>> Today image decoding happens synchronously when a BitmapImage is drawn
>>> into a GraphicsContext. We plan to use the draw operation as a trigger to
>>> start image decoding asynchronously. Which means the first draw of
>>> BitmapImage will get a transparent image, subsequent draws will have the
>>> most recently decoded bitmap.
>>>
>>
>> I don't think this approach will work terribly well (at least not by
>> itself).  It seems to require that we flash at least one frame without the
>> image data even when we really do have it available.  For instance, imagine
>> a page with a number of small image resources (small enough that they all
>> load completely in roughly the same instant), or cached image resources,
>> that impact the layout of the page.  Today, when the images load we will
>> redo the layout to accomodate the resources size, then at paint time decode
>> the images and render them.  The user never sees an intermediate state
>> unless the resources isn't fully loaded at paint time.  With this proposal,
>> the user would always see a flash where the images occupy layout space but
>> are not actually rendered.  I think that will be an unacceptably bad user
>> experience.
>>
>> To do this, I think you either need deeper integration with the raster
>> system to make sure that it actually renders the images on the first paint
>> (perhaps by deferring the actual raster ops to give the decoder some time),
>> or kicking off the decode steps sooner and adding synchronization at paint
>> time to make sure we actually see the pixels.
>>
>> I'm really happy someone is looking into these difficult issues.
>>
>> - James
>>
>>> This architecture will take several steps:
>>>
>>> https://bugs.webkit.org/show_bug.cgi?id=93467
>>>
>>> This modifies ImageSource to be asynchronous. ImageSource is used as the
>>> public interface for decoding images. By making this interface asynchronous
>>> individual port can implement parallel image decoding or a similar
>>> architecture. This change is ready for review. I would like to get more
>>> feedback on the interface since this touches all ports.
>>>
>>> https://bugs.webkit.org/show_bug.cgi?id=93590
>>>
>>> Progressive painting of an image may not be possible everywhere. For
>>> example <canvas> and accelerated-composited <img> requires synchronous
>>> decoding. We plan to keep synchronous decoding the default case. This change
>>> identifies code location where asynchronous decoding is possible and tell
>>> BitmapImage asynchronous image decoding is requested.
>>>
>>> After these two changes we will be able to implement multithreaded image
>>> decoder inside BitmapImage and ImageSource. I will report on the progress
>>> once we get to this point.
>>>
>>> Alpha
>>>
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo/webkit-dev
>>>
>>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>