[webkit-dev] image downscaling during decoding

[Ilyes Gouta]

Hi,

> Maintaining N rows is an implementation detail of the scaler. I mean by
> having a simple one-scanline-in, zero-or-more-scanlines-out interface, like
> this:
>
> scaler.setSourceSize(10, 10);
> scaler.setDestSize(100, 100);
> while (decoder.hasMoreScanines()) {
>   scaler.supplyScanLine(decoder.nextScanine());
>   while (scaler.hasMoreScanlines())
>     memcpy(dest.nextOutputScanline(), scaler.nextScanline(), width * 4);
> }
>
> Then what algorithm to use and quality is up to the scaler implementation to
> decide.

I'm still thinking that any scaling (if any) performed by image
decoder should remain implicit, i.e not exposed to the upper rendering
layer.

The abstract ImageDecoder class has a setSize(unsigned width, unsigned
height) method, that's called by the specialized decoder (let's say
JPEG) to set the original width and height of the picture, and a
ImageFrame* frameBufferAtIndex(size_t) = 0 method that return a
decoded ImageFrame frame. Now ImageFrame (or a derived class!) on its
turn has a NativeImagePtr asNewNativeImage() function that returns the
native representation of that picture, so the native buffer. It could
a system memory allocated buffer, or a surface allocated by the
underlying toolkit.

So in order to downscale when decoding a picture, the base class
ImageDecoder would just need to be augmented (a new method) to also
take the desired decoding target dimensions that the upper
rendering/compositing layer wants/expects. With that information, the
decoder can choose to decimate, downscale, upscale, etc. its original
picture in order to meet that target dimension. libjpeg can take
advantage of such info, to downscale *while* decoding (so it comes for
free). frameBufferAtIndex() would then return a frame with the hinted
target dimensions (they could be close to the expected dimensions, not
necessarily equal to) rather than the original ones. The drawing
toolkit/libraries such as Cairo or Qt could then leverage their
resizing routines (which could be h/w accelerated) when blitting the
ImageFrame onscreen when compositing for display.

I'm not sure about the utility of a s/w scaler anymore.

-Ilyes

>
> Alpha
>
>>
>>
>> -Ilyes
>>
>> > nice to have a callback based downscaling API that we can implement
>> > better
>> > algorithms.
>>
>>
>> On Mon, Aug 13, 2012 at 6:52 PM, Alpha Lam <hclam at chromium.org> wrote:
>> > I agree we should have a separate row-based downscaling phase.
>> >
>> > JPEGImageDecoder.cpp already supports downscaling with a given width and
>> > height, but it's using point sampling which has poor quality. It would
>> > be
>> > nice to have a callback based downscaling API that we can implement
>> > better
>> > algorithms.
>> >
>> > Alpha
>> >
>> > 2012/8/13 Alpha (Hin-Chung) Lam <hclam at google.com>
>> >
>> >> I agree we should have a separate row-based downscaling phase.
>> >>
>> >> JPEGImageDecoder.cpp already supports downscaling with a given width
>> >> and
>> >> height, but it's using point sampling which has poor quality. It would
>> >> be
>> >> nice to have a callback based downscaling API that we can implement
>> >> better
>> >> algorithms.
>> >>
>> >> Alpha
>> >>
>> >>
>> >> 2012/8/13 Ilyes Gouta <ilyes.gouta at gmail.com>
>> >>>
>> >>> Hi,
>> >>>
>> >>> On Mon, Aug 13, 2012 at 3:46 PM, Anton Obzhirov
>> >>> <a.obzhirov at samsung.com>
>> >>> wrote:
>> >>> > Libjpeg has some internal support for downscaling already (2x, 4x).
>> >>> > Not
>> >>> > sure
>> >>> > about
>> >>> >
>> >>> > Libpng and other libraries.  In general probably it can be
>> >>> > implemented
>> >>> > by
>> >>> > downscaling decoded row of the pixels on the fly using callback API
>> >>> > provided.
>> >>>
>> >>> If such a callback API for row-based resizing comes to fruition, then
>> >>> let's (also) not rule out whole-picture h/w accelerated resizing -
>> >>> such as by making that new API a mandatory stage in the decoding
>> >>> process.
>> >>>
>> >>> -Ilyes
>> >>>
>> >>> >
>> >>> > From: tomhudson at google.com [mailto:tomhudson at google.com] On Behalf
>> >>> > Of
>> >>> > Tom
>> >>> > Hudson
>> >>> > Sent: 13 August 2012 15:00
>> >>> > To: webkit-dev at lists.webkit.org; Anton Obzhirov
>> >>> > Subject: Fwd: [webkit-dev] image downscaling during decoding
>> >>> >
>> >>> >
>> >>> >
>> >>> > On Mon, Aug 13, 2012 at 9:39 AM, Anton Obzhirov
>> >>> > <a.obzhirov at samsung.com>
>> >>> > wrote:
>> >>> >
>> >>> >
>> >>> >
>> >>> > We are looking for ways to improve page loading speed and reduce
>> >>> > memory
>> >>> > usage for WebKit in general and for GTK port of WebKit in
>> >>> > particular.
>> >>> >
>> >>> > One of the ideas is to implement downscaling of the images during
>> >>> > decoding
>> >>> > for image elements with rectangle less then original image size.
>> >>> >
>> >>> > At the moment such images are full decoded to a full size buffer and
>> >>> > get
>> >>> > downscaled during rendering.
>> >>> >
>> >>> > It can be quite beneficial in term of memory usage and should speed
>> >>> > up
>> >>> > rendering of the pages like image galleries for example.
>> >>> >
>> >>> > So what are your thoughts about it?
>> >>> >
>> >>> >
>> >>> >
>> >>> > Interesting idea! I know Chrome sees memory pressure from the
>> >>> > downsampling
>> >>> > two-step on some pages, so I'd think this is could be useful for us,
>> >>> > too.
>> >>> >
>> >>> >
>> >>> >
>> >>> > Isn't it going to require fairly large and intrusive changes to
>> >>> > several
>> >>> > different third-party libraries, though?
>> >>> >
>> >>> > Also, different ports use different image downscaling algorithms; I
>> >>> > can
>> >>> > think of ways to try to enable incremental downscaling callbacks so
>> >>> > you
>> >>> > don't have to implement N downscales in each of M decoders, but none
>> >>> > that
>> >>> > are both general and performant.
>> >>> >
>> >>> >
>> >>> >
>> >>> > I'd love to see a proposal on this.
>> >>> >
>> >>> >
>> >>> >
>> >>> > Tom
>> >>> >
>> >>> > (from the right account this time)
>> >>> >
>> >>> >
>> >>> > _______________________________________________
>> >>> > webkit-dev mailing list
>> >>> > webkit-dev at lists.webkit.org
>> >>> > http://lists.webkit.org/mailman/listinfo/webkit-dev
>> >>> >
>> >>> _______________________________________________
>> >>> webkit-dev mailing list
>> >>> webkit-dev at lists.webkit.org
>> >>> http://lists.webkit.org/mailman/listinfo/webkit-dev
>> >>
>> >>
>> >
>
>