[webkit-qt] QtWebKit, the multimedia support today and tomorrow.

[Pierre Rossi]

On Fri, Jun 24, 2011 at 17:08, Alexis Menard <alexis.menard at openbossa.org>wrote:

> Hello,
>
> DISCLAIMER : It may be controversial for some people.
>
> For many months, the QtWebKit team has worked on the multimedia
> support coming with HTML5. We had a first Phonon backend, abandoned
> and really far from feature completion and far from working well. When
> QtMultimedia appeared a new backend was added to the tree providing a
> more extensive API than Phonon and supporting more platforms
> (Symbian). This is the default today if you have Qt Mobility
> installed. Unfortunately Qt Multimedia really lacks of QA and
> stability, the quality of the backends is really heterogenous and
> backends behaves differently. I tried hard to communicate with the Qt
> Multimedia team, they helped a lot but even today Qt Multimedia is not
> good enough and I feel the support is built on unstable foundations.
>
> On the other hand last year Simon and me investigated on the
> multimedia support in WebKit2, he took over when I moved to INdT but
> the result can be seen on the N9 Browser. This implementation is not
> using Qt Multimedia but instead a modified version of the GStreamer
> backend sitting in webkit.org and maintained + used mainly by the GTK
> port. The lesson learned from this implementation is that it was hard
> for us (we don't have particular knowledge of GStreamer) to go into
> the code and hacked it the way we wanted it. But with time we did it
> and at the end we are providing a really good and tailored
> implementation, by supporting texture streaming for example, things we
> couldn't have done in Qt Multimedia easily because we were tight to
> the API and their release schedule. While I was in INdT, I worked on
> the GStreamer backend upstream to make sure it was working well on
> regular linux desktop. It does now and has the same features as
> QtMultimedia. You can enable it by passing DEFINES+=USE_GSTREAMER=1 to
> your build-webkit script. At the end the code is fairly small and
> doesn't have the same complexity as Qt Multimedia GStreamer backend,
> at the end we just playback sounds and videos :D.
>
> After this successful effort I investigated other platforms like Mac.
> Again I looked at what Apple is shipping and I used it. If you pass
> DEFINES+=USE_QTKIT=1 when building QtWebKit on Mac you can enable the
> media support using the QuickTime backend the mac port is using. The
> implementation is working fairly well too and actually support more
> stuff than QtMultimedia does : the fullscreen support is working
> better (animated, Tor-Arne will love it) and has playback controls in
> fullscreen using the existing native theming. I found it pretty easy
> to use and It didn't require that much changes in Apple's code to make
> it work.
>
> So the advantages are :
>
> - We have pretty well working implementation on Mac and Linux (and N9
> by extension).
> - We supports more features than QtMultimedia does.
> - We can hack the webkit.org backend to shape them the way we want.
> - The backends are pretty well tested and covered by layout tests.
> - We got improvements for free.
> - It makes us closer to other ports by talking to them (both Apple and
> GTK teams were *very* nice to help me to implement everything).
> - We have a very low dependency on Qt4, both GStreamer and QTKIT
> backend are not using much (QImage, a top level QWidget from what I
> remember) stuff that will anyway be part of Qt5 (you need to be able
> to render into something, and you need a top level window).
> - We don't need to wait QtMultimedia to work on Qt5 to support media
> elements on QtWebKit next gen.
> - We have hard time to make the media layout tests to run reliably on
> the Qt Linux bot, using the GStreamer backend of webkit make us almost
> the same as with GTK (so we can have almost the same configuration).
>
> Now the drawback of using the backends of webkit.org :
>
> - Makes harder to maintain : we have multiple backends and code path
> (but in the other hand we are not alone to maintain the backends).
> - We don't have today Windows support, WebKit don't have native
> Windows backend (unless we use the QuickTime backend on Windows if
> QuickTime is installed). I would like to implement one based on MS
> APIs.
> - Symbian is missing too. Not sure it is relevant for the WebKit2
> activities as well as Qt5.
> - We don't have Mac bots on the build.webkit.org, making hard for
> Apple to check that they didn't break Qt if we use their backend.
>
> At the end I still think that using the upstream backend is more
> beneficial and some of the disadvantages can be solved by having a Qt
> Mac Bot on build.webkit.org and implementing the missing backend for
> example.
>
> So here is my proposal (for trunk) :
>
> - Removing Phonon backend from the tree. I asked on
> http://lists.kde.org/?l=kde-multimedia&m=130779689626212&w=2 and got
> no answer, which to me means nobody wants to take care of it. It
> brings confusion in the webkit community.
> - Make GStreamer and QuickTime webkit.org backend the default on Mac
> and Linux. I could investigate of using the feature of WebKit to have
> multiple player so that if the webkit.org player can't work it could
> fallback on QtMultimedia.
> - Use QtMultimedia on Windows and Symbian (up until Symbian disappear
> and I give a try to the MS backend). In the meantime again I could
> investigate on Windows to use the QuickTime backend by default and
> fallback to QtMultimedia if QuickTime is not installed.
>
> I think this makes us ready for working on a WebKit2/Qt5, helps Nokia
> but keep the desktop world functional too.
>
> Any thoughts? feedback? Please please answer :D.
>
> PS (running all the media layout tests on my mac) :
>
> QTKIT :
> 117 test cases (80%) succeeded
> 13 test cases (8%) had incorrect layout
> 16 test cases (10%) were new
> 7 test cases (4%) had stderr output
>
> QTMULTIMEDIA :
> 79 test cases (54%) succeeded
> 51 test cases (34%) had incorrect layout
> 16 test cases (10%) were new
> 29 test cases (19%) had stderr output
>
> On Linux :
>
> GSTREAMER :
> 109 test cases (74%) succeeded
> 19 test cases (13%) had incorrect layout
> 16 test cases (10%) were new
> 2 test cases (1%) timed out
> 9 test cases (6%) had stderr output
>
> QTMULTIMEDIA :
> 88 test cases (60%) succeeded
> 41 test cases (28%) had incorrect layout
> 16 test cases (10%) were new
> 1 test case (<1%) timed out
> 19 test cases (13%) had stderr output
>
> Some can be fixed by having pixel tests support but the results are
> much more encouraging with the webkit.org backends.
>
> --
> Alexis Menard
> Software Engineer
> INdT Recife Brazil
> _______________________________________________
> webkit-qt mailing list
> webkit-qt at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-qt



Yo, M. I-m-a-daddy-now !

Nice job, and good thing you took the time presenting all this. In the light
of recent discussions, for instance the fact that Symbian won't get Qt5
(said Sebastian at QCS), I think using the upstream backends make sense.
Your proposal for windows sounds pretty good too, and that should help
relieving a bit of pressure from the QtMultimedia guys at least for the
early days of Qt5, we can always re-evaluate the situation and decide when
the time comes to make the switch.

Cheers,
--
Piere
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-qt/attachments/20110625/7a93011e/attachment.html>