[webkit-qt] QtWebKit, the multimedia support today and tomorrow.

[Alexis Menard]

Hello,

DISCLAIMER : It may be controversial for some people.

For many months, the QtWebKit team has worked on the multimedia
support coming with HTML5. We had a first Phonon backend, abandoned
and really far from feature completion and far from working well. When
QtMultimedia appeared a new backend was added to the tree providing a
more extensive API than Phonon and supporting more platforms
(Symbian). This is the default today if you have Qt Mobility
installed. Unfortunately Qt Multimedia really lacks of QA and
stability, the quality of the backends is really heterogenous and
backends behaves differently. I tried hard to communicate with the Qt
Multimedia team, they helped a lot but even today Qt Multimedia is not
good enough and I feel the support is built on unstable foundations.

On the other hand last year Simon and me investigated on the
multimedia support in WebKit2, he took over when I moved to INdT but
the result can be seen on the N9 Browser. This implementation is not
using Qt Multimedia but instead a modified version of the GStreamer
backend sitting in webkit.org and maintained + used mainly by the GTK
port. The lesson learned from this implementation is that it was hard
for us (we don't have particular knowledge of GStreamer) to go into
the code and hacked it the way we wanted it. But with time we did it
and at the end we are providing a really good and tailored
implementation, by supporting texture streaming for example, things we
couldn't have done in Qt Multimedia easily because we were tight to
the API and their release schedule. While I was in INdT, I worked on
the GStreamer backend upstream to make sure it was working well on
regular linux desktop. It does now and has the same features as
QtMultimedia. You can enable it by passing DEFINES+=USE_GSTREAMER=1 to
your build-webkit script. At the end the code is fairly small and
doesn't have the same complexity as Qt Multimedia GStreamer backend,
at the end we just playback sounds and videos :D.

After this successful effort I investigated other platforms like Mac.
Again I looked at what Apple is shipping and I used it. If you pass
DEFINES+=USE_QTKIT=1 when building QtWebKit on Mac you can enable the
media support using the QuickTime backend the mac port is using. The
implementation is working fairly well too and actually support more
stuff than QtMultimedia does : the fullscreen support is working
better (animated, Tor-Arne will love it) and has playback controls in
fullscreen using the existing native theming. I found it pretty easy
to use and It didn't require that much changes in Apple's code to make
it work.

So the advantages are :

- We have pretty well working implementation on Mac and Linux (and N9
by extension).
- We supports more features than QtMultimedia does.
- We can hack the webkit.org backend to shape them the way we want.
- The backends are pretty well tested and covered by layout tests.
- We got improvements for free.
- It makes us closer to other ports by talking to them (both Apple and
GTK teams were *very* nice to help me to implement everything).
- We have a very low dependency on Qt4, both GStreamer and QTKIT
backend are not using much (QImage, a top level QWidget from what I
remember) stuff that will anyway be part of Qt5 (you need to be able
to render into something, and you need a top level window).
- We don't need to wait QtMultimedia to work on Qt5 to support media
elements on QtWebKit next gen.
- We have hard time to make the media layout tests to run reliably on
the Qt Linux bot, using the GStreamer backend of webkit make us almost
the same as with GTK (so we can have almost the same configuration).

Now the drawback of using the backends of webkit.org :

- Makes harder to maintain : we have multiple backends and code path
(but in the other hand we are not alone to maintain the backends).
- We don't have today Windows support, WebKit don't have native
Windows backend (unless we use the QuickTime backend on Windows if
QuickTime is installed). I would like to implement one based on MS
APIs.
- Symbian is missing too. Not sure it is relevant for the WebKit2
activities as well as Qt5.
- We don't have Mac bots on the build.webkit.org, making hard for
Apple to check that they didn't break Qt if we use their backend.

At the end I still think that using the upstream backend is more
beneficial and some of the disadvantages can be solved by having a Qt
Mac Bot on build.webkit.org and implementing the missing backend for
example.

So here is my proposal (for trunk) :

- Removing Phonon backend from the tree. I asked on
http://lists.kde.org/?l=kde-multimedia&m=130779689626212&w=2 and got
no answer, which to me means nobody wants to take care of it. It
brings confusion in the webkit community.
- Make GStreamer and QuickTime webkit.org backend the default on Mac
and Linux. I could investigate of using the feature of WebKit to have
multiple player so that if the webkit.org player can't work it could
fallback on QtMultimedia.
- Use QtMultimedia on Windows and Symbian (up until Symbian disappear
and I give a try to the MS backend). In the meantime again I could
investigate on Windows to use the QuickTime backend by default and
fallback to QtMultimedia if QuickTime is not installed.

I think this makes us ready for working on a WebKit2/Qt5, helps Nokia
but keep the desktop world functional too.

Any thoughts? feedback? Please please answer :D.

PS (running all the media layout tests on my mac) :

QTKIT :
117 test cases (80%) succeeded
13 test cases (8%) had incorrect layout
16 test cases (10%) were new
7 test cases (4%) had stderr output

QTMULTIMEDIA :
79 test cases (54%) succeeded
51 test cases (34%) had incorrect layout
16 test cases (10%) were new
29 test cases (19%) had stderr output

On Linux :

GSTREAMER :
109 test cases (74%) succeeded
19 test cases (13%) had incorrect layout
16 test cases (10%) were new
2 test cases (1%) timed out
9 test cases (6%) had stderr output

QTMULTIMEDIA :
88 test cases (60%) succeeded
41 test cases (28%) had incorrect layout
16 test cases (10%) were new
1 test case (<1%) timed out
19 test cases (13%) had stderr output

Some can be fixed by having pixel tests support but the results are
much more encouraging with the webkit.org backends.

-- 
Alexis Menard
Software Engineer
INdT Recife Brazil