[webkit-dev] Implementing the <device> element

[Leandro Graciá Gil]

El 3 de febrero de 2011 18:59, Adam Bergkvist <adam.bergkvist at ericsson.com>
 escribió:

> Hi,
>
> Leandro Graciá Gil wrote:
> > The approach proposed in https://bugs.webkit.org/show_bug.cgi?id=47264
> > brings the device probing and selection to the WebCore level. It does
> > so by first creating a list of available devices using an interface to
> > a platform-dependent client and then it passes this list to a client
> > dialog to perform the actual selection from it. With its current
> > design both operations are performed synchronously, possibly blocking
> > the device element event handler for a long time. I think this is not
> > a suitable design.
>
> The device selector dialog was never intended to block the event loop.
> That was a mistake. In GTK the main loop continues to run when the dialog
> is shown so although the click handler is blocked the event loop is not
> blocked. We should not have expected this behavior on other platforms.
> I have now added a callback from the dialog to the device element when a
> selection has been made, making it possible to implement a completely
> asynchronous dialog.


> The interface towards the platform for getting a list of available
> devices is called createDeviceList; it makes no assumption about how
> the probing is done. The only assumption is that when a device selector
> dialog is about to be presented to the user, a list of available devices
> is needed. Typically, probing involves asking the OS kernel for a list of
> available devices and this operation is usually fast. That's why we in our
> GTK implementation have chosen to simply do the probing synchronously from
> createDeviceList. I have now moved the createDeviceList call to device
> dialog code. This makes it possible for platforms that want to do
> on-demand probing but fear it might block, to call it off the main loop.
>

This is good news! Especially for the situations where WebCore can't
directly access the hardware. One existing case of this we should keep in
mind are the sandboxed environments, where both the probing and the
connections must be requested to somewhere outside the sandbox. Usually this
will require to communicate with another process, and in this case
asynchronous messages are preferred to avoid delays and to make
inter-process communication simpler.

However I have to disagree in one point. The specification doesn't say
anywhere that we should always present a dialog, only that the device
element represents a device selector. Consider this completely valid use
case: instead of asking repeatedly the user to select a device, a UA might
decide to create some kind of internal device settings configuration panel
to select a set of default devices. Later when visiting a page and clicking
the device element the UA will automatically use the preferred devices from
its internal settings if they are available and the page is trusted. Where
is the dialog here?


>
> > Given the drawbacks of the approach above, I would like to propose an
> > alternative design that solves these issues.
> > First of all, we think that is not necessary at all to bring the
> > device probing, available device lists, device selection or connection to
> WebCore.
> > The device element isn't really about actual devices, but about
> > connecting to them. We think that it would be the best for all
> > platforms to actually delegate the actual probing, selection and
> > connecting to them in an asynchronous client-based model and hold only
> > connection and handler information in WebCore.
> >
> The device element is about devices and specifically about selecting
> devices.
> <device> is inspired by <input type=file> and so is our implementation.
> When
> the change event should trigger on the device element, for example, is
> something that will eventually be specced out and so the code that handles
> this logic is best put in WebCore in order to preserve consistency between
> ports.
>

I agree that the device should perform selection, as the spec says. However
as I've already explained I don't think we need for example a selection
dialog for all use cases. Considering that we don't explicitly need a dialog
to perform the selection, the only reason to bring lists of available
devices back to WebCore is to send them again to a client, probably the same
one we asked for probing. Also, if we consider the possibility of sandboxed
environments then the device connection operation cannot be a synchronous
operation as commented before.

Reviewing the design with all these factors leaves us the following scheme:
- Request device selection asynchronously to the client (not necessarily
using a dialog).
- Retrieve the available device list.
- Forward the list to a client (probably the same that a moment ago probed
the devices) to connect them. Do it asynchronously to keep compatibility
with sandboxed environments.
- Receive the connection request result and some device specific data.

So, the available device lists are being sent back to the device element not
for making any specific use of them, but for forwarding them to a connection
client in an asynchronous model.

Wouldn't it be simpler if we refactor the process in this way?
- Request device selection asynchronously to a client.
- Receive the connection request result and some device specific data.

This is exactly what our model proposes. The same goal can be performed by
handling connections to devices instead to actual device lists, especially
when we're likely to give back the list to the same client that provided it
to us. It also avoids any list handling code outside the clients and to
implement an intermediate selection/connection element state.

To make our proposal clearer, we have uploaded a patch with most of our
WebCore implementation. It can be found here:
https://bugs.webkit.org/show_bug.cgi?id=53777
This patch it's not intended to be reviewed (it's too big for that) but to
serve as an implementation example of our proposed model. Our original plan
was to upload it in small, easy to review pieces. This patch would be
intended to be the second of them, after introducing the compilation guards:
https://bugs.webkit.org/show_bug.cgi?id=53776

I have also created a small diagram to explain how our implementation works:
https://docs.google.com/a/google.com/drawings/edit?id=1jSW-6MJd8mp2qPvwnvZnBVzll6UtBz3r1viZgTE4XVA&hl=en&authkey=CPLpy5oJ

The basic idea is that we have a device controller that handles all the
message forwarding to the platform specific clients in an asynchronous, easy
to extend way. Please note that the platform specific clients are not
necessarily required to be implemented by WebKit, but they are not designed
to be part of the common inter-platform code.

Our implementation already implements most details of the device element and
the stream / stream recorder objects in the specification by making use of
simple messages to the clients and their replies. It's also designed to be
easily extended to future media types if the specification requires so.

Please try to analyze and understand our proposed model. I'm doing the same
with yours, so we can discuss the details and reach a good design in common
that suits all of us.


> > 4. The device client receives the request and, in a non-blocking way,
> > implements the way it likes the device probing, selection, connecting
> > and all the required security operations (the latter also missing in
> > the existing patch). This allows every UA vendor to make their own
> > choices about the UI for device selection.
>
> Every vendor should be able to style the device selector as they want, but
> the user experience should be coherent between platforms since <device> is
> in its essence a security feature and user recognition is an important
> aspect
> of that.
>

Of course. Let's decide a common and clear way to render the element so that
it could be easily recognized by the user. I'm open to any suggestions.
However if you talk about a common user experience beyond that point, I'm
afraid that you can't expect that from even the device selection dialogs, if
there is any.



>
> > With this approach, not only we avoid to block WebCore with
> > potentially long operations but we provide a great flexibility to all
> > WebKit platforms to implement the nasty device handling details as
> > they want to. This should also potentially increase the amount of code
> > that can be reused by the different platforms, while avoiding any list
> > matching code that can be found in the existing patch.
> >
>
> Can you please elaborate on how putting more code in the platforms, instead
> of WebCore, will increase code reuse?
>

What I mean is that we should put only generic code in WebCore so that it
can be used by all platforms. If we code in WebCore some details on how the
selection or connection should be performed it may be the case that some
platforms won't use part of the WebCode code to avoid them. We should also
keep in mind that even if now the spec only focuses in media devices, it is
quite possible that in the future new types of devices may be added with
different contexts, availability or connection requests, or even security
requirements. The more generic we stay about selecting and connecting them,
the easier it will be later to adapt to any future changes in the
specification.


>
> We would gladly make any changes that could make it easier for you to
> implement the platform specific parts.
>
> BR
> Adam
>

Same for us. We would be glad to reach a common design and make any changes
to our proposal as required. Please do not hesitate to make any suggestions
to improve our designs. I'm going to try to involve spec-related people in
this discussion so they can help us with their opinion.

Thanks,
Leandro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110204/41eb188a/attachment.html>