[webkit-dev] Implementing the <device> element

[Adam Bergkvist]

Hi,

On 2011-02-04 19:21, Leandro Graciá Gil wrote:
> This is good news! Especially for the situations where WebCore can't 
> directly access the hardware. One existing case of this we should keep 
> in mind are the sandboxed environments, where both the probing and the 
> connections must be requested to somewhere outside the sandbox. Usually 
> this will require to communicate with another process, and in this case 
> asynchronous messages are preferred to avoid delays and to make 
> inter-process communication simpler.
> 
> However I have to disagree in one point. The specification doesn't say 
> anywhere that we should always present a dialog, only that the device 
> element represents a device selector.

The UI has been a central part of the discussions that lead up to the
device element proposal. The reason for that is that it should enforce
security.

See the thread "UI for enabling webcam use from untrusted content":
http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0149.html
especially
http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0194.html

You're right that we shouldn't limit ourselves by referring to the device
selector as a dialog. I've renamed ChromeClient::runDeviceDialog to
runDeviceSelector and the corresponding client interface accordingly.

> Consider this completely valid use 
> case: instead of asking repeatedly the user to select a device, a UA 
> might decide to create some kind of internal device settings 
> configuration panel to select a set of default devices. Later when 
> visiting a page and clicking the device element the UA will 
> automatically use the preferred devices from its internal settings if 
> they are available and the page is trusted. Where is the dialog here?
>

Couldn't you just see the internal device settings configuration panel,
you mention, as the device selector that produces a device list that's
reused several times? In that case you would skip the dialog and simply
apply the predefined selections (similar to the case where a "remember
my choices" check box would be available in the device dialog).


> I agree that the device should perform selection, as the spec says. 
> However as I've already explained I don't think we need for example a 
> selection dialog for all use cases. Considering that we don't explicitly 
> need a dialog to perform the selection, the only reason to bring lists 
> of available devices back to WebCore is to send them again to a client, 
> probably the same one we asked for probing. Also, if we consider the 
> possibility of sandboxed environments then the device connection 
> operation cannot be a synchronous operation as commented before.
> 

As mentioned above, I see a point in sending the list of available
devices to WebCore to determine if a new Stream (or other device
handler) should be created since this behavior should be consistent
across platforms, regardless if the device is of type "media" or
"fishtank".

The device "connection operation" is not handled by the device element. The
device element is used to simply select devices (similar to how you select
a file with <input type=file> and get a File object which is just a handle
to the actual file). The "connection" takes place when you use the device,
e.g., when you play a Stream in a video element; and that will happen
asynchronously. 

> Reviewing the design with all these factors leaves us the following scheme:
> - Request device selection asynchronously to the client (not necessarily 
> using a dialog).
> - Retrieve the available device list.
> - Forward the list to a client (probably the same that a moment ago 
> probed the devices) to connect them. Do it asynchronously to keep 
> compatibility with sandboxed environments.
> - Receive the connection request result and some device specific data.
> 
> So, the available device lists are being sent back to the device element 
> not for making any specific use of them, but for forwarding them to a 
> connection client in an asynchronous model.
> 
> Wouldn't it be simpler if we refactor the process in this way?
> - Request device selection asynchronously to a client.
> - Receive the connection request result and some device specific data.
> 
> This is exactly what our model proposes. The same goal can be performed 
> by handling connections to devices instead to actual device lists, 
> especially when we're likely to give back the list to the same client 
> that provided it to us. It also avoids any list handling code outside 
> the clients and to implement an intermediate selection/connection 
> element state.
> 
> To make our proposal clearer, we have uploaded a patch with most of our 
> WebCore implementation. It can be found here: 
> https://bugs.webkit.org/show_bug.cgi?id=53777
> This patch it's not intended to be reviewed (it's too big for that) but 
> to serve as an implementation example of our proposed model. Our 
> original plan was to upload it in small, easy to review pieces. This 
> patch would be intended to be the second of them, after introducing the 
> compilation guards: https://bugs.webkit.org/show_bug.cgi?id=53776
> 
> I have also created a small diagram to explain how our implementation works:
> https://docs.google.com/a/google.com/drawings/edit?id=1jSW-6MJd8mp2qPvwnvZnBVzll6UtBz3r1viZgTE4XVA&hl=en&authkey=CPLpy5oJ 
> <https://docs.google.com/a/google.com/drawings/edit?id=1jSW-6MJd8mp2qPvwnvZnBVzll6UtBz3r1viZgTE4XVA&hl=en&authkey=CPLpy5oJ>
> 
> The basic idea is that we have a device controller that handles all the 
> message forwarding to the platform specific clients in an asynchronous, 
> easy to extend way. Please note that the platform specific clients are 
> not necessarily required to be implemented by WebKit, but they are not 
> designed to be part of the common inter-platform code.
> 
> Our implementation already implements most details of the device element 
> and the stream / stream recorder objects in the specification by making 
> use of simple messages to the clients and their replies. It's also 
> designed to be easily extended to future media types if the 
> specification requires so.
> 
> Please try to analyze and understand our proposed model. I'm doing the 
> same with yours, so we can discuss the details and reach a good design 
> in common that suits all of us.
>

I've looked at your patch in http://webkit.org/b/53777 and have some
initial comments.

Source/WebCore/html/HTMLDeviceElement.cpp
121: return Stream::create(this, m_connection.streamUrl());
"data" is getter property on the device element and shouldn't create a
new Stream on every call.

Source/WebCore/dom/Stream.h
58: RefPtr<HTMLDeviceElement> m_deviceElement;
The Stream is tightly coupled to the device element that created it and
this doesn't work well with a remote Stream created by ConnectionPeer.
Your StreamRecorder is also dependent on the device element.

> What I mean is that we should put only generic code in WebCore so that 
> it can be used by all platforms. If we code in WebCore some details on 
> how the selection or connection should be performed it may be the case 
> that some platforms won't use part of the WebCode code to avoid them. We 
> should also keep in mind that even if now the spec only focuses in media 
> devices, it is quite possible that in the future new types of devices 
> may be added with different contexts, availability or connection 
> requests, or even security requirements. The more generic we stay about 
> selecting and connecting them, the easier it will be later to adapt to 
> any future changes in the specification.
>

Even though every platform will have to implement their own device selector
UI, it is quite likely that several platforms may share the same media
backend and could thus share the same probing and stream handling code.
For example, we've based our implementation on GStreamer which can be used
as a component in the GTK, EFL and WinCairo ports.

BR
Adam