[webkit-qt] API proposal for handling (some) hybrid use cases

[Caio Marcelo de Oliveira Filho]

Hello guys,

As stated in http://labs.qt.nokia.com/2011/08/31/an-oldnew-approach-to-qtwebkit-hybrid/,
because of the way WK2 works, some previous APIs we had are more
difficult to implement than it was in WK1. This email explains an API
proposal to handle some of the cases in the context of WK2.

In our API for WK1 we had the following "hybrid" features available:

1) QWebElement -> manipulating the DOM. Example: adblocker
functionality of a web browser can be implemented using this API.

2) QWebElement::render() -> rendering just a widget using a given
QPainter. Example: scraping just a "piece" of a website.

3) QWebFrame::evaluateJavaScript() -> execute code in the JS
environment of the frame.

4) QWebFrame::addToJavaScriptWindowObject() -> expose a QObject inside
the JS environment of the frame. Example: exposing some application
specific data to the JS environment so it can fill the web page,
exposing an object with slots that can be called to trigger
application specific behavior.

5) QNetworkAccessManager customization -> expose custom protocols to
the web page you are loading. It is a different way to expose content,
both to be loaded by the browser or the JS itself.

6) Exposing QPixmap/QImage through the bridge to be used by the
webpage, for example to "fill" and <img> in the page.

7) Embedding QWidgets inside a web page via QWebPluginFactory.

Note that (1), (2) and (3) are useful even if we are handling a
website that is not controlled by the application, like the "ad
blocker" example.


I propose the following:

P1 - Application communicate with the webpage by exchanging messages,
that are sent as MessageEvents;
P2 - Application can create separate "DOM wrapper worlds" to inject JS
code and communicate with them.

Example of P1, from the QML side the WebViews would have one
additional property "enableMessage", one additional signal
"messageReceived" and one additional slot "postMessage".

WebView { // Choose your favorite webview :P
    id: webView

    enableMessage: true // would be false by default.

    onMessageReceived: {
        console.log(message);
    }

    Button {
        onClicked: {
            webView.postMessage("some data");
        }
    }
}


Still for P1, the code in the webpage would look like:

    <script>
      function initialize() {
          //// RECEIVING MESSAGES from the QML to the WebPage
          window.addEventListener('message', function(e) {
              document.documentElement.innerHTML += '<br>Message2 from
UIProcess: ' + e.data;
          }, false);
      }

      function sendToUIProcess() {
          var input = document.getElementById("message");

          //// POSTING MESSAGE from WebPage to QML
          window.postMessageToQt(input.value);
      }
    </script>


This is the same kind of mechanism used in HTML for communicating
between frames. MessageEvent is described in the HTML5 spec, so we can
even "reuse some wheels".

For P2, the proposal is to expose another property in WebView(s)
called scripts, that is a list of WebScript objects with the signal
"messageReceived" and the slot "postMessage" each one. WebScript also
has a source, which is the javascript code to be loaded by the
webprocess. WebScripts would be loaded in isolated worlds, so they
have their own copy of the DOM Wrappers, and can't interact with page
existing JS code.

WebView {

    scripts: [
        WebScripts {
            id: myScript
            source: "script.js"
            signal done(string result)

            onMessageReceived: {
                myScript.done(message)
            }

            function doSomethingCool() {
                postMessage("doSomethingCool");
            }
        }
    ]

    Button {
        onClicked: myScript.doSomethingCool()
    }
}

// The script.js will add its listener and call postMessage in an
object exposed by us, for example a toplevel Qt object.


If you are familiar with Chrome extensions, this is the same idea of
content scripts. Running them in isolated "environment" prevents code
in webpages to interact with the scripts.

The idea of exchanging messages explicitly is also present on Noam's
proposal. The differences are: the channel to sent the messages will
be WK2 existing IPC instead of HTTP, and that it also want to cover
communication with new created environments (the P2).


I think this API would serve as a "functional" replacement of the
features: (1), (3) and (4). Of course will demand some changes, but I
think overall will reduce our API surface: we don't expose a
QWebElement anymore, users are encouraged to write DOM manipulation
code in JS and use WebScript {}.

Another upside is that future (and current) WK2 browsers using WebKit
can use P2 as a basis for extension mechanism. This will be really
useful for

One downside is that QObject bridge (4) was _very_ convenient API,
however I argue that one can build upon the proposed API to have
convenince wrapping of QObjects -- this could be even adopted by
QtWebKit later.

This proposal doesn't cover:

2) QWebElement::render(), in QtCS it was discussed the possibility of
having a QML component "WebSlice", with a source property pointing to
a WebView and a element property describing which HTML element to
"slice" (in a syntax similar to QWebElement/jQuery).

5) QNAM: Some discussion about special scheme handling was posted by
Simon. The team here in INdT office also had some discussions about
the possibilities of customizing the WebProcess.

6) QPixmap/QImage -> <img>. This is very bridge dependent.

7) QWidgets inside a page: do we have users for this?


Even not covering those other cases, I think this API stands on its
own and is useful. In summary:

- exchange messages with the webpage JS environment
- and with isolated environments that we can create and plug in the page
- small API surface
- people manipulate the DOM the same way they do in a webpage

What do you guys think?


PS: I have a very crude and hackish (but working!) implementation of
P1 in https://gist.github.com/0993e40bda5ceae37122, and I'm learning
my ways to get P2 done. The idea here is to get some feedback on the
API proposal and move those things to a master bug in Bugzilla. So, I
would prefer if we focus the discussion on the API proposed instead of
the implementation details of the patch. :-)

-- 
Caio Marcelo de Oliveira Filho
OpenBossa - INdT