[webkit-dev] WebKit support in web-platform-tests

[zan at falconsigh.net]

Hi,

in case of no additional comments against this, I'll be opening a pull request to merge the web-platform-tests changes into that repository in the following days. The WebKit changes will follow after some further work.

Cheers,
Zan

On Mon, Feb 12, 2018, at 4:06 PM, zan at falconsigh.net wrote:
> Hi,
> 
> the web-platform-tests repository includes tooling that enables running 
> those tests against a supported browser product. I'd like to propose 
> adding generic WebKit support there.
> 
> 
> Current changes only assume usage of the WebDriver protocol, and the 
> WebDriver binary accepting the --port flag. Selenium executors are used 
> for test harness and reftests. Same WebDriver implementation can also be 
> tested against the WebDriver tests included in the web-platform-tests 
> directory, presuming the tests are enabled or explicitly specified.
> 
> Only port-specific bit is the specification of capabilities that are 
> passed to the WebDriver binary, idea being that these capabilities are 
> the same as those supported by the WebDriver implementation.
> 
> GTK is for now the only port that's supported, and it's leveraging the 
> WebDriver implementation under Source/WebDriver/ in WebKit. WPE will be 
> doing the same. Safari I suppose could use its own WebDriver 
> implementation, or perhaps even a separate product.
> 
> Here's the current set of changes:
> https://github.com/zdobersek/web-platform-tests/commit/c2ee920876ca6df7c4739feb8a6e03c77dffdb7f
> 
> 
> The web-platform-tests suite can then be run like this for the GTK port, 
> assuming a tip-of-trunk build:
> 
> $ /work/web-platform-tests/wpt run --webkit-port=gtk \
>     --webdriver-binary=WebKitBuild/Release/bin/WebKitWebDriver \
>     --binary=WebKitBuild/Release/bin/MiniBrowser \
>     --binary-arg=--automation \
>     --binary-arg=--javascript-can-open-windows-automatically=true \
>     --binary-arg=--enable-xss-auditor=false \
>     webkit /2dcontext
> 
> This can be further wrapped into a python script and run as part of the 
> continuous integration system. These changes add a run-web-platform-
> tests script that invokes the web-platform-tests runner tool, also 
> allowing each port to specify what tests to enable and what the expected 
> failures are:
> https://github.com/Igalia/webkit/commit/df1aeeb9476c6dd220067f4fc3c6ad69a8f948ba
> 
> Only a small subset of tests is enabled there, for prototype purposes. 
> The expected results system could also be improved to avoid each 
> expected failure having to be marked as such in separate .ini files.
> 
> 
> But foremost, I'd like to have a consensus of sorts about how various 
> WebKit ports should be handled in the web-platform-tests repository, so 
> that the changes there can proceed -- whether it's fine to implement a 
> generic WebKit product, or whether Safari would like to be treated as a 
> separate browser[1], etc.
> 
> 
> Regards,
> Zan
> 
> 
> [1] There's for instance this from a year ago (though not sure about its 
> functionality):
> https://github.com/w3c/web-platform-tests/tree/wptrunner-safari