WebKit support in web-platform-tests
Hi, the web-platform-tests repository includes tooling that enables running those tests against a supported browser product. I'd like to propose adding generic WebKit support there. Current changes only assume usage of the WebDriver protocol, and the WebDriver binary accepting the --port flag. Selenium executors are used for test harness and reftests. Same WebDriver implementation can also be tested against the WebDriver tests included in the web-platform-tests directory, presuming the tests are enabled or explicitly specified. Only port-specific bit is the specification of capabilities that are passed to the WebDriver binary, idea being that these capabilities are the same as those supported by the WebDriver implementation. GTK is for now the only port that's supported, and it's leveraging the WebDriver implementation under Source/WebDriver/ in WebKit. WPE will be doing the same. Safari I suppose could use its own WebDriver implementation, or perhaps even a separate product. Here's the current set of changes: https://github.com/zdobersek/web-platform-tests/commit/c2ee920876ca6df7c4739... The web-platform-tests suite can then be run like this for the GTK port, assuming a tip-of-trunk build: $ /work/web-platform-tests/wpt run --webkit-port=gtk \ --webdriver-binary=WebKitBuild/Release/bin/WebKitWebDriver \ --binary=WebKitBuild/Release/bin/MiniBrowser \ --binary-arg=--automation \ --binary-arg=--javascript-can-open-windows-automatically=true \ --binary-arg=--enable-xss-auditor=false \ webkit /2dcontext This can be further wrapped into a python script and run as part of the continuous integration system. These changes add a run-web-platform-tests script that invokes the web-platform-tests runner tool, also allowing each port to specify what tests to enable and what the expected failures are: https://github.com/Igalia/webkit/commit/df1aeeb9476c6dd220067f4fc3c6ad69a8f9... Only a small subset of tests is enabled there, for prototype purposes. The expected results system could also be improved to avoid each expected failure having to be marked as such in separate .ini files. But foremost, I'd like to have a consensus of sorts about how various WebKit ports should be handled in the web-platform-tests repository, so that the changes there can proceed -- whether it's fine to implement a generic WebKit product, or whether Safari would like to be treated as a separate browser[1], etc. Regards, Zan [1] There's for instance this from a year ago (though not sure about its functionality): https://github.com/w3c/web-platform-tests/tree/wptrunner-safari
Hi Zan!
On Feb 12, 2018, at 7:06 AM, zan@falconsigh.net wrote:
Hi,
the web-platform-tests repository includes tooling that enables running those tests against a supported browser product. I'd like to propose adding generic WebKit support there.
Current changes only assume usage of the WebDriver protocol, and the WebDriver binary accepting the --port flag. Selenium executors are used for test harness and reftests. Same WebDriver implementation can also be tested against the WebDriver tests included in the web-platform-tests directory, presuming the tests are enabled or explicitly specified.
Only port-specific bit is the specification of capabilities that are passed to the WebDriver binary, idea being that these capabilities are the same as those supported by the WebDriver implementation.
Yep, these capabilities already differ between safaridriver and webkit[gtk]driver.
GTK is for now the only port that's supported, and it's leveraging the WebDriver implementation under Source/WebDriver/ in WebKit. WPE will be doing the same. Safari I suppose could use its own WebDriver implementation, or perhaps even a separate product.
Whether it makes sense depends who is running the tests. If it's for public WPT CI, then it won't have access to trunk safaridriver bits, so it won't be very useful to detect new passes or failures. If it's me or another Apple engineer using it to find conformance problems, then it can run against trunk safaridriver and be quite useful. It might also be useful to run Safari Technology Preview safaridriver in public CI like wpt.fyi just to have an independent pass/fail metric that is a lot more current (every ~2 weeks) than /usr/bin/safaridriver (only updates when system Safari is updated). Currently, safaridriver implements Selenium protocol, so this will be usable for running WPT tests in general (modulo bugs). However, the WebDriver tests in web-platform-tests repository only target W3C protocol and use a custom client library (not Selenium). I'm currently implementing W3C protocol; when it's ready for public testing via Safari Technology Preview, then I will make sure it's possible to target a 'safari' browser via wptrunner for WebDriver web-platform-tests. Until then it doesn't make much sense to add it upstream as a target for WebDriver tests.
Here's the current set of changes: https://github.com/zdobersek/web-platform-tests/commit/c2ee920876ca6df7c4739...
The web-platform-tests suite can then be run like this for the GTK port, assuming a tip-of-trunk build:
$ /work/web-platform-tests/wpt run --webkit-port=gtk \ --webdriver-binary=WebKitBuild/Release/bin/WebKitWebDriver \ --binary=WebKitBuild/Release/bin/MiniBrowser \ --binary-arg=--automation \ --binary-arg=--javascript-can-open-windows-automatically=true \ --binary-arg=--enable-xss-auditor=false \ webkit /2dcontext
This can be further wrapped into a python script and run as part of the continuous integration system. These changes add a run-web-platform-tests script that invokes the web-platform-tests runner tool, also allowing each port to specify what tests to enable and what the expected failures are: https://github.com/Igalia/webkit/commit/df1aeeb9476c6dd220067f4fc3c6ad69a8f9...
Only a small subset of tests is enabled there, for prototype purposes. The expected results system could also be improved to avoid each expected failure having to be marked as such in separate .ini files.
Carlos has a JSON-based expectations system for WPT WebDriver tests. Should we use that? I'm not sure if it is flexible enough to capture multiple port expectations. We'd probably want expectations to vary depending on enabled features somehow. Also, I'm not sure if we want to combine multiple port expectations, as this is a maintenance nightmare in the current layout test TestExpectation files.
But foremost, I'd like to have a consensus of sorts about how various WebKit ports should be handled in the web-platform-tests repository, so that the changes there can proceed -- whether it's fine to implement a generic WebKit product, or whether Safari would like to be treated as a separate browser[1], etc.
For WebDriver testing purposes, I would prefer that there be a generic WebKit driver (uses Source/WebDriver/ and MiniBrowser) and a Safari driver (uses safaridriver and Safari). They are two separate REST API implementations with different bugs and capabilities. It might be worth the effort to make Source/WebDriver work with MiniBrowser.app on Mac just for the purpose of having the entire system under test be compiled from one repository. However, this is generally not reflective of how safaridriver will perform since only the WebKit parts of WebDriver are shared between safaridriver and webkit[gtk]driver. It would just make it easier to run web-platform-tests against WebKit itself on Mac. I don't have an opinion about how to handle webkit vs safari for the rest of web-platform-tests. Again, we need to think about who will be running it that way (engineers? CI?). There is also the issue of WebKit on iOS... since safaridriver doesn't work with iOS Safari, we can't run web-platform-tests via WebDriver on iOS until such support exists.
Regards, Zan
[1] There's for instance this from a year ago (though not sure about its functionality): https://github.com/w3c/web-platform-tests/tree/wptrunner-safari <https://github.com/w3c/web-platform-tests/tree/wptrunner-safari>
I can clean up and submit a new PR for this, if that's the direction folks want to go now.
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
On Mon, Feb 12, 2018, at 6:26 PM, Brian Burg wrote:
Hi Zan!
On Feb 12, 2018, at 7:06 AM, zan@falconsigh.net wrote:
Hi,
the web-platform-tests repository includes tooling that enables running those tests against a supported browser product. I'd like to propose adding generic WebKit support there.
Current changes only assume usage of the WebDriver protocol, and the WebDriver binary accepting the --port flag. Selenium executors are used for test harness and reftests. Same WebDriver implementation can also be tested against the WebDriver tests included in the web-platform-tests directory, presuming the tests are enabled or explicitly specified.
Only port-specific bit is the specification of capabilities that are passed to the WebDriver binary, idea being that these capabilities are the same as those supported by the WebDriver implementation.
Yep, these capabilities already differ between safaridriver and webkit[gtk]driver.
GTK is for now the only port that's supported, and it's leveraging the WebDriver implementation under Source/WebDriver/ in WebKit. WPE will be doing the same. Safari I suppose could use its own WebDriver implementation, or perhaps even a separate product.
Whether it makes sense depends who is running the tests. If it's for public WPT CI, then it won't have access to trunk safaridriver bits, so it won't be very useful to detect new passes or failures. If it's me or another Apple engineer using it to find conformance problems, then it can run against trunk safaridriver and be quite useful. It might also be useful to run Safari Technology Preview safaridriver in public CI like wpt.fyi just to have an independent pass/fail metric that is a lot more current (every ~2 weeks) than /usr/bin/safaridriver (only updates when system Safari is updated).
Currently, safaridriver implements Selenium protocol, so this will be usable for running WPT tests in general (modulo bugs). However, the WebDriver tests in web-platform-tests repository only target W3C protocol and use a custom client library (not Selenium). I'm currently implementing W3C protocol; when it's ready for public testing via Safari Technology Preview, then I will make sure it's possible to target a 'safari' browser via wptrunner for WebDriver web-platform-tests. Until then it doesn't make much sense to add it upstream as a target for WebDriver tests.
Here's the current set of changes: https://github.com/zdobersek/web-platform-tests/commit/c2ee920876ca6df7c4739...
The web-platform-tests suite can then be run like this for the GTK port, assuming a tip-of-trunk build:
$ /work/web-platform-tests/wpt run --webkit-port=gtk \ --webdriver-binary=WebKitBuild/Release/bin/WebKitWebDriver \ --binary=WebKitBuild/Release/bin/MiniBrowser \ --binary-arg=--automation \ --binary-arg=--javascript-can-open-windows-automatically=true \ --binary-arg=--enable-xss-auditor=false \ webkit /2dcontext
This can be further wrapped into a python script and run as part of the continuous integration system. These changes add a run-web-platform-tests script that invokes the web-platform-tests runner tool, also allowing each port to specify what tests to enable and what the expected failures are: https://github.com/Igalia/webkit/commit/df1aeeb9476c6dd220067f4fc3c6ad69a8f9...
Only a small subset of tests is enabled there, for prototype purposes. The expected results system could also be improved to avoid each expected failure having to be marked as such in separate .ini files.
Carlos has a JSON-based expectations system for WPT WebDriver tests. Should we use that? I'm not sure if it is flexible enough to capture multiple port expectations. We'd probably want expectations to vary depending on enabled features somehow. Also, I'm not sure if we want to combine multiple port expectations, as this is a maintenance nightmare in the current layout test TestExpectation files.
I think it's likely that cross-port differences will persist, either in implementation detail or merely in what features each port enables and wants to have tested for conformance. For that purpose, I'd prefer to keep using port-specific test manifest and expectation files, allowing each port to specify what tests to run, and what the expected outcome is. Maintaining all the .ini files doesn't make sense -- I'd rather replicate the JSON approach, specifying expected subtest failures in a test, but then have the Python script generate the metadata source tree under the build directory, and point wptrunner towards that.
But foremost, I'd like to have a consensus of sorts about how various WebKit ports should be handled in the web-platform-tests repository, so that the changes there can proceed -- whether it's fine to implement a generic WebKit product, or whether Safari would like to be treated as a separate browser[1], etc.
For WebDriver testing purposes, I would prefer that there be a generic WebKit driver (uses Source/WebDriver/ and MiniBrowser) and a Safari driver (uses safaridriver and Safari). They are two separate REST API implementations with different bugs and capabilities.
It might be worth the effort to make Source/WebDriver work with MiniBrowser.app on Mac just for the purpose of having the entire system under test be compiled from one repository. However, this is generally not reflective of how safaridriver will perform since only the WebKit parts of WebDriver are shared between safaridriver and webkit[gtk]driver. It would just make it easier to run web-platform-tests against WebKit itself on Mac.
I don't have an opinion about how to handle webkit vs safari for the rest of web-platform-tests. Again, we need to think about who will be running it that way (engineers? CI?).
I think it would be useful for CI too, though not immediately and maybe not at the same priority as layout tests. I don't think it should run on EWS machines, but it would be useful to separately track progress or any regressions, and have that output accessible publicly on build.webkit.org (but in unofficial capacity compared to e.g. wpt.fyi).
There is also the issue of WebKit on iOS... since safaridriver doesn't work with iOS Safari, we can't run web-platform-tests via WebDriver on iOS until such support exists.
Regards, Zan
[1] There's for instance this from a year ago (though not sure about its functionality): https://github.com/w3c/web-platform-tests/tree/wptrunner-safari
I can clean up and submit a new PR for this, if that's the direction folks want to go now.
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Hi Zan, I like the idea of using WebDriver for WPT conformance testing. Such results will probably be more meaningful for conformance than what WTR or DRT could produce. For WPT regression testing, we would stick to using WTR/DRT and internals methods instead of WebDriver, am I right? y Le lun. 12 févr. 2018 à 07:08, <zan@falconsigh.net> a écrit :
Hi,
the web-platform-tests repository includes tooling that enables running those tests against a supported browser product. I'd like to propose adding generic WebKit support there.
Current changes only assume usage of the WebDriver protocol, and the WebDriver binary accepting the --port flag. Selenium executors are used for test harness and reftests. Same WebDriver implementation can also be tested against the WebDriver tests included in the web-platform-tests directory, presuming the tests are enabled or explicitly specified.
Only port-specific bit is the specification of capabilities that are passed to the WebDriver binary, idea being that these capabilities are the same as those supported by the WebDriver implementation.
GTK is for now the only port that's supported, and it's leveraging the WebDriver implementation under Source/WebDriver/ in WebKit. WPE will be doing the same. Safari I suppose could use its own WebDriver implementation, or perhaps even a separate product.
Here's the current set of changes:
https://github.com/zdobersek/web-platform-tests/commit/c2ee920876ca6df7c4739...
The web-platform-tests suite can then be run like this for the GTK port, assuming a tip-of-trunk build:
$ /work/web-platform-tests/wpt run --webkit-port=gtk \ --webdriver-binary=WebKitBuild/Release/bin/WebKitWebDriver \ --binary=WebKitBuild/Release/bin/MiniBrowser \ --binary-arg=--automation \ --binary-arg=--javascript-can-open-windows-automatically=true \ --binary-arg=--enable-xss-auditor=false \ webkit /2dcontext
This can be further wrapped into a python script and run as part of the continuous integration system. These changes add a run-web-platform-tests script that invokes the web-platform-tests runner tool, also allowing each port to specify what tests to enable and what the expected failures are:
https://github.com/Igalia/webkit/commit/df1aeeb9476c6dd220067f4fc3c6ad69a8f9...
Only a small subset of tests is enabled there, for prototype purposes. The expected results system could also be improved to avoid each expected failure having to be marked as such in separate .ini files.
But foremost, I'd like to have a consensus of sorts about how various WebKit ports should be handled in the web-platform-tests repository, so that the changes there can proceed -- whether it's fine to implement a generic WebKit product, or whether Safari would like to be treated as a separate browser[1], etc.
Regards, Zan
[1] There's for instance this from a year ago (though not sure about its functionality): https://github.com/w3c/web-platform-tests/tree/wptrunner-safari _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
On Mon, Feb 12, 2018, at 6:43 PM, youenn fablet wrote:
Hi Zan,
I like the idea of using WebDriver for WPT conformance testing. Such results will probably be more meaningful for conformance than what WTR or DRT could produce. For WPT regression testing, we would stick to using WTR/DRT and internals methods instead of WebDriver, am I right? y
Correct, this isn't meant as a replacement for WPT regression testing, but at some point in the future WebKit developers might agree that it can properly replace it. But as it stands, I'm only selling it as a conformance testing process that produces information that (in case of current failures or future regressions) nudges WebKit towards conformance compliance.
Le lun. 12 févr. 2018 à 07:08, <zan@falconsigh.net> a écrit :
Hi,
the web-platform-tests repository includes tooling that enables running those tests against a supported browser product. I'd like to propose adding generic WebKit support there.
Current changes only assume usage of the WebDriver protocol, and the WebDriver binary accepting the --port flag. Selenium executors are used for test harness and reftests. Same WebDriver implementation can also be tested against the WebDriver tests included in the web-platform-tests directory, presuming the tests are enabled or explicitly specified.
Only port-specific bit is the specification of capabilities that are passed to the WebDriver binary, idea being that these capabilities are the same as those supported by the WebDriver implementation.
GTK is for now the only port that's supported, and it's leveraging the WebDriver implementation under Source/WebDriver/ in WebKit. WPE will be doing the same. Safari I suppose could use its own WebDriver implementation, or perhaps even a separate product.
Here's the current set of changes: https://github.com/zdobersek/web-platform-tests/commit/c2ee920876ca6df7c4739...
The web-platform-tests suite can then be run like this for the GTK port, assuming a tip-of-trunk build:
$ /work/web-platform-tests/wpt run --webkit-port=gtk \ --webdriver-binary=WebKitBuild/Release/bin/WebKitWebDriver \ --binary=WebKitBuild/Release/bin/MiniBrowser \ --binary-arg=--automation \ --binary-arg=--javascript-can-open-windows-automatically=true \ --binary-arg=--enable-xss-auditor=false \ webkit /2dcontext
This can be further wrapped into a python script and run as part of the continuous integration system. These changes add a run-web-platform-tests script that invokes the web-platform-tests runner tool, also allowing each port to specify what tests to enable and what the expected failures are: https://github.com/Igalia/webkit/commit/df1aeeb9476c6dd220067f4fc3c6ad69a8f9...
Only a small subset of tests is enabled there, for prototype purposes. The expected results system could also be improved to avoid each expected failure having to be marked as such in separate .ini files.
But foremost, I'd like to have a consensus of sorts about how various WebKit ports should be handled in the web-platform-tests repository, so that the changes there can proceed -- whether it's fine to implement a generic WebKit product, or whether Safari would like to be treated as a separate browser[1], etc.
Regards, Zan
[1] There's for instance this from a year ago (though not sure about its functionality): https://github.com/w3c/web-platform-tests/tree/wptrunner-safari _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Hi, in case of no additional comments against this, I'll be opening a pull request to merge the web-platform-tests changes into that repository in the following days. The WebKit changes will follow after some further work. Cheers, Zan On Mon, Feb 12, 2018, at 4:06 PM, zan@falconsigh.net wrote:
Hi,
the web-platform-tests repository includes tooling that enables running those tests against a supported browser product. I'd like to propose adding generic WebKit support there.
Current changes only assume usage of the WebDriver protocol, and the WebDriver binary accepting the --port flag. Selenium executors are used for test harness and reftests. Same WebDriver implementation can also be tested against the WebDriver tests included in the web-platform-tests directory, presuming the tests are enabled or explicitly specified.
Only port-specific bit is the specification of capabilities that are passed to the WebDriver binary, idea being that these capabilities are the same as those supported by the WebDriver implementation.
GTK is for now the only port that's supported, and it's leveraging the WebDriver implementation under Source/WebDriver/ in WebKit. WPE will be doing the same. Safari I suppose could use its own WebDriver implementation, or perhaps even a separate product.
Here's the current set of changes: https://github.com/zdobersek/web-platform-tests/commit/c2ee920876ca6df7c4739...
The web-platform-tests suite can then be run like this for the GTK port, assuming a tip-of-trunk build:
$ /work/web-platform-tests/wpt run --webkit-port=gtk \ --webdriver-binary=WebKitBuild/Release/bin/WebKitWebDriver \ --binary=WebKitBuild/Release/bin/MiniBrowser \ --binary-arg=--automation \ --binary-arg=--javascript-can-open-windows-automatically=true \ --binary-arg=--enable-xss-auditor=false \ webkit /2dcontext
This can be further wrapped into a python script and run as part of the continuous integration system. These changes add a run-web-platform- tests script that invokes the web-platform-tests runner tool, also allowing each port to specify what tests to enable and what the expected failures are: https://github.com/Igalia/webkit/commit/df1aeeb9476c6dd220067f4fc3c6ad69a8f9...
Only a small subset of tests is enabled there, for prototype purposes. The expected results system could also be improved to avoid each expected failure having to be marked as such in separate .ini files.
But foremost, I'd like to have a consensus of sorts about how various WebKit ports should be handled in the web-platform-tests repository, so that the changes there can proceed -- whether it's fine to implement a generic WebKit product, or whether Safari would like to be treated as a separate browser[1], etc.
Regards, Zan
[1] There's for instance this from a year ago (though not sure about its functionality): https://github.com/w3c/web-platform-tests/tree/wptrunner-safari
Thanks for your work on this, Žan! That PR is now up at https://github.com/w3c/web-platform-tests/pull/9666 and in it I linked to https://github.com/w3c/web-platform-tests/pull/8979, which is a PR I updated last week to support Safari, including Technology Review. I wish I'd payed closer attention to webkit-dev to see this thread earlier :) Given that safaridriver and webkitdriver aren't the same piece of code, it makes sense to treat them as separate products in wpt run, and whether some code is shared is wpt-internal concern. The reason I tried to get Safari (TP) running is ultimately for wpt.fyi and, and I don't imagine it'll be useful for WebKit CI. Brian, I'll ask you on the PR about the WebDriver tests specifically. On Fri, Feb 23, 2018 at 7:25 PM <zan@falconsigh.net> wrote:
Hi,
in case of no additional comments against this, I'll be opening a pull request to merge the web-platform-tests changes into that repository in the following days. The WebKit changes will follow after some further work.
Cheers, Zan
On Mon, Feb 12, 2018, at 4:06 PM, zan@falconsigh.net wrote:
Hi,
the web-platform-tests repository includes tooling that enables running those tests against a supported browser product. I'd like to propose adding generic WebKit support there.
Current changes only assume usage of the WebDriver protocol, and the WebDriver binary accepting the --port flag. Selenium executors are used for test harness and reftests. Same WebDriver implementation can also be tested against the WebDriver tests included in the web-platform-tests directory, presuming the tests are enabled or explicitly specified.
Only port-specific bit is the specification of capabilities that are passed to the WebDriver binary, idea being that these capabilities are the same as those supported by the WebDriver implementation.
GTK is for now the only port that's supported, and it's leveraging the WebDriver implementation under Source/WebDriver/ in WebKit. WPE will be doing the same. Safari I suppose could use its own WebDriver implementation, or perhaps even a separate product.
Here's the current set of changes:
https://github.com/zdobersek/web-platform-tests/commit/c2ee920876ca6df7c4739...
The web-platform-tests suite can then be run like this for the GTK port, assuming a tip-of-trunk build:
$ /work/web-platform-tests/wpt run --webkit-port=gtk \ --webdriver-binary=WebKitBuild/Release/bin/WebKitWebDriver \ --binary=WebKitBuild/Release/bin/MiniBrowser \ --binary-arg=--automation \ --binary-arg=--javascript-can-open-windows-automatically=true \ --binary-arg=--enable-xss-auditor=false \ webkit /2dcontext
This can be further wrapped into a python script and run as part of the continuous integration system. These changes add a run-web-platform- tests script that invokes the web-platform-tests runner tool, also allowing each port to specify what tests to enable and what the expected failures are:
https://github.com/Igalia/webkit/commit/df1aeeb9476c6dd220067f4fc3c6ad69a8f9...
Only a small subset of tests is enabled there, for prototype purposes. The expected results system could also be improved to avoid each expected failure having to be marked as such in separate .ini files.
But foremost, I'd like to have a consensus of sorts about how various WebKit ports should be handled in the web-platform-tests repository, so that the changes there can proceed -- whether it's fine to implement a generic WebKit product, or whether Safari would like to be treated as a separate browser[1], etc.
Regards, Zan
[1] There's for instance this from a year ago (though not sure about its functionality): https://github.com/w3c/web-platform-tests/tree/wptrunner-safari
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
participants (4)
-
Brian Burg
-
Philip Jägenstedt
-
youenn fablet
-
zan＠falconsigh.net