<div dir="ltr">FWIW, in Blink we stopped rewriting the testharness.js paths before switching to wptserve, by instead rewriting those URLs only when running LayoutTests:<div><a href="https://cs.chromium.org/chromium/src/content/shell/renderer/layout_test/blink_test_runner.cc?type=cs&q=content/shell/renderer/layout_test/blink_test_runner.cc&l=221">https://cs.chromium.org/chromium/src/content/shell/renderer/layout_test/blink_test_runner.cc?type=cs&q=content/shell/renderer/layout_test/blink_test_runner.cc&l=221</a></div><div><br></div><div>So, we know that it's possible to run a lot of the tests without wptserve without modifying the tests. However, trying to list all the tests that do or don't need wptserve seems like a lot of work, and I think we're now using wptserve for all tests.<br><br><div class="gmail_quote"><div dir="ltr">On Sun, Feb 5, 2017 at 8:05 PM youenn fablet <<a href="mailto:youennf@gmail.com">youennf@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">I too am a big proponent of us moving more and more towards WPT.<div class="gmail_msg">As part of the streams and fetch API implementation, most of the related functional tests have been made as WPT.</div><div class="gmail_msg">The benefits of having tests being improved and updated largely outweighs the testharness.js initial cost.</div><div class="gmail_msg">Somehow, these tests are becoming part of their related specs.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">The two complaints I heard against testharness.js are:</div><div class="gmail_msg">- They are less easy to debug as test harness.js does not print out messages for each assert. There might be room for updating testharness to support that<br class="gmail_msg">- Async tests are less easy to write. While this is probably true, testharness has support for promise-based tests which are not verbose.</div><div class="gmail_msg">WPT has also some benefits of its own, like the possibility to run easily the same tests in worker/window environments, the flexible python-based server...</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">One complaint I heard against WPT tests is that they require being run behind a server.</div><div class="gmail_msg">This is becoming more and more true. </div><div class="gmail_msg">There is still a large portion of tests that could be run locally if we update the paths to test harness.js/testharnessreport,js.</div><div class="gmail_msg">I am not a big fan of modify any WPT test but maybe we should consider switching back to modifying these paths at import and export time.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I would also like to go in a direction where writing WPT tests is the default for WebKit layout test.</div><div class="gmail_msg">For that to happen, we need an easy way to upstream tests from WebKit to WPT GitHub.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I would tend to do the following:</div><div class="gmail_msg">- Write/submit WPT tests in WebKit as regular WebKit testharness.js layout tests through bugzilla</div><div class="gmail_msg">- At commit queue time, automatically create a PR to WPT GitHub containing the changes of the WebKit patch<br class="gmail_msg"></div><div class="gmail_msg">- Ask committer to fix the WPT PR should there be any issue with it (committer being informed of such issues through bugzilla).<br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">Le dim. 5 févr. 2017 à 15:42, Ryosuke Niwa <<a href="mailto:rniwa@webkit.org" class="gmail_msg" target="_blank">rniwa@webkit.org</a>> a écrit :<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Hi all,<br class="gmail_msg"><br class="gmail_msg"><b class="gmail_msg">Background</b><br class="gmail_msg"><br class="gmail_msg">Web Platform Tests is a suite of tests written for <a href="https://github.com/w3c/web-platform-tests/" class="gmail_msg" target="_blank">W3C specifications</a>=,<br class="gmail_msg">and Blink, Edge, Gecko, and WebKit all run them as a part of their continuous build system.<br class="gmail_msg"><br class="gmail_msg">Historically, each working group had their own repository for tests,<br class="gmail_msg">but with <a href="https://github.com/w3c/csswg-test" class="gmail_msg" target="_blank">CSS WG's tests</a> finally getting merged into the Web Platform Tests,<br class="gmail_msg">we will have a single repository with all the tests from W3C.<br class="gmail_msg"><br class="gmail_msg">A few of us attended a <a href="https://docs.google.com/document/d/1lqOBTvaoTUIFuxK0IQL1UfXtybq7Kw8JxBvD6FZSdYE/edit#heading=h.aha7njnu40vz" class="gmail_msg" target="_blank">meeting to discuss the future of this test suite</a> last Monday.<div class="gmail_msg">One of the major topic was that Blink and Gecko have adopted the process to write the tests</div><div class="gmail_msg">using testharness.js in their own test suites, and automatically merge into Web Platform Tests</div><div class="gmail_msg">without having to go through another round of code review for Web Platform Tests.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Given we do test-driven development in WebKit, I think WebKit should do the same.</div><div class="gmail_msg">This will benefit other browser vendors to catch their bugs with our tests, and we will also benefit</div><div class="gmail_msg">from other browser vendors "reviewing" our tests against their implementation and specifications.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">In the long term, I believe this will drastically improve the interoperability of the Web,<br class="gmail_msg"></div><div class="gmail_msg">and dramatically improve the quality of the tests we run against WebKit.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">In fact, there was a general consensus that we should even upstream pass-if-not-crash tests</div><div class="gmail_msg">as well as style and layout invalidation tests to web-platform-tests as they tend to be undertested</div><div class="gmail_msg">right now and almost all engines have bugs in this area.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><b class="gmail_msg">Problems & Questions</b></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">In order to auto-merge tests from WebKit to Web Platform Tests, there are a few problems.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><b class="gmail_msg">1. We need to start writing more if not all tests using testharness.js instead of our own js-test(-pre).js</b></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I've heard of many complaints about testharness.js being too verbose and cumbersome to use.</div><div class="gmail_msg">I agree with all those complaints but I don't think we can convince all other browser vendors</div><div class="gmail_msg">to use our own testing script either since both Blink & Gecko are onboard with using testharness.js</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">At this point, I'd argue that the benefit outweighs the cost of adopting testharness.js.</div><div class="gmail_msg">Also, W3C have dropped many styling requirements for tests so we no longer need to have</div><div class="gmail_msg">link/meta elements, etc... You simply include testharness.js and testharness-report.js.<br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><a href="https://github.com/w3c/web-platform-tests/commit/c4725e56c91954eaf51759feabec8c6560f10ade" class="gmail_msg" target="_blank">See my recent PR</a> for example.<br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Do people still object to using testharness.js? If so, what are your reasons?</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><b class="gmail_msg">2. Where should we put upstremable tests?</b></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">We need a mechanism to identify & merge tests back from WebKit into Web Platform Tests.<br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">One way to do this is to start adding tests directly into LayoutTests/imported/w3c/web-platform-tests</div><div class="gmail_msg">and then automatically identify those tests and upstream them.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Assuming all upstream merges happen in timely happen, this should work fine.</div><div class="gmail_msg">And it has a benefit that you get to see all related tests in one place.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Another approach is to create another top-level directory like LayoutTests/exports/web-platform-tests.</div><div class="gmail_msg">One downside of this approach is that we need to match the directory structure</div><div class="gmail_msg">in order for any script to upstream tests to appropriate locations.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Do people prefer one approach over another? Or have another idea?</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><b class="gmail_msg">3. What would be the process for upstreaming tests?</b></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">One possible approach is to create a PR request for every patch posted on Bugzilla</div><div class="gmail_msg">with tests that can be merged into Web Platform Tests.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><div class="gmail_msg">Because we don't want to make Web Platform Tests less reliably,</div><div class="gmail_msg">every PR request to Web Platform Tests go through <a href="https://travis-ci.org/" class="gmail_msg" target="_blank">Travis CI</a></div><div class="gmail_msg">which runs the test hundreds of times to make sure it's not flaky.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">This Travis CI job (stability check) will then show up as an additional EWS bubble.</div></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Another approach is to wait until a patch is landed, and automatically create a PR</div><div class="gmail_msg">for tests that can be upstream'ed. This has a downside that the stability check</div><div class="gmail_msg">wouldn't run until the patch is landed, and it's unclear what happens they do fail.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Do committers get emails about them? Do the patches then need to be rolled back?</div><div class="gmail_msg">If we don't come up with an appropriate process, we can end up with a lot of</div><div class="gmail_msg">broken PRs that never get fixed like those failing test expectations :(</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Again, do you prefer one or the other? Or do you have another proposal?</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">- R. Niwa<br class="gmail_msg"><br class="gmail_msg"></div></div>
_______________________________________________<br class="gmail_msg">
webkit-dev mailing list<br class="gmail_msg">
<a href="mailto:webkit-dev@lists.webkit.org" class="gmail_msg" target="_blank">webkit-dev@lists.webkit.org</a><br class="gmail_msg">
<a href="https://lists.webkit.org/mailman/listinfo/webkit-dev" rel="noreferrer" class="gmail_msg" target="_blank">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br class="gmail_msg">
</blockquote></div></div></div>
_______________________________________________<br class="gmail_msg">
webkit-dev mailing list<br class="gmail_msg">
<a href="mailto:webkit-dev@lists.webkit.org" class="gmail_msg" target="_blank">webkit-dev@lists.webkit.org</a><br class="gmail_msg">
<a href="https://lists.webkit.org/mailman/listinfo/webkit-dev" rel="noreferrer" class="gmail_msg" target="_blank">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br class="gmail_msg">
</blockquote></div></div></div>