<div dir="ltr">The two complaints I heard against testharness.js are:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg">- They are less easy to debug as test harness.js does not print out messages for each assert. There might be room for updating testharness to support that<br class="gmail_msg">- Async tests are less easy to write. While this is probably true, testharness has support for promise-based tests which are not verbose.</div><div class="gmail_msg">WPT has also some benefits of its own, like the possibility to run easily the same tests in worker/window environments, the flexible python-based server...</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">One complaint I heard against WPT tests is that they require being run behind a server.</div><div class="gmail_msg">This is becoming more and more true. </div><div class="gmail_msg">There is still a large portion of tests that could be run locally if we update the paths to test harness.js/testharnessreport,js.</div><div class="gmail_msg">I am not a big fan of modify any WPT test but maybe we should consider switching back to modifying these paths at import and export time.</div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">We should probably do this for the sake of making tests more debuggable. Having to run a HTTP/HTTPS server makes testing WebCore against a test needlessly harder especially for things like core DOM API which doesn't require any network stack.</div></div></div></div></blockquote><div><br></div><div>Or we could make our tooling better to also handle LayoutTests/http tests more easily.</div><div>In any case, our CI infrastructure should run WPT tests as closely as specified by WPT.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg">I would tend to do the following:</div><div class="gmail_msg">- Write/submit WPT tests in WebKit as regular WebKit testharness.js layout tests through bugzilla</div><div class="gmail_msg">- At commit queue time, automatically create a PR to WPT GitHub containing the changes of the WebKit patch<br class="gmail_msg"></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">I think commit queue time is too late. Remember that not everyone uses commit queue. Also, if there was any issue with a test, we should be able to tell that prior to landing the patch.</div></div></div></div></blockquote><div><br></div><div>Ideally, having one EWS bullet/bot capturing/handling PR status would be very good.</div><div>Knowing the state of a test from various browsers would be helpful at review time.</div><div><br></div><div>But this might require some extra work to support PR updates and so on.</div><div>CQ time seems like a reasonable target for step 1.</div><div>Well, step 0 might be to have a script that committers would run themselves locally.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"></div><div class="gmail_msg">- Ask committer to fix the WPT PR should there be any issue with it (committer being informed of such issues through bugzilla).<br class="gmail_msg"></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">I think this is too much of an overhead / a burden for WebKit contributors. Now patch contributors need to worry about EWS, <a href="http://build.webkit.org" class="gmail_msg" target="_blank">build.webkit.org</a>, and some random PR that gets created on Github failing.</div></div></div></div></blockquote><div> </div><div>Contributors would just need to worry about monitoring bugzilla, like they are doing for EWS/CQ.</div><div>The number of conflicts should be fairly small hopefully.</div><div>I am not sure that we can come up with a solution that would handle conflicts automatically.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">The fact people have to worry about EWS & <a href="http://build.webkit.org" class="gmail_msg" target="_blank">build.webkit.org</a> separately is bad enough. We shouldn't be introducing yet another place people have to monitor / care about.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">- R. Niwa</div><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div>
</blockquote></div></div>