<div dir="ltr"><div>It seems we agree in moving this forward. Any objection?</div><div>If so, let's start with small practical steps, something like a script to automatically generate WPT PR requests from a WebKit repo.</div></div><br><div class="gmail_quote"><div dir="ltr">Le lun. 6 févr. 2017 à 12:28, Ryosuke Niwa <<a href="mailto:rniwa@webkit.org">rniwa@webkit.org</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="gmail_msg">On Monday, February 6, 2017, youenn fablet <<a href="mailto:youennf@gmail.com" class="gmail_msg" target="_blank">youennf@gmail.com</a>> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">The two complaints I heard against testharness.js are:<br class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg">- They are less easy to debug as test harness.js does not print out messages for each assert. There might be room for updating testharness to support that<br class="gmail_msg">- Async tests are less easy to write. While this is probably true, testharness has support for promise-based tests which are not verbose.</div><div class="gmail_msg">WPT has also some benefits of its own, like the possibility to run easily the same tests in worker/window environments, the flexible python-based server...</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">One complaint I heard against WPT tests is that they require being run behind a server.</div><div class="gmail_msg">This is becoming more and more true. </div><div class="gmail_msg">There is still a large portion of tests that could be run locally if we update the paths to test harness.js/testharnessreport,js.</div><div class="gmail_msg">I am not a big fan of modify any WPT test but maybe we should consider switching back to modifying these paths at import and export time.</div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">We should probably do this for the sake of making tests more debuggable. Having to run a HTTP/HTTPS server makes testing WebCore against a test needlessly harder especially for things like core DOM API which doesn't require any network stack.</div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Or we could make our tooling better to also handle LayoutTests/http tests more easily.</div><div class="gmail_msg">In any case, our CI infrastructure should run WPT tests as closely as specified by WPT.</div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">The concern I've heard about is not how run-webkit-tests run the tests. It's about how a test is opened inside a browser, DRT, WTR while debugging.</div><div class="gmail_msg"><br class="gmail_msg"></div><br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg">I would tend to do the following:</div><div class="gmail_msg">- Write/submit WPT tests in WebKit as regular WebKit testharness.js layout tests through bugzilla</div><div class="gmail_msg">- At commit queue time, automatically create a PR to WPT GitHub containing the changes of the WebKit patch<br class="gmail_msg"></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">I think commit queue time is too late. Remember that not everyone uses commit queue. Also, if there was any issue with a test, we should be able to tell that prior to landing the patch.</div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Ideally, having one EWS bullet/bot capturing/handling PR status would be very good.</div><div class="gmail_msg">Knowing the state of a test from various browsers would be helpful at review time.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">But this might require some extra work to support PR updates and so on.</div><div class="gmail_msg">CQ time seems like a reasonable target for step 1.</div><div class="gmail_msg">Well, step 0 might be to have a script that committers would run themselves locally.</div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Again, many people don't use CQ. What do those people do? Manually run a script? I don't think that's a workable solution. There are many people just land patches from Xcode UI for example.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">In general, I don't think it's acceptable to introduce any extra step for WebKit contributors. Let it be running a new script, potentially fix an issue in GitHub PR, etc...</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">There should be absolutely no new thing contributor has to run or monitor. That should be the minimum bar for this upstreaming system to exist.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Again, we can't even keep up with test failures on our own bots, and people frequently land patches with broken tests or tests that fail on some bots. There is no way we can succeed by introducing yet another step / place humans has to care. That's simply unacceptable.</div><div class="gmail_msg"><br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"></div><div class="gmail_msg">- Ask committer to fix the WPT PR should there be any issue with it (committer being informed of such issues through bugzilla).<br class="gmail_msg"></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">I think this is too much of an overhead / a burden for WebKit contributors. Now patch contributors need to worry about EWS, <a href="http://build.webkit.org" class="gmail_msg" target="_blank">build.webkit.org</a>, and some random PR that gets created on Github failing.</div></div></div></div></blockquote><div class="gmail_msg"> </div><div class="gmail_msg">Contributors would just need to worry about monitoring bugzilla, like they are doing for EWS/CQ.</div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div>It depends on what kind of "monitoring" is required.<br class="gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">The number of conflicts should be fairly small hopefully.</div><div class="gmail_msg">I am not sure that we can come up with a solution that would handle conflicts automatically.</div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">In the case of a conflict, the patch should be rejected in EWS & CQ just the same way a WebKit patch conflict would.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">- R. Niwa </div><br class="gmail_msg"><br class="gmail_msg">-- <br class="gmail_msg">- R. Niwa<br class="gmail_msg">
</blockquote></div>