[webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Maciej Stachowiak]

> On Nov 17, 2017, at 7:53 AM, Frédéric WANG <fred.wang at free.fr> wrote:
> 
> On 17/11/2017 16:26, Maciej Stachowiak wrote:
>> WebKit has a lot of tests that were regression tests for a specific
>> bug fix, not as conformance tests (though they might b useful for that
>> too). In such cases, the association with the bugs.webkit.org bug is
>> more important than the spec URL. That’s particularly the case when
>> the test has been changed multiple times to reflect further WebKit
>> behavior changes. I know that I’ve personally found it very useful to
>> look at revision history of tests, or search for bug numbers or
>> keywords in LayoutTests/ChangeLog to find related tests.
>> Not saying this is the sole purpose of tests, but losing this ability or making it more awkward would be a downside to deleting tests from the WebKit repo.
> Well, I think we agree here, that's why "conformance" tests was
> specified in my message. For such tests, connecting the history of tests
> with the development of the specs is actually more important. For
> example I recently noticed that some bugs with WOFF2 (on Apple's ports
> only) and WebVTT have been ignored in WebKit because we don't
> import/sync WPT tests for these specs.

I think importing new test suites is a different question than upstreaming/removing tests. In general importing more test suites is probably good, but we probably need to tackle the WPT performance problem before we pull in too many new WPT suites.

I should also note that pulling in the test suite won't automatically get the bugs fixed or prioritized, or even filed in the bug tracker necessarily.eratwerATWeRTWAerATWertwAerer

>> Maybe I’m missing something, but isn’t that already how it works in WebKit? EWS and buildbot run all the tests, but on your own machine you can get run-webkit-tests to run any subset you want.
> Not everybody has the hardware to produce debug builds of WebKit on all
> existing ports and run a big amount of tests :-)
> 
> AFAIK, EWS run all the builds as well as all the tests on macOS/iOS, you
> don't have options to select a subset of the tests or the ports.
> Buildbots run all the builds & tests on all platforms *after* a patch
> lands in the main repo. So we do too much for WIP patches (extra runtime
> cost I'm mentioning here) and not enough to check the final patch before
> landing (which causes different issues like extra rollout or gardening
> commits or difficult regression bisecting).

I find it super convenient that EWS runs all the tests even on WIP patches. It often catches test failures in tests I didn't think to run myself. I think it would be great if EWS could run tests on more platforms. Just running release regression tests on more platforms would be a big win, if debug builds are the gating factor.

I guess it would be kind of neat to have a feature of "run the tests across all configurations, but only some of them". But I don't think I would prioritize it over having more kinds of tests, or finding ways to make the full test suite run faster.

> 
> -- 
> Frédéric Wang - frederic-wang.fr
> 
>