[webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Frédéric WANG]

On 17/11/2017 16:26, Maciej Stachowiak wrote:
> WebKit has a lot of tests that were regression tests for a specific
> bug fix, not as conformance tests (though they might b useful for that
> too). In such cases, the association with the bugs.webkit.org bug is
> more important than the spec URL. That’s particularly the case when
> the test has been changed multiple times to reflect further WebKit
> behavior changes. I know that I’ve personally found it very useful to
> look at revision history of tests, or search for bug numbers or
> keywords in LayoutTests/ChangeLog to find related tests.
> Not saying this is the sole purpose of tests, but losing this ability or making it more awkward would be a downside to deleting tests from the WebKit repo.
Well, I think we agree here, that's why "conformance" tests was
specified in my message. For such tests, connecting the history of tests
with the development of the specs is actually more important. For
example I recently noticed that some bugs with WOFF2 (on Apple's ports
only) and WebVTT have been ignored in WebKit because we don't
import/sync WPT tests for these specs.
> Maybe I’m missing something, but isn’t that already how it works in WebKit? EWS and buildbot run all the tests, but on your own machine you can get run-webkit-tests to run any subset you want.
Not everybody has the hardware to produce debug builds of WebKit on all
existing ports and run a big amount of tests :-)

AFAIK, EWS run all the builds as well as all the tests on macOS/iOS, you
don't have options to select a subset of the tests or the ports.
Buildbots run all the builds & tests on all platforms *after* a patch
lands in the main repo. So we do too much for WIP patches (extra runtime
cost I'm mentioning here) and not enough to check the final patch before
landing (which causes different issues like extra rollout or gardening
commits or difficult regression bisecting).

-- 
Frédéric Wang - frederic-wang.fr


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20171117/6186b6bb/attachment.bin>