[webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Maciej Stachowiak]

> On Nov 17, 2017, at 7:15 AM, Konstantin Tokarev <annulen at yandex.ru> wrote:
> 
> 
> 
> 16.11.2017, 20:10, "Alexey Proskuryakov" <ap at webkit.org>:
>> Migrating WebKit tests seems undesirable to me, as that would make us lose all their modification history. As mentioned before, WPT tests are inherently getting less attention in CI support, largely because of unclear benefit when there isn't any history for why the particular test was added.
> 
> Possible workaround would be to keep original file path as a comment inside new file. In this case history of ex-WebKit tests can be easily retraced from WebKit repository.

From Apple’s point of view, we are ok with leaving duplicate tests rather than removing them, especially when they were created as regression tests for specific bugs rather than as broad standards compliance tests.

>> Regression tests are taking an excessive amount of time to run for us (more than 1.5 hours in a debug build), a very large proportion of which is WPT. I do not think that the strategy of adopting WPT tests is working well for WebKit.
> 
> If existing tests are converted to WPT, do they take more time to run? If yes, it's quite unfortunate and needs to be fixed in WPT infrastructure.

I talked to Alexey bout this and he says there is indeed a higher per-test time cost, and that it’s a likely fixable problem with the WPT server. If anyone fixed this, EWS and buildbot turnaround time would improve significantly.

> 
>> 
>> - Alexey
>> 
>>> 15 нояб. 2017 г., в 2:02, Philip Jägenstedt <foolip at chromium.org> написал(а):
>>> 
>>> Hello webkit-dev! (ecosystem-infra in Bcc)
>>> 
>>> TPAC was last week, and there was much talk about web-platform-tests. Some notes are at https://lists.w3.org/Archives/Public/public-test-infra/2017OctDec/thread.html and in particular https://www.w3.org/2017/11/07-testing-minutes.html.
>>> 
>>> In Blink, we're thinking seriously about what it'd take to upstream large parts of LayoutTests into web-platform-tests, and we've realized that there are some risks here, beyond just making sure the tests are good and per spec. Specifically, one bad outcome would be if Blink successfully upstreamed thousands of tests that are also in WebKit, which then begin to diverge in small and large ways. That'd leave WebKit with more duplication between its own tests and web-platform-tests than any other engine, and a headache that grows over time.
>>> 
>>> So, I think this would require close cooperation with the WebKit project. Some different ways this might happen:
>>> 
>>> * A Blink developer and WebKit developer work together at the same time to move their common tests in some area into web-platform-tests, each removing identical tests that were upstreamed by the other and consolidating what remains.
>>> * A Blink developer works to first upstream tests from WebKit (using WebKit's coming export mechanism), waits for it to be imported into Blink, and then removes the tests from Blink.
>>> * The reverse situation.
>>> 
>>> Are there already some areas where the first approach might be doable, where the Blink and WebKit folks already know each other and are eager to do this? Is LayoutTests/css3/flexbox/ by any chance such a case?
>>> 
>>> I realize it's hard to judge these approaches in the abstract, but am curious to hear any enthusiasm or concerns about it.
>>> 
>>> Thanks!
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>> 
>> ,
>> 
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> 
> -- 
> Regards,
> Konstantin
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev