[webkit-dev] Proposal to start automating the process of updating imported WPT tests inside WebKit.

[Carlos Alberto Lopez Perez]

Hello webkit-dev,

I would like to start a discussion about our current process for updating
WPT tests, and to see if we can agree on some process to semi-automate it.

Currently the process of updating WPT tests its a very manual task which involves:
1. Raise the WPT commit in LayoutTests/imported/w3c/resources/TestRepositories
2. Run the importer script: Tools/Scripts/import-w3c-tests
3. Check that what the script did makes sense (usually those changes
   are so big that's impossible to check manually)
4. Run layout tests to generate new expectations for new tests.
5. Upload the patch to bugzilla, wait for the EWS.
6. Check what failed in each platform.
7. Open new bugs for the new failures in reftests.
8. Add the new failures to the TestExpectations.
9. Repeat from 5. until everything is green in the EWS.
10. Ask for review: the reviewer is usually also unable to review the changes in
   details because the patch is so big that its impossible to review in detail.
11. Land it

This is so tedious task, that the last time we updated this globally was
on 2018-September (that its the date of WPT commit 0313d9f which is what
its pointed on the file LayoutTests/imported/w3c/resources/TestRepositories)

Engineers interested only in a subset of WPT tests (ex: domparsing/)
what they do, its to update individual folders from WPT ToT (instead of
updating all the imported WPT checkout). But this has its own problems:
sometimes the updated folder depends on file from other folder that its
not updated and unexpected problems happpen (example: <https://webkit.org/b/205445>)

I think that having the whole imported WPT checkout inside WebKit updated
often would be a benefit for everybody.

So, we have the idea of trying to semi-automate this. On the steps above
it would mean automating everything except the step of reviewing (10.)

Our proposal would be that a bot runs weekly and generates a patch
with the update, then uploads it to Bugzilla, waits for the EWS, opens
bugs for new failures and updates expectations accordingly; and repeats
until everything is green in the EWS. Then the bot would CC the reviewers
and ask for r? and cq?.

Note that this bot would *not* try to import new folders from WPT (folders
for new specs) into WebKit. It would only update the ones already imported.

The bigger benefit would be having our set of imported WPT tests updated
often, allowing WebKit engineers to run the regression tests with the last
version of WPT tests and to get noticed early if some update on a WPT test
breaks.

Also, engineers interested in WPT tests would not need anymore to manually
update the tests since that its done now for them. Only they would need
to import the new tests if those tests belong to a WPT folder (spec)
still not imported inside WebKit.

Other side benefit of this is that reviewing the patch with the WPT checkout
update that the bot generates would be something humanly possible to review.
Since we would do that each few days, the changes hopefully would be in the dozens
of files instead of the thousands of files.

The are also disadvantages to this, like the extra gardening work that will
be generated due to this constant update of tests. Specially regarding flakies
or failures not catched by the EWS (this may be a big problem for platforms
that don't have an EWS running layout tests. [1])

So there its a trade-off to be made regarding the frequency of updates
(daily, weekly, etc). Maybe we can start with weekly and see how it goes.

Just out of curiosity, I have checked how often Chromium updates their WPT
imported tests, and they do that several times per day and fully automated
[2].

We previously outlined this proposed process in this document:
https://hackmd.io/b4ViVtDsSk6Qg8fAycexVA#3-Import-w3c-tests-running-in-a-bot

Thanks for reading so far.

Please let me know your thoughts.

Best regards!
-------------


[1]
For GTK and WPE this should not be a problem as we are already
looking forward to make our EWS bots run layout tests.
[2]
See:
http://sprunge.us/a56RIo and
https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_platform_tests.md#Automatic-import-process

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 897 bytes
Desc: OpenPGP digital signature
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20200302/0307889a/attachment.bin>