[webkit-dev] WPT import status

[Sam Sneddon]

https://github.com/WebKit/WebKit/pull/21912 hopefully serves as good progress here: this PR is almost largely done by automated tooling.

The first commit is the result of `import-w3c-tests` (the long standing command), but with no further work (to the expectations or baselines).

It then ran through EWS with the `no-failure-limits` label, before I added a second commit: the output of `update-test-expectations github-pr`, albeit limited to only changes it made to the relevant (dom) directory. This is where all the baselines come from in the PR.

This a roughly a workflow that’s useable in some cases today.

The most obvious limitation is that it only deals with baseline files (*-expected.txt), and not TestExpectations files (which means for things where the result is stored there—including reftest failures, debug differences, timeouts and crashes—it doesn’t work). Of the limitations this causes, reftests are by far the more apparent and mean this doesn’t work for most CSS tests yet.

The other obvious limitation is that it doesn’t deal with flaky tests, as shown by the PR failing after the second commit.

The final limitation, hinted at above, is that this doesn’t take any consideration of the current results on main—so it rebaselines far too much, depending on how well gardened main is when the PR run.

That said, the first and last of these are very much things we care about fixing, and we’ll see how problematic the second requiring manual intervention is.

I’d encourage people importing directories which have few reftests to try using the scripts mentioned above, and file any bugs they find.

What would be super helpful, to avoid making “turn on automated import” too painful, is updating directories we have imported that haven’t been updated recently.

And finally, as an aside, the `import-expectations.json` file now explicitly skips every top level directory we don’t import: I’d encourage you to take a look at what’s marked as skip, and if there’s more you think we should be importing (especially because we have implemented the spec in question!), I’d encourage you to do so by running `import-w3c-tests` with that directory, which will also change the relevant line in `import-expectations.json` from “skip” to “import”.

Happy New Year, (Web)Kittens,

Sam

PS: Apparently I messed up and my original draft with many more links to bugs didn’t get saved to a server-side draft mailbox, and I’m now on vacation, so please accept this much less-useful-than-intended email on current WPT import work! (I’ll try to reply with all the links once I get home in the New Year, although I will remain on leave for a while beyond.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20231218/398dccbc/attachment.htm>