[webkit-dev] [PSA] WebKitGTK layout testers available on the Bugzilla EWS bubbles

[Carlos Alberto Lopez Perez]

Hi,

Some of you might have noticed that since a few days ago on the Bugzilla
there is a new EWS bubble for GTK WK2 layout tests.

Until now the GTK port was without layout test coverage on the EWS.

It has been a challenge to add this testers on the EWS because the GTK
port layout tests are usually not 100% green and there are usually
unexpected failures and also flakies. This means that the EWS was
usually either reporting false positives (due to flakies) or exiting
early (because of more than 30 unexpected failures on the clean tree or
more than 30 accumulated between the ones caused by patch and the ones
in the clean tree)

We tried to do extra gardening efforts to try to mark all the flakies
and also to try to keep it always green, but for several reasons we
couldn't keep it on this always-green status for more than a few
consecutive days. Also new flakies would keep appearing.

So we ended deploying a different version of the EWS that has a much
higher tolerance to pre-existent failures (up to 500 before exiting
early) and also that tries hard to discard pre-existent failures and
flakies by repeating each failure 10 times with patch and 10 times
without it. [1]
This new version of the EWS WK2 layout tester is only used on the GTK
port for now, and we plan to use it also for the WPE port when we deploy
layout tests on the EWS for it.

So this has the advantage that this EWS should not report false
positives or flakies. Any failure reported by it should be a consistent
new failure (fails always with the patch but never without it after 10
runs with patch and 10 without it). If you see it reporting any false
positive please let me know it.

However, it has also the disadvantage that the view to see the layout
test results is a bit confusing because you see there also failures
unrelated with your patch (those are either pre-existent failures or
flakies)

My tip here is to only pay attention to the test failures reported by
the EWS as new, which are those that are shown on the Bugzilla bubble
when you hover the mouse over it (or those that appear on the e-mail
that you receive from the EWS)

Perhaps I'm thinking it can be a further improvement to add an extra
step to do a final run with the patch just for the list of new failures,
in order to present a clean view of the results with only the new failures.
This can be also very useful to be able to properly use the script
update-test-expectations-from-bugzilla to automatically apply the new
test expectations from the EWS locally.

Any comments welcome :)


Best regards!

[1] https://bugs.webkit.org/show_bug.cgi?id=231999