[webkit-dev] Skipping Flakey Tests

[Peter Kasting]

On Thu, Oct 1, 2009 at 10:41 AM, Drew Wilson <atwilson at google.com> wrote:

> In this case, there was a failure in one of the layout tests on the windows
> platform, so following the advice below, aroben correctly checked in an
> update to the test expectations instead of skipping the tests.
>
> Downstream, this busted the Chromium tests, because that failure was not
> happening in Chromium, and now our correct test output doesn't match the
> incorrect test output that's been codified in the test expecations. We can
> certainly manage this downstream by rebaselining the test and managing a
> custom chromium test expectation, but that's a pain and is somewhat fragile
> as it requires maintenance every time someone adds a new test case to the
> test.
>

This came up again this past Friday.
http://trac.webkit.org/changeset/52324added purposefully-failing
results for WebKit Windows, which broke Chromium
downstream because we don't fail the test.

Darin's original reply here included the line "And we should structure test
results and exceptions so that it’s easy to get the expected failure on the
right platforms and success on others."  It seems like this isn't the case
currently, since the framework doesn't seem to have a way of distinguishing
Safari/Win from Chromium/Win, meaning the only way we can express the
current state of affairs via result snapshots is to check in a bad baseline
over the good one for all Windows ports and then have each port that passes
check in a good baseline.  This is a pretty poor experience :(, and more
than "a slight inconvenience" as Darin dismissively termed it.

I liked Dirk's idea of being able to note that a test is failing, rather
than skipping it or checking in a bogus baseline.  I don't see a lot of
value in the bad-baseline strategy beyond keeping the test running, and
noting "this test fails on Safari/Win" accomplishes that same objective in a
less-misleading and more-other-port-friendly fashion.

PK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20091221/79343e51/attachment.html>