[webkit-dev] A proposal for handling "failing" layout tests and TestExpectations

[Ojan Vafai]

On Thu, Aug 16, 2012 at 2:32 PM, Filip Pizlo <fpizlo at apple.com> wrote:

> 1) Switching to skipping flaky tests wholesale in all ports would be
> great, and then we could get rid of the flakiness support.
>

Then you don't notice when a flaky tests stops being flaky. The cost of
flakiness support on the project does not seem large to me and it's pretty
frequent that a test will only be flaky for a few hundred runs (e.g. due to
a bad checkin that gets fixed), so we can then remove it from
TestExpectations and run the test as normal.

2) The WONTFIX mode in TestExpectations feels to me more like a statement
> that you're just trying to see if the test doesn't crash.  Correct?  Either
> way, it's confusing.
>

WONTFIX is for tests that don't make sense to fix for the given port (e.g.
dashboard-specific tests for non-Apple ports). It's a way of distinguishing
tests that we're skipping because of a bug on our part vs. tests that we're
skipping because the test doesn't apply to the port.

3) Your new mechanism feels like it's already covered by our existing use
> of -expected files.  I'm not quite convinced that having -failing in
> addition to -expected files would be all that helpful.
>

But there are many cases where we *know* the result is incorrect, e.g. it
doesn't match a spec. Sure, there are also many cases where it's not clear
what the correct behavior is.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120816/0300d1d8/attachment.html>