[webkit-dev] A proposal for handling "failing" layout tests and TestExpectations
ojan at chromium.org
Thu Aug 16 14:50:58 PDT 2012
On Thu, Aug 16, 2012 at 2:32 PM, Filip Pizlo <fpizlo at apple.com> wrote:
> 1) Switching to skipping flaky tests wholesale in all ports would be
> great, and then we could get rid of the flakiness support.
Then you don't notice when a flaky tests stops being flaky. The cost of
flakiness support on the project does not seem large to me and it's pretty
frequent that a test will only be flaky for a few hundred runs (e.g. due to
a bad checkin that gets fixed), so we can then remove it from
TestExpectations and run the test as normal.
2) The WONTFIX mode in TestExpectations feels to me more like a statement
> that you're just trying to see if the test doesn't crash. Correct? Either
> way, it's confusing.
WONTFIX is for tests that don't make sense to fix for the given port (e.g.
dashboard-specific tests for non-Apple ports). It's a way of distinguishing
tests that we're skipping because of a bug on our part vs. tests that we're
skipping because the test doesn't apply to the port.
3) Your new mechanism feels like it's already covered by our existing use
> of -expected files. I'm not quite convinced that having -failing in
> addition to -expected files would be all that helpful.
But there are many cases where we *know* the result is incorrect, e.g. it
doesn't match a spec. Sure, there are also many cases where it's not clear
what the correct behavior is.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the webkit-dev