[webkit-dev] A proposal for handling "failing" layout tests and TestExpectations

[Filip Pizlo]

On Aug 15, 2012, at 5:51 PM, Peter Kasting <pkasting at chromium.org> wrote:

> On Wed, Aug 15, 2012 at 5:45 PM, Filip Pizlo <fpizlo at apple.com> wrote:
> The typical approach used in situations that you describe is to rebase, not skip.
> 
> Which is precisely what Dirk is proposing.  Literally the only difference here is that the rebased test expectation would contain an optional notation about whether we believe the new baseline to be correct or incorrect.

Can you articulate the value that this gives?  We must weigh that against the downsides:

1) Increased complexity of infrastructure.

2) Possibility of the sheriff getting it wrong.

(2) concerns me most.  We're talking about using filenames to serve as a kind of unchecked comment.  We already know that comments are usually bad because there is no protection against them going stale.

> 
> Ports that don't care, or tests where we don't know, will be totally unaffected.  I am not sure why this bothers you so much.  You talk about making the infrastructure more complicated, but it doesn't seem like there is much additional complication involved, and what minor complication is involved is being volunteered to be handled by Dirk and other folks.

Ultimately it will lead to a further multiplication of number of possible ways of doing things, which is bad.  I'm sure some would love to get rid of Skipped files just as much as I would love to get rid of TestExpectations files.  Both are valid things to love, and imply that there must surely exist a middle ground: a way of doing things that is strictly better than the sum of the two.

In particular, to further clarify my position, if someone were to argue that Dirk's proposal would be a wholesale replacement for TestExpectations, then I would be more likely to be on board, since I very much like the idea of reducing the number of ways of doing things.  Maybe that's a good way to reach compromise.

Dirk, what value do you see in TestExpectations were your change to be landed?  Do scenarios still exist where there would be a test for which (a) there is no -fail.* file, (b) the test is not skipped, and (c) it's marked with some annotation in TestExpectations?  I'm most interested in the question of such scenarios exist, since in my experience, whenever a test is not rebased, is not skipped, and is marked as failing in TestExpectations, it ends up just causing gardening overhead later.

-F

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120815/2d06b583/attachment.html>