[webkit-dev] A proposal for handling "failing" layout tests and TestExpectations

[Peter Kasting]

On Wed, Aug 15, 2012 at 6:02 PM, Filip Pizlo <fpizlo at apple.com> wrote:

> 2) Possibility of the sheriff getting it wrong.
>
> (2) concerns me most.  We're talking about using filenames to serve as a
> kind of unchecked comment.  We already know that comments are usually bad
> because there is no protection against them going stale.
>

I don't see how this is parallel to stale comments.  Tests get continually
compared to the given output and we see immediately when something changes.

It is certainly possible for whoever assigns the filenames to get things
wrong.  There are basically two mitigations of this.  One is allowing the
existing "expected.xxx" file extensions to remain, and encouraging people
to leave them as-is when they're not sure whether the existing result is
correct.  The other is for sheriffs to use this signal as just that -- a
signal -- just as today we use the "expected.xxx" files as a signal of what
the correct output might be.  The difference is that this can generally be
considered a stronger signal.  Historically, there's been no real attempt
to guarantee that an "expected" result is anything other than the test's
current behavior.

I'm sure some would love to get rid of Skipped files just as much as I
> would love to get rid of TestExpectations files.  Both are valid things to
> love, and imply that there must surely exist a middle ground: a way of
> doing things that is strictly better than the sum of the two.
>

That's exactly what we're trying to do.

The value of this change is that hopefully it would dramatically reduce the
amount of content in these, especially in TestExpectations files.  If you
want to kill these so much, then this is a change you should at least let
us test!

In particular, to further clarify my position, if someone were to argue
> that Dirk's proposal would be a wholesale replacement for TestExpectations,
> then I would be more likely to be on board, since I very much like the idea
> of reducing the number of ways of doing things.  Maybe that's a good way to
> reach compromise.
>

It's hard to know if we could completely eliminate them without testing
this, but yes, one goal here is to greatly reduce the need for
TestExpectations lines.  A related goal is to make the patterns and
mechanisms used by all ports more similar.  As someone who has noted his
frustration with both "different ways of doing things" and "philosophical
directions chosen by one port", you would hopefully be well-served by this
direction.

PK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120815/b9aabeb0/attachment.html>