[webkit-dev] test_expectations.txt for non-chromium ports

[Ojan Vafai]

On Mon, Feb 13, 2012 at 1:06 PM, Maciej Stachowiak <mjs at apple.com> wrote:

> On Feb 10, 2012, at 4:20 PM, Dirk Pranke wrote:
>
> > I think at one point Adam indicated he wanted to use them for the
> > Apple Win port, but he is still using the Skipped files since the Win
> > port is still using ORWT on the bots.
> >
> > That said, I understand why you're asking this (I think), but I would
> > prefer that we not have to support both options indefinitely into the
> > future. I would like to merge whatever features we need from both
> > approaches into one solution if possible (of course, we can do that
> > after forcing ports to use only one or the other for now).
>
> I agree that combining the features of both approach is the best long-term
> solution.
>
> I haven't checked with others who have an interest in the ports maintained
> by Apple, but I at least like different aspects of the two systems and
> would love to settle on a unified solution.
>

Sounds good to me if there is someone willing to find consensus on a final
design and actually implement it.

> The last time this was discussed at any length, people seemed to want
> > expectations to cascade, but I never got any clear feedback on what
> > the semantics of the cascade would be, and how that might interact
> > with different modifiers in the test_expectations file.
> >
> > Since that time, I've come to believe that even the way Chromium uses
> > expectations files is just making things harder for developers and
> > maintainers, and so I would like to change how Chromium does things as
> > well ... this is a long-winded way of saying most things are on the
> > table for discussion.
> >
> > For example, we might want to use only Skipped files for tests that
> > are always planned to be skipped, and test_expectations for things
> > that are supposed to be temporary workarounds to keep the tree green
> > until bugs can be filed and new baselines generated. Or, we might want
> > to do something else ...
>
> I'd much prefer a single file, but perhaps with different status and uses
> of states than the current test_expectations.txt format.
>

I think agreeing on the list of states and whether to have a single file or
cascading files are the two controversial points. While (if?) we figure
this out, it seems better to me if we do the simple work of only allowing
either of a test_expectations.txt file or Skipped lists. Until someone does
the work of unifying the two systems, it will continue to be a source of
confusion to allow both for a given port.


> I do agree that distinguishing "test not applicable to this port" from
> "this test is temporarily failing for unknown reasons" is a good thing to
> do. It is unfortunate that we don't make the distinction very well right
> now.
>

test_expectations.txt has a WONTFIX modifier for this purpose. Chromium
used to have two files test_expectations.txt and
test_expectations_wontfix.txt instead of having the modifier. I would kind
of like us to move back to that model because then test_expectations.txt is
a file that you hope to keep completely empty
and test_expectations_wontfix.txt is a file that your rarely touch.

Ojan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120213/e7c01771/attachment.html>