[webkit-dev] test_expectations.txt for non-chromium ports

[Maciej Stachowiak]

On Feb 10, 2012, at 4:20 PM, Dirk Pranke wrote:

> I think at one point Adam indicated he wanted to use them for the
> Apple Win port, but he is still using the Skipped files since the Win
> port is still using ORWT on the bots.
> 
> That said, I understand why you're asking this (I think), but I would
> prefer that we not have to support both options indefinitely into the
> future. I would like to merge whatever features we need from both
> approaches into one solution if possible (of course, we can do that
> after forcing ports to use only one or the other for now).

I agree that combining the features of both approach is the best long-term solution.

I haven't checked with others who have an interest in the ports maintained by Apple, but I at least like different aspects of the two systems and would love to settle on a unified solution.

> 
> The last time this was discussed at any length, people seemed to want
> expectations to cascade, but I never got any clear feedback on what
> the semantics of the cascade would be, and how that might interact
> with different modifiers in the test_expectations file.
> 
> Since that time, I've come to believe that even the way Chromium uses
> expectations files is just making things harder for developers and
> maintainers, and so I would like to change how Chromium does things as
> well ... this is a long-winded way of saying most things are on the
> table for discussion.
> 
> For example, we might want to use only Skipped files for tests that
> are always planned to be skipped, and test_expectations for things
> that are supposed to be temporary workarounds to keep the tree green
> until bugs can be filed and new baselines generated. Or, we might want
> to do something else ...

I'd much prefer a single file, but perhaps with different status and uses of states than the current test_expectations.txt format.

I do agree that distinguishing "test not applicable to this port" from "this test is temporarily failing for unknown reasons" is a good thing to do. It is unfortunate that we don't make the distinction very well right now.

Regards,
Maciej