[webkit-dev] Skipping Flakey Tests

[Maciej Stachowiak]

On Dec 22, 2009, at 10:31 AM, Darin Adler wrote:

> On Dec 21, 2009, at 6:14 PM, Dirk Pranke wrote:
>
>> Given all that, Darin, what were you suggesting when you said  
>> "Let's fix that"?
>
> Lets add a feature so something in the tests tree can indicate a  
> Chromium Windows result should be the base result, rather than the  
> platform/win one. We can debate exactly how it should work, but lets  
> come up with a design and do it.

One possibility is to have three windows-related result directories,  
win/, win-cg/ and win-chromium/. We'd use the latter two for test  
results that are different between the CoreGraphics/CFNetwork/Apple  
port vs. the Chromium/Google port. Any results that should be common  
to all windows-based platforms.

Note: As far as I can tell, this problem is all about granularity of  
applying the expected results, not about whether we check in known  
failing results instead of out-of-band FAIL expectations. The latter  
may or may not be a good idea(*), but it seems independent of the  
original problem. Some of Dirk and Peter's messages seemed to conflate  
these two points.

Regards,
Maciej

* - I tend to think we should track expected failure results, for the  
reasons Darin cited and also because tests that "fail" already can  
still catch further regressions.