[webkit-dev] Skipping Flakey Tests
mjs at apple.com
Tue Dec 22 20:11:01 PST 2009
On Dec 22, 2009, at 10:31 AM, Darin Adler wrote:
> On Dec 21, 2009, at 6:14 PM, Dirk Pranke wrote:
>> Given all that, Darin, what were you suggesting when you said
>> "Let's fix that"?
> Lets add a feature so something in the tests tree can indicate a
> Chromium Windows result should be the base result, rather than the
> platform/win one. We can debate exactly how it should work, but lets
> come up with a design and do it.
One possibility is to have three windows-related result directories,
win/, win-cg/ and win-chromium/. We'd use the latter two for test
results that are different between the CoreGraphics/CFNetwork/Apple
port vs. the Chromium/Google port. Any results that should be common
to all windows-based platforms.
Note: As far as I can tell, this problem is all about granularity of
applying the expected results, not about whether we check in known
failing results instead of out-of-band FAIL expectations. The latter
may or may not be a good idea(*), but it seems independent of the
original problem. Some of Dirk and Peter's messages seemed to conflate
these two points.
* - I tend to think we should track expected failure results, for the
reasons Darin cited and also because tests that "fail" already can
still catch further regressions.
More information about the webkit-dev