[webkit-dev] Skipping Flakey Tests

[Dirk Pranke]

Somewhere in between the two of you I got lost.

Which "the framework" are you referring to? If you're referring to the
run-webkit-tests in WebKitTools/Scripts, you are correct that it has
no way to distinguish Safari/Win from Chromium/Win. This doesn't
really matter, since this framework isn't used by Chromium.

The Chromium framework doesn't look at the Skipped files, but does
look at the Safari Win baselines (we'll use those if we don't find a
better match). To Peter's point, this means that we have to check in a
correct expected result to override the incorrect one in platform/win,
which is annoying. I don't see any obvious way to get around this,
except that I think it probably does us little to no good to use the
Win expected results at all, and we should probably just skip them in
general. This doesn't really solve the problem in general (or if Mac
has this problem).

Note that if/when we happen to upstream the Chromium version of
run_webkit_tests, it does support marking files as 'FAIL', although it
does not currently allow us to capture 'expected fail' results
separately from 'expected pass' results. So, if we were all using
Chromium's run_webkit_tests, we could mark the Safari/Win version as
expected to fail (separately from SKIP), but I'm not sure if this is
what the Safari/Win guys want.

Given all that, Darin, what were you suggesting when you said "Let's fix that"?

-- Dirk

On Mon, Dec 21, 2009 at 1:54 PM, Darin Adler <darin at apple.com> wrote:
> On Dec 21, 2009, at 1:50 PM, Peter Kasting wrote:
>
>> the framework doesn't seem to have a way of distinguishing Safari/Win from Chromium/Win
>
> Lets fix that.
>
>    -- Darin
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>