[webkit-dev] handling failing tests (test_expectations, Skipped files, etc.)

[Ryosuke Niwa]

On Mon, Apr 9, 2012 at 3:50 PM, Julien Chaffraix <jchaffraix at webkit.org>wrote:
>
>  > If there's consensus in the mean time that it is better on balance to
> > check in suppressions, perhaps we can figure out a better way to do
> > that. Maybe (shudder) a second test_expectations file? Or maybe it
> > would be better to actually check in suppressions marked as REBASELINE
> > (or something like that)?
>
> That sounds quirky as it involves maintaining 2 sets of files.
>
> From my perspective, saying that we should discard the EWS result and
> allow changes to get in WebKit trunk, knowing they will turn the bots
> red, is a bad proposal regardless of how you justify it. In the small
> delta where the bots are red, you can bet people will miss something
> else that breaks.
>

But that's the status quo. Until we get to the ideal world where every
port's EWS at least runs tests, there's no way around it since we can't
possibly require all contributors to run tests on all ports and platforms.

Also, adding new lines to test_expectations.txt will only work on Chromium
port since other ports use Skipped file instead, and skipping the test
won't give you new baseline on the bot. And this discrepancy between
Chromium and non-Chromium ports imposes a significant cognitive stress on
contributors and is not justifiable in my opinion.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120409/e3c3a3b4/attachment.html>