[webkit-dev] handling failing tests (test_expectations, Skipped files, etc.)

[Julien Chaffraix]

>>> If there's consensus in the mean time that it is better on balance to
>>> check in suppressions, perhaps we can figure out a better way to do
>>> that. Maybe (shudder) a second test_expectations file? Or maybe it
>>> would be better to actually check in suppressions marked as REBASELINE
>>> (or something like that)?
>>
>> That sounds quirky as it involves maintaining 2 sets of files.
>>
>> From my perspective, saying that we should discard the EWS result and
>> allow changes to get in WebKit trunk, knowing they will turn the bots
>> red, is a bad proposal regardless of how you justify it. In the small
>> delta where the bots are red, you can bet people will miss something
>> else that breaks.
>>
>
> As Ryosuke points out, practically we're already in that situation -
> from what I can tell, the tree is red virtually all of the time, at
> least during US/Pacific working hours. It's not clear to me if the EWS
> has made this better or worse, but perhaps others have noticed a
> difference. That said, I doubt I like red trees any more than you do
> :)

I wasn't talking about the tree's status quo here as it shouldn't
impact the discussion. Just because the tree is red, doesn't mean it's
the right thing (tm) to just drop the towel and make it more red (even
if we seem to agree on the badness of that :)).

To add some thoughts here, saying that the tree is red covers several
states (failing tests, not building...) and IMHO the EWS has at least
helped on the building side. As far as the tests goes, a lot of
platform differences are unfortunately uncovered on the bots.

Thanks,
Julien