[webkit-dev] handling failing tests (test_expectations, Skipped files, etc.)

[Julien Chaffraix]

> In my ideal world, you would be able to get updated baselines *prior*
> to trying to land a patch. This is of course not really possible today
> for any test that fails on multiple ports with different results, as
> it's practically impossible to run more than a couple of ports by
> hand, and we don't have a bot infrastructure to help you here.

That would be the best outcome indeed, however that would more or less
require all the EWS to be able to run the tests (including pixel tests
for the platforms that enable them).

> 3) the current tool-of-the-day for managing rebaselining,
> garden-o-matic, is much better suited to handling the "unexpected"
> failures on the bots rather than the "expected" failures you've
> introduced.

You could see it the other way. How could we make garden-o-matic
handle newly added suppression better? Maybe some new sub-tool listing
the newly added suppressions would help? Ignore the suppressions added
XX hours ago?

Your issue seems to be suppressions sticking into the tree not
strictly suppressing in itself. Our current tool (garden-o-matic)
handles failures a lot better than it handles suppressions.

> If there's consensus in the mean time that it is better on balance to
> check in suppressions, perhaps we can figure out a better way to do
> that. Maybe (shudder) a second test_expectations file? Or maybe it
> would be better to actually check in suppressions marked as REBASELINE
> (or something like that)?

That sounds quirky as it involves maintaining 2 sets of files.

>From my perspective, saying that we should discard the EWS result and
allow changes to get in WebKit trunk, knowing they will turn the bots
red, is a bad proposal regardless of how you justify it. In the small
delta where the bots are red, you can bet people will miss something
else that breaks.

Thanks,
Julien