[webkit-dev] Please don't leave entries for rebaseline in TestExpectation files

[Ryosuke Niwa]

To give you a perspective on how bad the current system is, just while I
was removing those 30 entires, I've found out that
fast/css-generated-content/table-row-group-to-inline.html has regressed
since it was first added. This regression should have caught by people
running pixel tests only if we had rebaselined it promptly.

I have seen dozens of cases where we had introduced regressions that would
have been caught by existing layout tests only if they had not been marked
flaky, failing, etc...

On Wed, Mar 20, 2013 at 11:46 PM, Robert Hogan <lists at roberthogan.net>wrote:

> On Wednesday, 20 March 2013, Ryosuke Niwa wrote:
>
>> Please don't add lines to TestExpectations saying that they just need
>> rebaselines and then leave.
>>
>
> OK. That means I will have to pull the new results from the bots, which is
> fine - but in the case of the Mac port (and any other bot that does not run
> pixel tests) the result will be that trunk will get fresh text results but
> retain stale png results.
>
> If that is OK then you need to publish that information somewhere as I
> suspect I'm not the only contributor who has hesitated to make Mac's test
> results inconsistent.
>

That's what non-Chromium ports do anyways.

That would reduce the test coverage we have, and effectively disables the
>> test. If you're adding those entires, please be sure to remove them
>> ASAP. Better yet, don't add them unless you have to rebaseline hundreds of
>> tests. It's not acceptable to leave those entries in TestExpectations for
>> days.
>>
>
> We've batted back and forth on this list for at least a year on the
> correct approach for landing and rebaselining. My approach is to land
> results for the platform that I build, suppress tests that require
> rebaselining on other platforms, and open a bug so sheriffs can
> add/rebaseline results as appropriate.
>

I don't think this approach scales.

My impression from recent discussion on this topic was that this was the
> way that worked best for everybody.
>

That was NOT my impression. I made it pretty clear that I dislike this
approach.

If you're referring to
https://lists.webkit.org/pipermail/webkit-dev/2013-February/023960.html,
then most of people who replied on that thread hadn't even contributed much
code to WebCore, and I highly doubt that their opinions represent the whole
WebKit community.

I used to pull results from the bots where possible but creating
> inconsistency between png/text results is not good.
>

It is unfortunate but it's much better than losing the complete test
coverage.

Presumably this will be discussed at the contributors' meeting - it would
> be good to make sure that all the relevant people required for consensus on
> this topic can attend the discussion and settle this once and for all!
>

Definitely.

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130321/058f673e/attachment.html>