[webkit-dev] Pixel test experiment

[Nikolas Zimmermann]

Am 12.10.2010 um 22:43 schrieb James Robinson:

> To add a concrete data point, http://trac.webkit.org/changeset/69517  
> caused a number of SVG tests to fail.  It required 14 text  
> rebaselines for Mac and a further two more for Leopard (done by Adam  
> Barth).  In order to pass the pixel tests in Chromium, it required  
> 1506 new pixel baselines (checked in by the very brave Albert Wong, http://trac.webkit.org/changeset/69543) 
> .  None of the rebaselining was done by the patch authors and in  
> general I would not expect a patch author that didn't work in  
> Chromium to be expected to update Chromium-specific baselines.  I'm  
> a little skeptical of the claim that all SVG changes are run through  
> the pixel tests given that to date none of the affected platform/mac  
> SVG pixel baselines have been updated.  This sort of mass- 
> rebaselining is required fairly regularly for minor changes in SVG  
> and in other parts of the codebase.

Dirk & me are running _every_ single SVG change through run-webkit- 
tests --tolerance 0 -p svg.
Andreas Kling doesn't have access to a mac, and Dirk & me agreed to  
rebaseline all affected SVG tests after his comment. Dirk did the 10.6  
baseline updates, I added platform exceptions for 10.5 after Dirks  
commit.

You can trust me, that we're running pixel tests.


> I'd really like for the bots to run the pixel tests on every run,  
> preferably with 0 tolerance.  We catch a lot of regressions by  
> running these tests on the Chromium bots that would probably  
> otherwise go unnoticed.  However there is a large maintenance cost  
> associated with this coverage.  We normally have two engineers (one  
> in PST, one elsewhere in the world) who watch the Chromium bots to  
> triage, suppress, and rebaseline tests as churn is introduced.
>
> Questions:
> - If the pixel tests were running either with a tolerance of 0 or  
> 0.1, what would the expectation be for a patch like http://trac.webkit.org/changeset/69517 
>  which requires hundreds of pixel rebaselines?  Would the patch  
> author be expected to update the baselines for the platform/mac  
> port, or would someone else?  Thus far the Chromium folks have been  
> the only ones actively maintaining the pixel baselines - which I  
> think is entirely reasonable since we're the only ones trying to run  
> the pixel tests on bots.
As I said before, Dirk & me maintain the mac pixel baselines for SVG.  
If I would have written the patch, I had included the rebaselines,  
though ppl like Andreas who don't have access to pixel tests, have to  
be able to produce patches as well.

>
> - Do we have the tools and infrastructure needed to do mass  
> rebaselines in WebKit currently?  We've built a number of tools to  
> deal with the Chromium expectations, but since this has been a need  
> unique to Chromium so far the tools only work for Chromium.
Yes, webkit-patch rebaseline but I think Adam already mentioned it.

Cheers,
Niko

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20101013/e645237f/attachment.html>