[webkit-dev] Pixel test experiment
zimmermann at physik.rwth-aachen.de
Tue Oct 12 23:59:36 PDT 2010
Am 12.10.2010 um 22:43 schrieb James Robinson:
> To add a concrete data point, http://trac.webkit.org/changeset/69517
> caused a number of SVG tests to fail. It required 14 text
> rebaselines for Mac and a further two more for Leopard (done by Adam
> Barth). In order to pass the pixel tests in Chromium, it required
> 1506 new pixel baselines (checked in by the very brave Albert Wong, http://trac.webkit.org/changeset/69543)
> . None of the rebaselining was done by the patch authors and in
> general I would not expect a patch author that didn't work in
> Chromium to be expected to update Chromium-specific baselines. I'm
> a little skeptical of the claim that all SVG changes are run through
> the pixel tests given that to date none of the affected platform/mac
> SVG pixel baselines have been updated. This sort of mass-
> rebaselining is required fairly regularly for minor changes in SVG
> and in other parts of the codebase.
Dirk & me are running _every_ single SVG change through run-webkit-
tests --tolerance 0 -p svg.
Andreas Kling doesn't have access to a mac, and Dirk & me agreed to
rebaseline all affected SVG tests after his comment. Dirk did the 10.6
baseline updates, I added platform exceptions for 10.5 after Dirks
You can trust me, that we're running pixel tests.
> I'd really like for the bots to run the pixel tests on every run,
> preferably with 0 tolerance. We catch a lot of regressions by
> running these tests on the Chromium bots that would probably
> otherwise go unnoticed. However there is a large maintenance cost
> associated with this coverage. We normally have two engineers (one
> in PST, one elsewhere in the world) who watch the Chromium bots to
> triage, suppress, and rebaseline tests as churn is introduced.
> - If the pixel tests were running either with a tolerance of 0 or
> 0.1, what would the expectation be for a patch like http://trac.webkit.org/changeset/69517
> which requires hundreds of pixel rebaselines? Would the patch
> author be expected to update the baselines for the platform/mac
> port, or would someone else? Thus far the Chromium folks have been
> the only ones actively maintaining the pixel baselines - which I
> think is entirely reasonable since we're the only ones trying to run
> the pixel tests on bots.
As I said before, Dirk & me maintain the mac pixel baselines for SVG.
If I would have written the patch, I had included the rebaselines,
though ppl like Andreas who don't have access to pixel tests, have to
be able to produce patches as well.
> - Do we have the tools and infrastructure needed to do mass
> rebaselines in WebKit currently? We've built a number of tools to
> deal with the Chromium expectations, but since this has been a need
> unique to Chromium so far the tools only work for Chromium.
Yes, webkit-patch rebaseline but I think Adam already mentioned it.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the webkit-dev