[webkit-gtk] Pixel tests - 'Shall we?'

[Martin Robinson]

On Wed, Feb 2, 2011 at 7:18 AM, Zan Dobersek <zandobersek at gmail.com> wrote:
> Pixel tests are a good way of checking if any rendering has changed when it
> shouldn't. It may seem as a pain to either generate the results or keep them
> up to date, but with the right approach this should not be a problem. The
> benefits are bigger as I think more progressive testing allows for more
> progressive development.

Hey Zan, I'm glad you brought this up. I think this is a worthy goal.
Below are some of my observations for what's necessary before the bots
can run pixel tests. I think we can consider these blockers.

1. As you said we need to have a standard set of packages and versions
for running the layout tests. This needs to be documented somewhere.
Not only do we need to standardize on package versions, we need to
provide package repositories for (at least) Debian, Ubuntu and Fedora
(the distributions most of our developers use). Generally
non-WebKitGTK+ people do not have time to install things from source
to test their changes. This is vital to keeping our port green and up
to date.

2. We need to figure out how to account for the way that the default
GTK+ icons (used in input type=search, for instance) change between
releases.

3. We need faster bots. Our bots are already bogged down just making
things build. Running pixel tests will only exacerbate this need.

4. We still lack support for repaint tests.

--Martin