[webkit-dev] Running pixel tests on build.webkit.org

[Dirk Pranke]

On Thu, Jan 7, 2010 at 8:18 PM, Darin Fisher <darin at chromium.org> wrote:
> On Thu, Jan 7, 2010 at 5:17 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>
>> Do we really need a separate set of bots for pixel tests? Lets just turn
>> the pixel tests on for the current bots. The only thing stopping us doing
>> that is the currently failing tests, hence the suggestion for adding an
>> expectations file (or we could skip all the failures).
>> I don't know enough about text metrics changes between Mac releases. With
>> Windows releases, we've been able to support XP, Vista and 7 pretty easily
>> by using a generic theme for OS controls. Also, I think we have some hooks
>> to turn off cleartype or something.
>
>
> ...and making sure all of the right / same fonts are installed :)
> -darin
>

We disable ClearType and do some minimal checking of the system UI
theme to make sure you don't have custom font or widget sizes. All of
our baselines use the fonts installed with the system, so there are no
custom fonts that need to be installed. You do need QuickTime
installed, or a few tests will fail, but otherwise it's pretty stock.

IIRC, there are far fewer diffs between Leopard and Snow Leopard than
there were between XP and Vista, so I doubt you would need the
"generic theme" that we did on Windows; it's probably easier to just
rebaseline the few files that do diff. I can't speak to the diffs
between Tiger and Leopard.

-- Dirk

>>
>> There are only ~10 tests that needed custom results for Vista && Windows
>> 7. I wonder if a similar set of steps could be taken for supporting
>> different Mac releases.
>> Ojan
>> On Thu, Jan 7, 2010 at 5:08 PM, Eric Seidel <eric at webkit.org> wrote:
>>>
>>> I'm totally in favor of adding test_expectations.txt like
>>> functionality to webkit (and we'll get it for free when Dirk finishes
>>> up-streaming run_webkit_tests.py)
>>>
>>> But the troubles with the pixel tests in the past were more to do with
>>> text metrics changing between OS releases, and individual font
>>> differences between machines.  I suspect that those issues are very
>>> solvable.
>>>
>>> I think we mostly need someone willing to set up the pixel test bots.
>>>
>>> -eric
>>>
>>> On Thu, Jan 7, 2010 at 5:01 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>> > On Thu, Jan 7, 2010 at 10:22 AM, Darin Adler <darin at apple.com> wrote:
>>> >>
>>> >> On Jan 7, 2010, at 10:19 AM, Dimitri Glazkov wrote:
>>> >> > Are we planning to run pixel tests on the build bots?
>>> >>
>>> >> If we can get them green, we should. It’s a lot of work. We need a
>>> >> volunteer to do that work. We’ve tried before.
>>> >
>>> > Two possible long-term solutions come to mind:
>>> > 1. Turn the bots orange on pixel failures. They still need fixing, but
>>> > are
>>> > not as severe as text diff failures. I'm not a huge fan of this, but
>>> > it's an
>>> > option.
>>> > 2. Add in a concept of expected failures and only turn the bots red for
>>> > *unexpected* failurs. More details on this below.
>>> > In chromium-land, there's an expectations file that lists expected
>>> > failures
>>> > and allows for distinguishing different types of failures (e.g. IMAGE
>>> > vs.
>>> > TEXT). It's like Skipped lists, but doesn't necessarily skip the test.
>>> > Fixing the expected failures still needs doing of course, but can be
>>> > done
>>> > asynchronously. The primary advantage of this approach is that we can
>>> > turn
>>> > on pixel tests, keep the bots green and avoid further regressions.
>>> > Would something like that make sense for WebKit as a whole? To be
>>> > clear, we
>>> > would be nearly as loathe to add tests to this file as we are about
>>> > adding
>>> > them to the Skipped lists. This just provides a way forward.
>>> > While it's true that the bots used to be red more frequently with pixel
>>> > tests turned on, for the most part, there weren't significant pixel
>>> > regressions. Now, if you run the pixel tests on a clean build, there
>>> > are a
>>> > number of failures and a very large number of hash-mismatches that are
>>> > within the failure tolerance level.
>>> > -Ojan
>>> > For reference, the format of the expectations file is something like
>>> > this:
>>> > // Fails the image diff but not the text diff.
>>> > fast/forms/foo.html = IMAGE
>>> > // Fails just the text diff.
>>> > fast/forms/bar.html = TEXT
>>> > // Fails both the image and text diffs.
>>> > fast/forms/baz.html = IMAGE+TEXT
>>> > // Skips this test (e.g. because it hangs run-webkit-tests or causes
>>> > other
>>> > tests to fail).
>>> > SKIP : fast/forms/foo1.html = IMAGE
>>> > _______________________________________________
>>> > webkit-dev mailing list
>>> > webkit-dev at lists.webkit.org
>>> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>> >
>>> >
>>
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>