[webkit-dev] new-run-webkit-tests and Mac pixel tests

[Dirk Pranke]

Hi all,

I have just landed a patch that updates a bunch of incorrect image
checksums, and adds all of the currently failing pixel tests to the
"test_expectations.txt" file in LayoutTests/platform/mac.

If you take a look at that file you'll see that they are marked with a
BUG36620 string and the string "IMAGE" to indicate that they are
expected to pass as text-only tests but fail image diffs.

If you use new-run-webkit-tests, pixel tests are now enabled by
default on the Mac, and you should get a clean test run indicating
that all tests run as expected.

Hopefully this is a first step to getting pixel tests passing all the
time (which I realize may be a controversial goal).

Apart from the fact that the patch fixed a bunch of incorrect
checksums, the standard run-webkit-tests should be unaffected.

-- Dirk

[1] If you disable pixel tests (using --no-pixel-tests), then you will
get a bunch of warnings that tests expected to fail with pixel
failures are passing (because the test infrastructure isn't smart
enough yet to realize that we're not doing pixel tests). See
https://bugs.webkit.org/show_bug.cgi?id=36689

[2] Also, if you run on a machine with more than 4 CPUs, you may find
new-run-webkit-tests hanging quite frequently. This is a known bug [3]
that we are trying to hunt down. I have a patch out to change the
default in the meantime [4]. To work around this, simply use
--num-test-shells=4 (or less).

[3] https://bugs.webkit.org/show_bug.cgi?id=36622

[4] https://bugs.webkit.org/show_bug.cgi?id=36687