[webkit-qt] A quick look at LayoutTests (more specifically, http/tests/ and media/)

[Jesus Sanchez-Palencia]

Hi folks,

Today I spent some time taking a look at the status of our Layout
Tests. After taking a look at the Skipped lists I decided to give it a try
to two largely skipped folders: http/tests/  and media/ .

I used:   "run-webkit-tests --no-launch-safari --no-sample-on-timeout
--no-new-test-results --results-directory layout-test-results
--use-remote-links-to-tests --release --qt --force FOLDER", which was
copied from the Szeged bots.

I ran the tests for WebKit1 (6582a115083311f52297cde636c82c96c97f9f29) with
Qt 4.8 (6582a115083311f52297cde636c82c96c97f9f29) and for WebKit2 with Qt5
(6df6e99be09731991a1689cf1a71c601f4814865). For WK2 I ran it on 2 different
machines, running different Linux distros.

Perhaps I'm missing something or I did something wrong but, without _any_
fix to WebKit nor Qt, and for my surprise, I manage to get a reasonable
amount of "Expected to fail, but passed" tests:

- media/ WK2: ~110 of "unexpected-passing" tests.
- http/tests/ WK2: ~120 of "unexpected-passing" tests.
- media/ WK1: 68 "unexpected-passing" tests.
- http/tests/ WK1: 45 "unexpected-passing" tests.

Which, according to Ossy, are Skipped tests that were marked as fail, but
that can be passing now.
Here you can find the list of unexpected-passing tests for WK2:
http://pastebin.com/zBGHcYa3
Here you can find the list of unexpected-passing tests for WK1:
http://pastebin.com/ZyMpK2T1

Since I don't have access to a proper buildbot environment, I wonder if it
wouldn't be possible for the Szeged folks to take a look at this list and
try to unskip these tests. Perhaps just running the same command as I did
might spot a certain amount of tests that could be unskipped right now. I
would be surprise if there isn't at least 10 tests among these ~200 that
are passing already, not to say more...

Moreover, I believe this is a good way to easily spot tests that could be
passing; Just running the tests with --force to a certain folder and then
cross-checking the output list marked as "Expected to fail, but passed". I
was told this is the "Skipped list reducer week at Szeged", so I hope you
guys can use this somehow.

Cheers,
jesus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-qt/attachments/20111108/a3f30ec1/attachment.html>