[webkit-qt] A quick look at LayoutTests (more specifically, http/tests/ and media/)

[Balazs Kelemen]

Nice results, I hope we will be able to unksip a lot of tests. The guys 
here are working on it right now ;-)
One problem with this could be that even if some tests are passing in a 
particular directory, unskipping them can sometime cause other tests in 
other directories to fail. It should not be too frequent but it means 
you need to rerun the tests to check it. And of course the boring issue 
of divergent results between the bot and other systems. So far we didn't 
take much effort in reducing the skip list because even mantaining bot 
greenness (with continuosly growing skip lists) needs a lot of efforts, 
but it's time to change on it!

-KBalazs

On 11/08/2011 11:17 PM, Jesus Sanchez-Palencia wrote:
> Hi folks,
>
> Today I spent some time taking a look at the status of our Layout 
> Tests. After taking a look at the Skipped lists I decided to give it a 
> try to two largely skipped folders: http/tests/  and media/ .
>
> I used:   "run-webkit-tests --no-launch-safari --no-sample-on-timeout 
> --no-new-test-results --results-directory layout-test-results 
> --use-remote-links-to-tests --release --qt --force FOLDER", which was 
> copied from the Szeged bots.
>
> I ran the tests for WebKit1 (6582a115083311f52297cde636c82c96c97f9f29) 
> with Qt 4.8 (6582a115083311f52297cde636c82c96c97f9f29) and for WebKit2 
> with Qt5 (6df6e99be09731991a1689cf1a71c601f4814865). For WK2 I ran it 
> on 2 different machines, running different Linux distros.
>
> Perhaps I'm missing something or I did something wrong but, without 
> _any_ fix to WebKit nor Qt, and for my surprise, I manage to get a 
> reasonable amount of "Expected to fail, but passed" tests:
>
> - media/ WK2: ~110 of "unexpected-passing" tests.
> - http/tests/ WK2: ~120 of "unexpected-passing" tests.
> - media/ WK1: 68 "unexpected-passing" tests.
> - http/tests/ WK1: 45 "unexpected-passing" tests.
>
> Which, according to Ossy, are Skipped tests that were marked as fail, 
> but that can be passing now.
> Here you can find the list of unexpected-passing tests for WK2: 
> http://pastebin.com/zBGHcYa3
> Here you can find the list of unexpected-passing tests for WK1: 
> http://pastebin.com/ZyMpK2T1
>
> Since I don't have access to a proper buildbot environment, I wonder 
> if it wouldn't be possible for the Szeged folks to take a look at this 
> list and try to unskip these tests. Perhaps just running the same 
> command as I did might spot a certain amount of tests that could be 
> unskipped right now. I would be surprise if there isn't at least 10 
> tests among these ~200 that are passing already, not to say more...
>
> Moreover, I believe this is a good way to easily spot tests that could 
> be passing; Just running the tests with --force to a certain folder 
> and then cross-checking the output list marked as "Expected to fail, 
> but passed". I was told this is the "Skipped list reducer week at 
> Szeged", so I hope you guys can use this somehow.
>
> Cheers,
> jesus
>
>
>
>
>
> _______________________________________________
> webkit-qt mailing list
> webkit-qt at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-qt

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-qt/attachments/20111110/3c8da29c/attachment-0001.html>