[webkit-help] How to make all run-webkit-tests test pass? Or is it possible?

[Benjamin Poulain]

The errors are odd. They seem to be randomly distributed.

Do you use Mac Ports? I have seen strange errors before when Mac Ports 
override system libraries.

Benjamin

On 9/3/13 4:11 PM, Jeff wrote:
> Thank you Benjamin, Alexey
>
> This time, "I use run-webkit-tests -2" and following are my test this 
> time. There are not so much fail, but is it normal?
>
> From results.html, I can find:
> Tests that failed text/pixel/audio diff (5)
> Tests that had stderr output (5)
> Flaky tests (failed the first run and passed on retry) (5)
> Tests expected to fail but passed (20)
> Maybe I just mis-understanding the results.html?
>
> BEGIN OF TEST RESULT
> =========================
> Zheng-no-Mac-Pro:WebKit_git zhengxu$ Tools/Scripts/run-webkit-tests -2
> Using port 'mac-mountainlion-wk2'
> Test configuration: <mountainlion, x86_64, debug>
> Placing test results in 
> /Users/zhengxu/03_workspace/webkit/WebKit_git/WebKitBuild/Debug/layout-test-results
> Baseline search path: mac-wk2 -> wk2 -> mac -> generic
> Using Debug build
> Pixel tests disabled
> Regular timeout: 80000, slow test timeout: 400000
> Command line: 
> /Users/zhengxu/03_workspace/webkit/WebKit_git/WebKitBuild/Debug/WebKitTestRunner 
> -
>
> --lint-test-files warnings:
> LayoutTests/platform/mac/TestExpectations:1236 More specific entry for 
> fast/block/float/overhanging-tall-block.html on line 
> LayoutTests/platform/mac/TestExpectations:1236 overrides line 
> LayoutTests/platform/mac/TestExpectations:1102. 
> fast/block/float/overhanging-tall-block.html
>
> Found 34997 tests; running 32001, skipping 2996.
> Running 18 WebKitTestRunners in parallel.
>
> [19/32001] compositing/backgrounds/fixed-background-on-descendant.html 
> passed unexpectedly
> [20/32001] compositing/background-color/background-color-alpha.html 
> passed unexpectedly
>   compositing/backgrounds/fixed-backgrounds.html -> ref test hashes 
> didn't match but diff passed
> [47/32001] compositing/backgrounds/fixed-backgrounds.html passed 
> unexpectedly
> compositing/background-color/background-color-change-to-text.html -> 
> ref test hashes didn't match but diff passed
> [47/32001] 
> compositing/background-color/background-color-change-to-text.html 
> passed unexpectedly
> [68/32001] 
> compositing/background-color/background-color-composite.html passed 
> unexpectedly
> [79/32001] 
> compositing/background-color/background-color-container.html passed 
> unexpectedly
> [106/32001] 
> compositing/background-color/background-color-content-clip.html passed 
> unexpectedly
> [138/32001] 
> compositing/background-color/background-color-padding-clip.html passed 
> unexpectedly
> [165/32001] compositing/background-color/background-color-simple.html 
> passed unexpectedly
> [186/32001] 
> compositing/background-color/background-color-text-change.html passed 
> unexpectedly
> [276/32001] compositing/images/positioned-image-content-rect.html 
> passed unexpectedly
> [594/32001] 
> compositing/geometry/fixed-position-composited-page-scale-smaller-than-viewport.html 
> passed unexpectedly
> [694/32001] 
> http/tests/cache/willsendrequest-returns-null-for-memory-cache-load.html 
> passed unexpectedly
>   css3/filters/custom/custom-filter-blend-modes.html -> ref test 
> hashes didn't match but diff passed
>   css3/unicode-bidi-isolate-aharon-failing.html -> ref test hashes 
> didn't match but diff passed
> [938/32001] css3/unicode-bidi-isolate-aharon-failing.html passed 
> unexpectedly
> [1081/32001] compositing/video/video-object-fit.html failed 
> unexpectedly (text diff)
>   css3/filters/custom/composited/custom-filter-blend-modes.html -> ref 
> test hashes didn't match but diff passed
> [2770/32001] 
> css3/filters/custom/effect-custom-transform-parameters.html passed 
> unexpectedly
> [5468/32001] editing/style/make-text-writing-direction-inline-mac.html 
> failed unexpectedly (text diff)
> [5577/32001] editing/style/make-text-writing-direction-inline-win.html 
> failed unexpectedly (text diff)
>   fast/css/sticky/sticky-left-percentage.html -> ref test hashes 
> didn't match but diff passed
> [9087/32001] http/tests/inspector/inspect-element.html failed 
> unexpectedly (text diff)
> [9694/32001] fast/dom/Range/range-expand.html failed unexpectedly 
> (text diff)
> [12142/32001] 
> fast/layers/no-clipping-overflow-hidden-added-after-transition.html 
> passed unexpectedly
> [14235/32001] fast/events/scrollbar-double-click.html failed 
> unexpectedly (text diff)
> [15010/32001] fast/repaint/region-painting-in-composited-view.html 
> passed unexpectedly
>   fast/regions/fixed-inside-named-flow-zIndex.html -> ref test hashes 
> didn't match but diff passed
>   fast/regions/fixed-pos-elem-in-region.html -> ref test hashes didn't 
> match but diff passed
> [15560/32001] fast/regions/fixed-pos-elem-in-region.html passed 
> unexpectedly
> [15619/32001] fast/regions/fixed-pos-region-in-nested-flow.html failed 
> unexpectedly (reference mismatch)
> [15671/32001] fast/text/international/cjk-segmentation.html failed 
> unexpectedly (text diff)
> [19221/32001] inspector/storage-panel-dom-storage-update.html failed 
> unexpectedly (text diff)
> [20326/32001] jquery/manipulation.html failed unexpectedly (text diff)
>   svg/repaint/buffered-rendering-static-image.html -> ref test hashes 
> didn't match but diff passed
> [27261/32001] svg/repaint/buffered-rendering-static-image.html passed 
> unexpectedly
> [29599/32001] http/tests/navigation/back-twice-without-commit.html 
> passed unexpectedly
>
> Retrying 10 unexpected failure(s) ...
>
> Running 1 WebKitTestRunner.
>
> [3/10] editing/style/make-text-writing-direction-inline-mac.html 
> failed unexpectedly (text diff)
> [4/10] editing/style/make-text-writing-direction-inline-win.html 
> failed unexpectedly (text diff)
> [5/10] fast/dom/Range/range-expand.html failed unexpectedly (text diff)
> [7/10] fast/regions/fixed-pos-region-in-nested-flow.html failed 
> unexpectedly (reference mismatch)
> [8/10] fast/text/international/cjk-segmentation.html failed 
> unexpectedly (text diff)
>
> 31971 tests ran as expected, 30 didn't:
>
>
> Expected to timeout, but passed: (1)
>   http/tests/navigation/back-twice-without-commit.html
>
> Expected to fail, but passed: (20)
>   compositing/background-color/background-color-alpha.html
> compositing/background-color/background-color-change-to-text.html
>   compositing/background-color/background-color-composite.html
>   compositing/background-color/background-color-container.html
> compositing/background-color/background-color-content-clip.html
> compositing/background-color/background-color-padding-clip.html
>   compositing/background-color/background-color-simple.html
>   compositing/background-color/background-color-text-change.html
>   compositing/backgrounds/fixed-background-on-descendant.html
>   compositing/backgrounds/fixed-backgrounds.html
> compositing/geometry/fixed-position-composited-page-scale-smaller-than-viewport.html
>   compositing/images/positioned-image-content-rect.html
>   css3/filters/custom/effect-custom-transform-parameters.html
>   css3/unicode-bidi-isolate-aharon-failing.html
> fast/layers/no-clipping-overflow-hidden-added-after-transition.html
>   fast/regions/fixed-pos-elem-in-region.html
>   fast/repaint/region-painting-in-composited-view.html
> http/tests/cache/willsendrequest-returns-null-for-memory-cache-load.html
>   svg/repaint/buffered-rendering-static-image.html
>   tables/mozilla_expected_failures/bugs/bug178855.xml
>
>
> Unexpected flakiness: text-only failures (5)
>   compositing/video/video-object-fit.html [ Failure Pass ]
>   fast/events/scrollbar-double-click.html [ Failure Pass ]
>   http/tests/inspector/inspect-element.html [ Failure Pass ]
>   inspector/storage-panel-dom-storage-update.html [ Failure Pass ]
>   jquery/manipulation.html [ Failure Pass ]
>
>
> Regressions: Unexpected text-only failures (4)
>   editing/style/make-text-writing-direction-inline-mac.html [ Failure ]
>   editing/style/make-text-writing-direction-inline-win.html [ Failure ]
>   fast/dom/Range/range-expand.html [ Failure ]
>   fast/text/international/cjk-segmentation.html [ Failure ]
>
> Regressions: Unexpected image-only failures (1)
>   fast/regions/fixed-pos-region-in-nested-flow.html [ ImageOnlyFailure ]
>
> Failed to execute Tools/Scripts/new-run-webkit-tests at 
> Tools/Scripts/run-webkit-tests line 100.
> =========================
> END OF TEST RESULT
>
> (2013/09/04 7:08), Benjamin Poulain wrote:
>> On 9/3/13 6:43 AM, 徐征 wrote:
>>> I am not familiar with layout test of webkit. I created some patch 
>>> but hope it is easy to find any degression via layout test.
>>>
>>> Everytime, I create some patch, I just run 
>>> |Tools/Script/run-webkit-tests and all the time there are many 
>>> fail(more than 100).|
>>> ||I think maybe I just use wrong option for this script.
>>> |It there anyway to pass all the test in run-webkit-tests?
>>> Or, if it is normal that not all test are passed, is there any way 
>>> to confirm degression efficient?
>>>
>>> I am using OSX10.8.4.
>>> And I find in http://www.webkit.org/quality/testing.html, it is 
>>> written that "Regression tests must be run on Leopard", should I run 
>>> test on Leopard?|
>> The testing webpage is wrong, I'll fix it.
>>
>> The common ways to run the tests are "run-webkit-tests -2" and 
>> "run-webkit-test".
>>
>> OS X 10.8.4 is the best platform to run the tests, you should be able 
>> to run them without problems.
>> What kind of errors do you get?
>>
>> Benjamin
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-help/attachments/20130903/3b52e848/attachment.html>