[webkit-dev] Flaky test hit list

[Adam Barth]

Currently the WebSocket tests are served over HTTP from the WebSocket
server itself (which is written in Python).  It looks like we can
resolve the flakiness by serving the WebSocket tests from the normal
Apache server that servers the rest of our HTTP tests.  I'm going to
work up a patch that does that.  Please let me know if there's any
reason we shouldn't make that change.

Thanks,
Adam


On Thu, Sep 30, 2010 at 3:10 PM, Adam Barth <abarth at webkit.org> wrote:
> I'm investigating the websocket issue.  It seems these tests are flaky
> because they time out.  If you know about websockets, I'd appreciate
> any tips you have via #webkit.
>
> Adam
>
>
> On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth <abarth at webkit.org> wrote:
>> Tonight I wrote a new webkit-patch command for detecting flaky tests.
>> Here the tests that have flaked out on the Snow Leopard (Tests) build
>> bot during the last 2000 revisions.  This makes a good "hit list" of
>> tests to fix to reduce flakiness.  (The worst offenders are at the
>> bottom).
>>
>> Take aways:
>>
>> 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html
>> is by far the worst offender.
>> 2) The websocket tests are ridiculously flaky.
>> 3) The appcache tests also have a serious flakiness problem.
>>
>> I'll run the last 10000 revisions over night and report on the results.
>>
>> Adam
>>
>>
>> === Results ===
>> Occurances Test name
>>         1 compositing/reflections/nested-reflection-animated.html
>>         1 fast/css/font-face-download-error.html
>>         1 fast/dom/collection-null-like-arguments.html
>>         1 fast/history/history-subframe-with-name.html
>>         1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html
>>         1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html
>>         1 fast/js/vardecl-preserve-arguments.html
>>         1 http/tests/appcache/different-https-origin-resource-main.html
>>         1 http/tests/appcache/fallback.html
>>         1 http/tests/appcache/manifest-redirect.html
>>         1 http/tests/appcache/origin-quota.html
>>         1 http/tests/appcache/resource-redirect.html
>>         1 http/tests/appcache/top-frame-3.html
>>         1 http/tests/appcache/update-cache.html
>>         1 http/tests/appcache/xhr-foreign-resource.html
>>         1 http/tests/cache/subresource-expiration.html
>>         1 http/tests/loading/basic-credentials-sent-automatically.html
>>         1 http/tests/misc/uncacheable-script-repeated.html
>>         1 http/tests/navigation/changing-frame-hierarchy-in-onload.html
>>         1 http/tests/navigation/ping-cross-origin-from-https.html
>>         1 http/tests/navigation/ping-cross-origin.html
>>         1 http/tests/navigation/post-goback-same-url.html
>>         1 http/tests/plugins/get-url.html
>>         1 http/tests/plugins/npapi-response-headers.html
>>         1 http/tests/plugins/third-party-cookie-accept-policy.html
>>         1 http/tests/security/credentials-in-referer.html
>>         1 http/tests/security/cross-frame-access-protocol-explicit-domain.html
>>         1 inspector/debugger-pause-on-breakpoint.html
>>         1 inspector/extensions-events.html
>>         1 media/audio-constructor.html
>>         1 media/video-currentTime-set.html
>>         1 plugins/destroy-stream-twice.html
>>         1 plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html
>>         1 svg/custom/use-invalid-style.svg
>>         1 transitions/transition-end-event-transform.html
>>         1 websocket/tests/send.html
>>         1 websocket/tests/simple-stress.html
>>         1 websocket/tests/sub-protocol-with-space.html
>>         1 websocket/tests/url-no-trailing-slash.html
>>         1 websocket/tests/url-with-empty-query.html
>>         1 websocket/tests/url-with-query.html
>>         1 websocket/tests/websocket-pending-activity.html
>>         1 websocket/tests/workers/close-in-onmessage-crash.html
>>         2 http/tests/appcache/foreign-iframe-main.html
>>         2 http/tests/appcache/local-content.html
>>         2 http/tests/appcache/main-resource-hash.html
>>         2 http/tests/appcache/non-html.xhtml
>>         2 http/tests/appcache/reload.html
>>         2 http/tests/css/css-image-loading.html
>>         2 http/tests/plugins/cross-frame-object-access.html
>>         2 http/tests/security/cross-frame-access-port-explicit-domain.html
>>         2 security/block-test.html
>>         2 websocket/tests/bad-sub-protocol-non-ascii.html
>>         2 websocket/tests/handshake-fail-by-sub-protocol-mismatch.html
>>         2 websocket/tests/simple.html
>>         2 websocket/tests/unicode.html
>>         3 http/tests/appcache/cyrillic-uri.html
>>         3 http/tests/appcache/deferred-events-delete-while-raising.html
>>         3 http/tests/appcache/remove-cache.html
>>         3 http/tests/appcache/top-frame-4.html
>>         3 http/tests/navigation/image-load-in-unload-handler.html
>>         3 websocket/tests/handshake-fail-by-cross-origin.html
>>         3 websocket/tests/httponly-cookie.pl
>>         3 websocket/tests/long-invalid-header.html
>>         3 websocket/tests/sub-protocol.html
>>         3 websocket/tests/url-with-query-for-no-query.html
>>         4 fast/canvas/webgl/gl-object-get-calls.html
>>         4 http/tests/navigation/anchor-basic.html
>>         4 websocket/tests/bad-sub-protocol-control-chars.html
>>         4 websocket/tests/frame-length-longer-than-buffer.html
>>         4 websocket/tests/frame-length-skip.html
>>         4 websocket/tests/frame-lengths.html
>>         4 websocket/tests/reload-crash.html
>>         4 websocket/tests/url-with-credential.html
>>         5 websocket/tests/bad-sub-protocol-empty.html
>>         5 websocket/tests/error-detect.html
>>         5 websocket/tests/handshake-error.html
>>         6 websocket/tests/handshake-challenge-randomness.html
>>         6 websocket/tests/null-character.html
>>         7 http/tests/security/cross-origin-css.html
>>         7 websocket/tests/bufferedAmount-after-close.html
>>         7 websocket/tests/close-on-unload.html
>>         7 websocket/tests/frame-length-overflow.html
>>         7 websocket/tests/handshake-fail-by-no-cr.html
>>         8 websocket/tests/multiple-connections.html
>>         9 websocket/tests/alert-in-event-handler.html
>>         9 websocket/tests/close-on-unload-and-force-gc.html
>>        10 websocket/tests/bad-handshake-crash.html
>>        11 websocket/tests/cross-origin.html
>>        12 http/tests/navigation/ping-same-origin.html
>>        13 websocket/tests/close-on-unload-reference-in-parent.html
>>        15 websocket/tests/close-on-navigate-new-location.html
>>        19 media/audio-controls-rendering.html
>>        50 compositing/geometry/limit-layer-bounds-transformed-overflow.html
>>
>