[webkit-dev] Flaky test hit list

[Adam Barth]

In case you're interested in Leopard Debug, here's the hit list from
the last 2000 revisions.  Interesting notes:

1) Leopard seems way less flaky than Snow Leopard.
2) The websocket tests aren't flaky here.
3) http/tests/appcache/idempotent-update.html fails with an ASSERT
about a webView being null.

=== Results ===
Occurances Test name
         1 compositing/geometry/composited-html-size.html
         1 compositing/iframes/composited-parent-iframe.html
         1 fast/dom/collection-null-like-arguments.html
         1 fast/events/tabindex-focus-blur-all.html
         1 fast/js/vardecl-preserve-arguments.html
         1 inspector/debugger-pause-on-breakpoint.html
         1 inspector/extensions-events.html
         1 media/video-load-networkState.html
         1 media/video-pause-immediately.html
         1 media/video-size-intrinsic-scale.html
         1 media/video-zoom.html
         1 platform/mac/accessibility/area-with-aria-label.html
         1 platform/mac/fast/events/objc-event-api.html
         1 svg/dynamic-updates/SVGMaskElement-svgdom-maskUnits-prop.html
         2 animations/dynamic-stylesheet-loading.html
         2 http/tests/cache/subresource-expiration.html
         2 media/video-played-reset.html
         2 media/video-preload.html
         3 fast/events/wheelevent-in-scrolling-div.html
         3 svg/dom/SVGScriptElement/script-set-href.svg
         6 fast/files/workers/worker-read-blob-sync.html
         7 svg/W3C-SVG-1.1/fonts-glyph-02-t.svg
         8 media/video-display-aspect-ratio.html
        10 fast/files/workers/worker-read-blob-async.html
        22 media/video-src-change.html
        76 http/tests/appcache/idempotent-update.html


On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth <abarth at webkit.org> wrote:
> Tonight I wrote a new webkit-patch command for detecting flaky tests.
> Here the tests that have flaked out on the Snow Leopard (Tests) build
> bot during the last 2000 revisions.  This makes a good "hit list" of
> tests to fix to reduce flakiness.  (The worst offenders are at the
> bottom).
>
> Take aways:
>
> 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html
> is by far the worst offender.
> 2) The websocket tests are ridiculously flaky.
> 3) The appcache tests also have a serious flakiness problem.
>
> I'll run the last 10000 revisions over night and report on the results.
>
> Adam
>
>
> === Results ===
> Occurances Test name
>         1 compositing/reflections/nested-reflection-animated.html
>         1 fast/css/font-face-download-error.html
>         1 fast/dom/collection-null-like-arguments.html
>         1 fast/history/history-subframe-with-name.html
>         1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html
>         1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html
>         1 fast/js/vardecl-preserve-arguments.html
>         1 http/tests/appcache/different-https-origin-resource-main.html
>         1 http/tests/appcache/fallback.html
>         1 http/tests/appcache/manifest-redirect.html
>         1 http/tests/appcache/origin-quota.html
>         1 http/tests/appcache/resource-redirect.html
>         1 http/tests/appcache/top-frame-3.html
>         1 http/tests/appcache/update-cache.html
>         1 http/tests/appcache/xhr-foreign-resource.html
>         1 http/tests/cache/subresource-expiration.html
>         1 http/tests/loading/basic-credentials-sent-automatically.html
>         1 http/tests/misc/uncacheable-script-repeated.html
>         1 http/tests/navigation/changing-frame-hierarchy-in-onload.html
>         1 http/tests/navigation/ping-cross-origin-from-https.html
>         1 http/tests/navigation/ping-cross-origin.html
>         1 http/tests/navigation/post-goback-same-url.html
>         1 http/tests/plugins/get-url.html
>         1 http/tests/plugins/npapi-response-headers.html
>         1 http/tests/plugins/third-party-cookie-accept-policy.html
>         1 http/tests/security/credentials-in-referer.html
>         1 http/tests/security/cross-frame-access-protocol-explicit-domain.html
>         1 inspector/debugger-pause-on-breakpoint.html
>         1 inspector/extensions-events.html
>         1 media/audio-constructor.html
>         1 media/video-currentTime-set.html
>         1 plugins/destroy-stream-twice.html
>         1 plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html
>         1 svg/custom/use-invalid-style.svg
>         1 transitions/transition-end-event-transform.html
>         1 websocket/tests/send.html
>         1 websocket/tests/simple-stress.html
>         1 websocket/tests/sub-protocol-with-space.html
>         1 websocket/tests/url-no-trailing-slash.html
>         1 websocket/tests/url-with-empty-query.html
>         1 websocket/tests/url-with-query.html
>         1 websocket/tests/websocket-pending-activity.html
>         1 websocket/tests/workers/close-in-onmessage-crash.html
>         2 http/tests/appcache/foreign-iframe-main.html
>         2 http/tests/appcache/local-content.html
>         2 http/tests/appcache/main-resource-hash.html
>         2 http/tests/appcache/non-html.xhtml
>         2 http/tests/appcache/reload.html
>         2 http/tests/css/css-image-loading.html
>         2 http/tests/plugins/cross-frame-object-access.html
>         2 http/tests/security/cross-frame-access-port-explicit-domain.html
>         2 security/block-test.html
>         2 websocket/tests/bad-sub-protocol-non-ascii.html
>         2 websocket/tests/handshake-fail-by-sub-protocol-mismatch.html
>         2 websocket/tests/simple.html
>         2 websocket/tests/unicode.html
>         3 http/tests/appcache/cyrillic-uri.html
>         3 http/tests/appcache/deferred-events-delete-while-raising.html
>         3 http/tests/appcache/remove-cache.html
>         3 http/tests/appcache/top-frame-4.html
>         3 http/tests/navigation/image-load-in-unload-handler.html
>         3 websocket/tests/handshake-fail-by-cross-origin.html
>         3 websocket/tests/httponly-cookie.pl
>         3 websocket/tests/long-invalid-header.html
>         3 websocket/tests/sub-protocol.html
>         3 websocket/tests/url-with-query-for-no-query.html
>         4 fast/canvas/webgl/gl-object-get-calls.html
>         4 http/tests/navigation/anchor-basic.html
>         4 websocket/tests/bad-sub-protocol-control-chars.html
>         4 websocket/tests/frame-length-longer-than-buffer.html
>         4 websocket/tests/frame-length-skip.html
>         4 websocket/tests/frame-lengths.html
>         4 websocket/tests/reload-crash.html
>         4 websocket/tests/url-with-credential.html
>         5 websocket/tests/bad-sub-protocol-empty.html
>         5 websocket/tests/error-detect.html
>         5 websocket/tests/handshake-error.html
>         6 websocket/tests/handshake-challenge-randomness.html
>         6 websocket/tests/null-character.html
>         7 http/tests/security/cross-origin-css.html
>         7 websocket/tests/bufferedAmount-after-close.html
>         7 websocket/tests/close-on-unload.html
>         7 websocket/tests/frame-length-overflow.html
>         7 websocket/tests/handshake-fail-by-no-cr.html
>         8 websocket/tests/multiple-connections.html
>         9 websocket/tests/alert-in-event-handler.html
>         9 websocket/tests/close-on-unload-and-force-gc.html
>        10 websocket/tests/bad-handshake-crash.html
>        11 websocket/tests/cross-origin.html
>        12 http/tests/navigation/ping-same-origin.html
>        13 websocket/tests/close-on-unload-reference-in-parent.html
>        15 websocket/tests/close-on-navigate-new-location.html
>        19 media/audio-controls-rendering.html
>        50 compositing/geometry/limit-layer-bounds-transformed-overflow.html
>