[webkit-dev] Flaky test hit list
Fumitoshi Ukai (鵜飼文敏)
ukai at chromium.org
Fri Oct 1 00:05:30 PDT 2010
On Fri, Oct 1, 2010 at 12:03, Adam Barth <abarth at webkit.org> wrote:
> https://bugs.webkit.org/show_bug.cgi?id=46956
>
> Unfortunately, it doesn't solve the whole problem. It does seem to
> reduce the flakiness by a lot though.
>
Thanks!
I think it might be race between DRT and pywebsocekt server, that is, DRT
tries to run tests while pywebsocket server is still initializing and not
yet ready to serve html. I'll try to fix pywebsocket to reduce
initialization latency.
--
ukai
>
> Adam
>
>
> On Thu, Sep 30, 2010 at 6:12 PM, Adam Barth <abarth at webkit.org> wrote:
> > Currently the WebSocket tests are served over HTTP from the WebSocket
> > server itself (which is written in Python). It looks like we can
> > resolve the flakiness by serving the WebSocket tests from the normal
> > Apache server that servers the rest of our HTTP tests. I'm going to
> > work up a patch that does that. Please let me know if there's any
> > reason we shouldn't make that change.
> >
> > Thanks,
> > Adam
> >
> >
> > On Thu, Sep 30, 2010 at 3:10 PM, Adam Barth <abarth at webkit.org> wrote:
> >> I'm investigating the websocket issue. It seems these tests are flaky
> >> because they time out. If you know about websockets, I'd appreciate
> >> any tips you have via #webkit.
> >>
> >> Adam
> >>
> >>
> >> On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth <abarth at webkit.org> wrote:
> >>> Tonight I wrote a new webkit-patch command for detecting flaky tests.
> >>> Here the tests that have flaked out on the Snow Leopard (Tests) build
> >>> bot during the last 2000 revisions. This makes a good "hit list" of
> >>> tests to fix to reduce flakiness. (The worst offenders are at the
> >>> bottom).
> >>>
> >>> Take aways:
> >>>
> >>> 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html
> >>> is by far the worst offender.
> >>> 2) The websocket tests are ridiculously flaky.
> >>> 3) The appcache tests also have a serious flakiness problem.
> >>>
> >>> I'll run the last 10000 revisions over night and report on the results.
> >>>
> >>> Adam
> >>>
> >>>
> >>> === Results ===
> >>> Occurances Test name
> >>> 1 compositing/reflections/nested-reflection-animated.html
> >>> 1 fast/css/font-face-download-error.html
> >>> 1 fast/dom/collection-null-like-arguments.html
> >>> 1 fast/history/history-subframe-with-name.html
> >>> 1
> fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html
> >>> 1
> fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html
> >>> 1 fast/js/vardecl-preserve-arguments.html
> >>> 1 http/tests/appcache/different-https-origin-resource-main.html
> >>> 1 http/tests/appcache/fallback.html
> >>> 1 http/tests/appcache/manifest-redirect.html
> >>> 1 http/tests/appcache/origin-quota.html
> >>> 1 http/tests/appcache/resource-redirect.html
> >>> 1 http/tests/appcache/top-frame-3.html
> >>> 1 http/tests/appcache/update-cache.html
> >>> 1 http/tests/appcache/xhr-foreign-resource.html
> >>> 1 http/tests/cache/subresource-expiration.html
> >>> 1 http/tests/loading/basic-credentials-sent-automatically.html
> >>> 1 http/tests/misc/uncacheable-script-repeated.html
> >>> 1 http/tests/navigation/changing-frame-hierarchy-in-onload.html
> >>> 1 http/tests/navigation/ping-cross-origin-from-https.html
> >>> 1 http/tests/navigation/ping-cross-origin.html
> >>> 1 http/tests/navigation/post-goback-same-url.html
> >>> 1 http/tests/plugins/get-url.html
> >>> 1 http/tests/plugins/npapi-response-headers.html
> >>> 1 http/tests/plugins/third-party-cookie-accept-policy.html
> >>> 1 http/tests/security/credentials-in-referer.html
> >>> 1
> http/tests/security/cross-frame-access-protocol-explicit-domain.html
> >>> 1 inspector/debugger-pause-on-breakpoint.html
> >>> 1 inspector/extensions-events.html
> >>> 1 media/audio-constructor.html
> >>> 1 media/video-currentTime-set.html
> >>> 1 plugins/destroy-stream-twice.html
> >>> 1
> plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html
> >>> 1 svg/custom/use-invalid-style.svg
> >>> 1 transitions/transition-end-event-transform.html
> >>> 1 websocket/tests/send.html
> >>> 1 websocket/tests/simple-stress.html
> >>> 1 websocket/tests/sub-protocol-with-space.html
> >>> 1 websocket/tests/url-no-trailing-slash.html
> >>> 1 websocket/tests/url-with-empty-query.html
> >>> 1 websocket/tests/url-with-query.html
> >>> 1 websocket/tests/websocket-pending-activity.html
> >>> 1 websocket/tests/workers/close-in-onmessage-crash.html
> >>> 2 http/tests/appcache/foreign-iframe-main.html
> >>> 2 http/tests/appcache/local-content.html
> >>> 2 http/tests/appcache/main-resource-hash.html
> >>> 2 http/tests/appcache/non-html.xhtml
> >>> 2 http/tests/appcache/reload.html
> >>> 2 http/tests/css/css-image-loading.html
> >>> 2 http/tests/plugins/cross-frame-object-access.html
> >>> 2
> http/tests/security/cross-frame-access-port-explicit-domain.html
> >>> 2 security/block-test.html
> >>> 2 websocket/tests/bad-sub-protocol-non-ascii.html
> >>> 2 websocket/tests/handshake-fail-by-sub-protocol-mismatch.html
> >>> 2 websocket/tests/simple.html
> >>> 2 websocket/tests/unicode.html
> >>> 3 http/tests/appcache/cyrillic-uri.html
> >>> 3 http/tests/appcache/deferred-events-delete-while-raising.html
> >>> 3 http/tests/appcache/remove-cache.html
> >>> 3 http/tests/appcache/top-frame-4.html
> >>> 3 http/tests/navigation/image-load-in-unload-handler.html
> >>> 3 websocket/tests/handshake-fail-by-cross-origin.html
> >>> 3 websocket/tests/httponly-cookie.pl
> >>> 3 websocket/tests/long-invalid-header.html
> >>> 3 websocket/tests/sub-protocol.html
> >>> 3 websocket/tests/url-with-query-for-no-query.html
> >>> 4 fast/canvas/webgl/gl-object-get-calls.html
> >>> 4 http/tests/navigation/anchor-basic.html
> >>> 4 websocket/tests/bad-sub-protocol-control-chars.html
> >>> 4 websocket/tests/frame-length-longer-than-buffer.html
> >>> 4 websocket/tests/frame-length-skip.html
> >>> 4 websocket/tests/frame-lengths.html
> >>> 4 websocket/tests/reload-crash.html
> >>> 4 websocket/tests/url-with-credential.html
> >>> 5 websocket/tests/bad-sub-protocol-empty.html
> >>> 5 websocket/tests/error-detect.html
> >>> 5 websocket/tests/handshake-error.html
> >>> 6 websocket/tests/handshake-challenge-randomness.html
> >>> 6 websocket/tests/null-character.html
> >>> 7 http/tests/security/cross-origin-css.html
> >>> 7 websocket/tests/bufferedAmount-after-close.html
> >>> 7 websocket/tests/close-on-unload.html
> >>> 7 websocket/tests/frame-length-overflow.html
> >>> 7 websocket/tests/handshake-fail-by-no-cr.html
> >>> 8 websocket/tests/multiple-connections.html
> >>> 9 websocket/tests/alert-in-event-handler.html
> >>> 9 websocket/tests/close-on-unload-and-force-gc.html
> >>> 10 websocket/tests/bad-handshake-crash.html
> >>> 11 websocket/tests/cross-origin.html
> >>> 12 http/tests/navigation/ping-same-origin.html
> >>> 13 websocket/tests/close-on-unload-reference-in-parent.html
> >>> 15 websocket/tests/close-on-navigate-new-location.html
> >>> 19 media/audio-controls-rendering.html
> >>> 50
> compositing/geometry/limit-layer-bounds-transformed-overflow.html
> >>>
> >>
> >
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20101001/beffb70b/attachment.html>
More information about the webkit-dev
mailing list