[webkit-gtk] Status of the threaded compositor

[Michael Catanzaro]

On Thu, 2016-07-21 at 14:45 +0200, Carlos Garcia Campos wrote:
> Now the results look much better:
> 
> Unexpected image-only failures (62)
> Unexpected crashes (10)
> 
> Many of the remaining image only failures are because the mock
> scrollbars used in layout tests dont't support overlay.

I spent today gardening and found a total of 93 tests which I'm fairly
certain were broken by threaded compositor. We're unfortunately seeing
wild swings in the number of flaky tests between each run, e.g. r203601
with 220 flakes, r203602 with only 122 flakes, and r203603 with 172
flakes. The swings are going to continue for a while until we can get
these marked as flaky; I think I've gotten us roughly halfway (reduced
the variance down to a difference of roughly 50 flakes at most between
runs, from ~100), but that means we've got many more threaded
compositor failures yet to be recorded.

In a few cases, it looks like some general bug in threaded compositor
results in random tests crashing, but but in the vast majority of
cases, threaded compositor is definitely responsible for making a
particular test very flaky. We will need extra gardening during the
next few days, to keep watching the bots and keep marking tests as
flaky, to get our expectations under control again. We might want to
modify the gardening schedule accordingly to have multiple people
working this week, as we probably want to get a handle on this sooner
rather than later.

We're also seeing wild swings in the number of tests passing, for
instance, I got our unexpected passes down to 11 in r203650 after
gardening, then it shot up back to 68 unexpected passes in r203651
despite no relevant changes. (These tests must be flaky too, just
happened to pass WTR twice in one run. I haven't updated expectations
for these yet.)

Anyway, I'm going to copy/paste the tests I've found broken so far. I
emphasize that we're not done finding failures until the flake count
stops swinging wildly.

#////////////////////////////////////////////////////////////////////////////////////////
# Tests failing due to switch to THREADED COMPOSITOR
#////////////////////////////////////////////////////////////////////////////////////////

webkit.org/b/160119 accessibility/media-element.html [ Timeout Crash Pass ]

webkit.org/b/160119 animations/negative-delay.html [ Failure Pass ]
webkit.org/b/160119 animations/trigger-container-scroll-empty.html [ Failure Pass ]

webkit.org/b/160119 css3/calc/simple-composited-mask.html [ ImageOnlyFailure Pass ]
webkit.org/b/160119 css3/calc/table-empty-cells.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 css3/masking/mask-base64.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 editing/input/caret-at-the-edge-of-contenteditable.html [ Failure ]
webkit.org/b/160119 editing/input/reveal-caret-of-multiline-contenteditable.html [ Failure ]

# Note that caret-at-bidi-boundary was previously marked Slow before threaded
# compositor introduced timeouts.
webkit.org/b/160119 editing/selection/caret-at-bidi-boundary.html [ Failure Pass Timeout ]
webkit.org/b/160119 editing/selection/select-bidi-run.html [ Timeout Pass ]

webkit.org/b/160119 media/media-fullscreen-inline.html [ Crash Pass ]
webkit.org/b/160119 media/video-playbackrate.html [ Crash Pass ]

webkit.org/b/160119 fast/backgrounds/size/backgroundSize-in-background-shorthand.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 fast/block/basic/020.html [ Failure ]

webkit.org/b/160119 fast/block/margin-collapse/empty-clear-blocks-complex.html [ ImageOnlyFailure Pass ]
webkit.org/b/160119 fast/block/positioning/016.html [ Failure ]
webkit.org/b/160119 fast/block/positioning/025.html [ Failure ]
webkit.org/b/160119 fast/block/positioning/fixed-position-stacking-context.html [ Failure ]

webkit.org/b/160119 fast/css3-text/css3-text-decoration/repaint/underline-outside-of-layout-rect-altered.html [ Failure Pass ]
webkit.org/b/160119 fast/css3-text/css3-text-decoration/repaint/underline-outside-of-layout-rect-removed.html [ Failure ]
webkit.org/b/160119 fast/css3-text/css3-text-decoration/repaint/underline-outside-of-layout-rect.html [ Failure ]

webkit.org/b/160119 fast/css3-text/css3-text-justify/text-justify-last-line-simple-line-layout.html [ Failure ]

webkit.org/b/160119 fast/dom/rtl-scroll-to-leftmost-and-resize.html [ Failure Pass ]

webkit.org/b/160119 fast/dynamic/012.html [ Failure ]
webkit.org/b/160119 fast/dynamic/layer-hit-test-crash.html [ Failure ]

webkit.org/b/160119 fast/flexbox/012.html [ Failure Pass ]
webkit.org/b/160119 fast/flexbox/aspect-ratio-intrinsic-adjust.html [ ImageOnlyFailure Pass ]
webkit.org/b/160119 fast/flexbox/flex-hang.html [ Failure Pass ]
webkit.org/b/160119 fast/flexbox/image-percent-max-height.html [ Failure Pass ]

webkit.org/b/160119 fast/forms/enter-clicks-buttons.html [ Failure Pass ]
webkit.org/b/160119 fast/forms/focus-selection-textarea.html [ Failure Pass ]

webkit.org/b/160119 fast/forms/select/listbox-disabled-no-autoscroll.html [ ImageOnlyFailure Pass ]
webkit.org/b/160119 fast/forms/select/select-painting.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 fast/images/animated-png.html [ ImageOnlyFailure Pass ]
webkit.org/b/160119 fast/images/background-position.html [ ImageOnlyFailure Pass ]
webkit.org/b/160119 fast/images/composited-animated-gif-outside-viewport.html [ Timeout Pass ]

webkit.org/b/160119 fast/layers/layer-visibility-sublayer.html [ Failure ]
webkit.org/b/160119 fast/layers/layer-visibility.html [ Failure ]

webkit.org/b/160119 fast/media/mq-relative-constraints-08.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 fast/multicol/border-radius-overflow-columns.html [ ImageOnlyFailure Pass ]
webkit.org/b/160119 fast/multicol/mixed-opacity-fixed-test.html [ Failure ]

webkit.org/b/160119 fast/regions/frame-view-overflow-scroll.html [ ImageOnlyFailure Pass ]
webkit.org/b/160119 fast/regions/regions-overflow-tile.html [ Failure Pass ]

webkit.org/b/160119 fast/repaint/backgroundSizeRepaint.html [ Failure Pass ]
webkit.org/b/160119 fast/repaint/selection-gap-fixed-child.html [ Failure ]
webkit.org/b/160119 fast/repaint/selection-gap-flipped-fixed-child.html [ Failure ]

webkit.org/b/160119 fast/scrolling/iframe-scrollable-after-back.html [ Failure Pass ]
webkit.org/b/160119 fast/scrolling/rtl-scrollbars-alternate-iframe-body-dir-attr-does-not-update-scrollbar-placement.html [ Failure ]
webkit.org/b/160119 fast/scrolling/rtl-scrollbars-sticky-document.html [ ImageOnlyFailure Pass ]
webkit.org/b/160119 fast/scrolling/rtl-scrollbars-sticky-document-2.html [ Failure ]

webkit.org/b/160119 fast/text/crash-complex-unknown-font.html [ Failure Pass ]
webkit.org/b/160119 fast/text/orthogonal-writing-mode-containing-block-frameView-resize-relayout.html [ ImageOnlyFailure ]

webkit.org/b/160119 fast/text/whitespace/028.html [ Failure ]

webkit.org/b/160119 http/tests/css/vertical-align-baseline-after-image-load-2.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 imported/blink/animations/display-inline-style-adjust.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 imported/blink/compositing/squashing/squash-above-fixed-subpixel-2.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 imported/blink/css3/blending/mix-blend-mode-has-ancestor-clipping-layer.html [ ImageOnlyFailure ]

webkit.org/b/160119 imported/blink/css3/filters/effect-reference-source-alpha-hw.html [ ImageOnlyFailure ]

webkit.org/b/160119 imported/blink/fast/css/first-letter-range-insert.html [ Crash Pass ]

webkit.org/b/160119 imported/blink/fast/gradients/gradient-transparency.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 imported/blink/fast/pagination/body-make-unpaginated.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 imported/blink/fast/multicol/span/overflow-on-viewport.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 imported/blink/fast/overflow/scroll-html-hidden-body.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 imported/blink/svg/animations/viewspec-animated-viewbox.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 imported/blink/transforms/inline-transform-and-clipping-roots.html [ Crash Pass ]

webkit.org/b/160119 imported/mozilla/svg/dynamic-small-object-scaled-up-01.svg [ ImageOnlyFailure Pass ]
webkit.org/b/160119 imported/mozilla/svg/stroke-width-percentage-02a.svg [ ImageOnlyFailure Pass ]

webkit.org/b/160119 imported/w3c/web-platform-tests/IndexedDB/idbcursor_iterating.htm [ Failure Pass ]

webkit.org/b/160119 imported/w3c/web-platform-tests/fetch/api/redirect/redirect-location.html [ Failure Pass ]

webkit.org/b/160119 imported/w3c/web-platform-tests/html/semantics/embedded-content/media-elements/event_progress.html [ Timeout Crash Pass ]

webkit.org/b/160119 mathml/opentype/large-operators-displaystyle-dynamic.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 media/broken-video.html [ Crash Pass ]
webkit.org/b/160119 media/click-volume-bar-not-pausing.html [ Timeout Failure Pass ]
webkit.org/b/160119 media/media-continues-playing-after-replace-source.html [ Timeout Pass ]
webkit.org/b/160119 media/media-captions.html [ Timeout Failure Crash ]
webkit.org/b/160119 media/media-controller-drag-crash.html [ Timeout Pass ]
webkit.org/b/160119 media/media-controller-time.html [ Timeout Pass ]
webkit.org/b/160119 media/media-controller-time-constant.html [ Timeout Pass ]
webkit.org/b/160119 media/media-fullscreen-inline.html [ Crash Pass ]
webkit.org/b/160119 media/restore-from-page-cache.html [ Timeout Pass Failure ]
webkit.org/b/160119 media/video-poster.html [ Crash Pass ]
webkit.org/b/160119 media/video-object-fit-change.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 media/track/audio/audio-track-mkv-vorbis-enabled.html [ Crash Pass ]

webkit.org/b/160119 media/track/track-css-matching-timestamps.html [ Crash Pass ]
webkit.org/b/160119 media/track/track-cues-seeking.html [ Timeout Pass ]
webkit.org/b/160119 media/track/track-cues-sorted-before-dispatch.html [ Timeout Pass ]
webkit.org/b/160119 media/track/track-delete-during-setup.html [ Timeout Pass ]

# This one has its own bug because the crashes were preexisting; only the
# timeout is caused by the switch to threaded compositor.
webkit.org/b/160131 media/track/track-in-band-duplicate-tracks-when-source-changes.html [ Crash Timeout Pass ]

webkit.org/b/160119 storage/indexeddb/index-cursor.html [ Timeout Pass ]
webkit.org/b/160119 storage/indexeddb/objectstore-cursor.html [ Timeout Pass ]

webkit.org/b/160119 svg/custom/symbol-zoom.html [ ImageOnlyFailure Pass ]

webkit.org/b/160119 svg/repaint/add-outline-property-on-root.html [ ImageOnlyFailure ]
webkit.org/b/160119 svg/repaint/remove-outline-property-on-root.html [ ImageOnlyFailure ]