[Webkit-unassigned] [Bug 266533] New: no-failure-limits should still run run-layout-tests-without-change

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=266533

            Bug ID: 266533
           Summary: no-failure-limits should still run
                    run-layout-tests-without-change
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: gsnedders at apple.com
                CC: aakash_jain at apple.com, ap at webkit.org,
                    jbedard at apple.com

As it is, when doing analysis of results from no-failure-limits runs on EWS, there's no way to distinguish what's a pre-existing failure from what's a novel failure (though obviously we can't find flakiness, given a single run without retries). This means if you try to rebaseline based on the results with `./Tools/Scripts/update-test-expectations github-pr` or similar you'll rebaseline far too much. And we don't want any sort of automation to accidentally rebaseline too much. (Though obviously ideally we wouldn't have pre-existing failures in the tree!)

For example, after re-importing imported/w3c/web-platform-tests/dom (https://ews-build.webkit.org/#/builders/36/builds/27895 for mac-wk2) after running update-text-expectations:

gsnedders at gsnedders-marsha OS2 % git add 'LayoutTests/imported/w3c/web-platform-tests/dom' LayoutTests/platform/*/imported/w3c/web-platform-tests/dom 
gsnedders at gsnedders-marsha OS2 % git status
On branch wpt-dom-import
Your branch is up to date with 'fork/wpt-dom-import'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/events/Event-dispatch-single-activation-behavior-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/events/remove-all-listeners-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/events/scrolling/scrollend-event-fires-to-iframe-window-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/events/scrolling/scrollend-with-snap-on-fractional-offset-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/observable/tentative/observable-constructor.any-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/observable/tentative/observable-constructor.any.worker-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/observable/tentative/observable-constructor.window-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/observable/tentative/observable-event-target.any-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/observable/tentative/observable-event-target.any.worker-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/parts/basic-dom-part-declarative-brace-syntax-innerhtml.tentative-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/parts/basic-dom-part-declarative-brace-syntax.tentative-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/parts/basic-dom-part-declarative-pi-syntax.tentative-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/parts/basic-dom-part-objects.tentative-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/parts/dom-parts-parseparts-on-body.tentative-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/parts/dom-parts-parseparts-on-head.tentative-expected.txt
        new file:   LayoutTests/imported/w3c/web-platform-tests/dom/parts/dom-parts-parseparts-on-root.tentative-expected.txt
        new file:   LayoutTests/platform/gtk/imported/w3c/web-platform-tests/dom/events/Event-dispatch-single-activation-behavior-expected.txt
        new file:   LayoutTests/platform/wpe/imported/w3c/web-platform-tests/dom/events/Event-dispatch-single-activation-behavior-expected.txt

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   LayoutTests/compositing/layer-creation/scale-rotation-transition-overlap-expected.txt
        modified:   LayoutTests/http/tests/webgpu/webgpu/api/operation/rendering/stencil-expected.txt
        modified:   LayoutTests/http/tests/xmlhttprequest/resetting-timeout-to-zero-expected.txt
        modified:   LayoutTests/imported/w3c/web-platform-tests/html/canvas/element/fill-and-stroke-styles/2d.fillStyle.get.transparent-expected.txt
        modified:   LayoutTests/imported/w3c/web-platform-tests/mathml/relations/css-styling/padding-border-margin/margin-003-expected.txt
        modified:   LayoutTests/imported/w3c/web-platform-tests/permissions/permissions-garbage-collect.https-expected.txt
        modified:   LayoutTests/platform/gtk/editing/selection/triple-click-in-pre-expected.txt
        modified:   LayoutTests/platform/gtk/fast/forms/time/time-input-rendering-basic-expected.txt
        modified:   LayoutTests/platform/gtk/imported/w3c/web-platform-tests/accname/name/comp_host_language_label-expected.txt
        modified:   LayoutTests/platform/gtk/svg/repaint/text-repainting-after-modifying-transform-repaint-rects-expected.txt
        modified:   LayoutTests/platform/wpe/fast/table/text-field-baseline-expected.txt
        modified:   LayoutTests/platform/wpe/fast/text/softbank-emoji-expected.txt
        modified:   LayoutTests/platform/wpe/imported/w3c/web-platform-tests/css/css-cascade/all-prop-initial-xml-expected.txt
        deleted:    LayoutTests/platform/wpe/imported/w3c/web-platform-tests/css/css-masking/inheritance.sub-expected.txt
        modified:   LayoutTests/platform/wpe/imported/w3c/web-platform-tests/webxr/ar-module/xrDevice_requestSession_immersive-ar.https-expected.txt
        deleted:    LayoutTests/platform/wpe/imported/w3c/web-platform-tests/workers/modules/shared-worker-import-referrer-expected.txt
        modified:   LayoutTests/platform/wpe/svg/custom/invisible-text-after-scrolling-expected.txt
        modified:   LayoutTests/platform/wpe/svg/repaint/text-repainting-after-modifying-transform-repaint-rects-expected.txt
        modified:   LayoutTests/tiled-drawing/scrolling/scroll-snap/scroll-snap-mandatory-mainframe-vertical-then-horizontal-expected.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        LayoutTests/platform/gtk/compositing/layer-creation/scale-rotation-transition-overlap-expected.txt
        LayoutTests/platform/gtk/fast/events/monotonic-event-time-expected.txt
        LayoutTests/platform/gtk/fast/preloader/
        LayoutTests/platform/gtk/http/tests/resourceLoadStatistics/
        LayoutTests/platform/gtk/imported/w3c/web-platform-tests/css/css-align/blocks/align-content-block-003-expected.txt
        LayoutTests/platform/gtk/imported/w3c/web-platform-tests/css/css-break/
        LayoutTests/platform/gtk/imported/w3c/web-platform-tests/css/css-transitions/
        LayoutTests/platform/gtk/imported/w3c/web-platform-tests/editing/run/
        LayoutTests/platform/gtk/imported/w3c/web-platform-tests/html/cross-origin-opener-policy/
        LayoutTests/platform/gtk/imported/w3c/web-platform-tests/media-source/
        LayoutTests/platform/gtk/imported/w3c/web-platform-tests/webcodecs/
        LayoutTests/platform/gtk/media/modern-media-controls/
        LayoutTests/platform/gtk/perf/
        LayoutTests/platform/gtk/svg/repaint/text-repainting-after-modifying-container-transform-repaint-rects-expected.txt
        LayoutTests/platform/gtk/webrtc/
        LayoutTests/platform/ios-wk2/compositing/layer-creation/scale-rotation-transition-overlap-expected.txt
        LayoutTests/platform/ios-wk2/fast/forms/ios/
        LayoutTests/platform/ios-wk2/fast/scrolling/ios/scroll-anchoring-momentum-scroll-expected.txt
        LayoutTests/platform/ios-wk2/http/tests/xmlhttprequest/
        LayoutTests/platform/ios-wk2/platform/
        LayoutTests/platform/ios-wk2/requestidlecallback/
        LayoutTests/platform/mac-wk1/compositing/layer-creation/scale-rotation-transition-overlap-expected.txt
        LayoutTests/platform/mac-wk1/http/tests/cache/
        LayoutTests/platform/mac-wk1/http/tests/loading/main-resource-delegates-on-back-navigation-expected.txt
        LayoutTests/platform/mac-wk1/http/tests/media/
        LayoutTests/platform/mac-wk1/http/tests/websocket/tests/hybi/client-close-2-expected.txt
        LayoutTests/platform/mac-wk1/imported/w3c/web-platform-tests/html/semantics/interactive-elements/
        LayoutTests/platform/wpe/css3/scroll-snap/
        LayoutTests/platform/wpe/fast/canvas/webgl/tex-image-and-sub-image-2d-with-video-expected.txt
        LayoutTests/platform/wpe/fast/canvas/webgl/tex-image-and-sub-image-2d-with-video-rgb565-expected.txt
        LayoutTests/platform/wpe/fast/canvas/webgl/tex-image-and-sub-image-2d-with-video-rgba4444-expected.txt
        LayoutTests/platform/wpe/fast/canvas/webgl/tex-image-and-sub-image-2d-with-video-rgba5551-expected.txt
        LayoutTests/platform/wpe/fast/canvas/webgl/texImage2D-mse-flipY-false-expected.txt
        LayoutTests/platform/wpe/fast/canvas/webgl/texImage2D-mse-flipY-true-expected.txt
        LayoutTests/platform/wpe/fast/canvas/webgl/texImage2D-mse-in-dom-flipY-false-expected.txt
        LayoutTests/platform/wpe/fast/canvas/webgl/texImage2D-mse-in-dom-flipY-true-expected.txt
        LayoutTests/platform/wpe/fast/canvas/webgl/texImage2D-video-flipY-false-expected.txt
        LayoutTests/platform/wpe/fast/mediastream/MediaStream-video-element-displays-buffer-expected.txt
        LayoutTests/platform/wpe/fast/mediastream/mediastreamtrack-video-zoom-expected.txt
        LayoutTests/platform/wpe/fast/mediastream/resize-trim-expected.txt
        LayoutTests/platform/wpe/fast/preloader/
        LayoutTests/platform/wpe/fullscreen/exit-full-screen-video-crash-expected.txt
        LayoutTests/platform/wpe/http/wpt/
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/css/css-align/blocks/align-content-block-002-expected.txt
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/css/css-break/
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/css/css-transitions/
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/editing/run/
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/html/cross-origin-opener-policy/
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/html/dom/idlharness.https_include=HTML._-expected.txt
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/html/semantics/embedded-content/
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/html/semantics/selectors/pseudo-classes/inrange-outofrange-expected.txt
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/media-source/
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/preload/
        LayoutTests/platform/wpe/imported/w3c/web-platform-tests/webcodecs/
        LayoutTests/platform/wpe/media/media-source/
        LayoutTests/platform/wpe/scrollbars/
        LayoutTests/platform/wpe/svg/repaint/text-repainting-after-modifying-container-transform-repaint-rects-expected.txt
        LayoutTests/platform/wpe/webgl/
        LayoutTests/platform/wpe/webrtc/

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20231216/0d43639d/attachment-0001.htm>