[Webkit-unassigned] [Bug 72080] New: Layout tests marked as SKIP should not be overridable

bugzilla-daemon at webkit.org bugzilla-daemon at webkit.org
Thu Nov 10 18:04:46 PST 2011


https://bugs.webkit.org/show_bug.cgi?id=72080

           Summary: Layout tests marked as SKIP should not be overridable
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: scherkus at chromium.org
                CC: tony at chromium.org, ojan at chromium.org,
                    dpranke at chromium.org


Split off from https://bugs.webkit.org/show_bug.cgi?id=72001

"""
One possible option would be to make any directories marked as SKIP override expected results for specific files inside the directory; I am not aware of any case in practice where we want to skip all but X files in a dir, and if we did, we could support that by requiring the user to explicitly list all of the tests to skip.
"""

The canonical example is with GPU tests. Most of the layout tests don't need to run so there are expectations such as:
WONTFIX SKIP GPU GPU-CG LEOPARD : media = PASS TIMEOUT FAIL

However individual tests might start running again if a more specific expectation is added later:
BUGWK70748 GPU : media/audio-controls-rendering.html = IMAGE

This leads to increased test cycle time and generates noise on the flakiness dashboard as tests that were previously not generating results are now showing up with results.

As discussed in bug 72001, the current way GPU tests handle this is by manually adding/removing directories in Tools/Scripts/webkitpy/layout_tests/port/chromium_gpu.py

The chromium_gpu.py method isn't ideal either, and perhaps the non-overridable SKIP isn't either so perhaps we can think of something better here.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.



More information about the webkit-unassigned mailing list