[Webkit-unassigned] [Bug 72080] New: Layout tests marked as SKIP should not be overridable
bugzilla-daemon at webkit.org
bugzilla-daemon at webkit.org
Thu Nov 10 18:04:46 PST 2011
https://bugs.webkit.org/show_bug.cgi?id=72080
Summary: Layout tests marked as SKIP should not be overridable
Product: WebKit
Version: 528+ (Nightly build)
Platform: Unspecified
OS/Version: Unspecified
Status: NEW
Severity: Normal
Priority: P2
Component: Tools / Tests
AssignedTo: webkit-unassigned at lists.webkit.org
ReportedBy: scherkus at chromium.org
CC: tony at chromium.org, ojan at chromium.org,
dpranke at chromium.org
Split off from https://bugs.webkit.org/show_bug.cgi?id=72001
"""
One possible option would be to make any directories marked as SKIP override expected results for specific files inside the directory; I am not aware of any case in practice where we want to skip all but X files in a dir, and if we did, we could support that by requiring the user to explicitly list all of the tests to skip.
"""
The canonical example is with GPU tests. Most of the layout tests don't need to run so there are expectations such as:
WONTFIX SKIP GPU GPU-CG LEOPARD : media = PASS TIMEOUT FAIL
However individual tests might start running again if a more specific expectation is added later:
BUGWK70748 GPU : media/audio-controls-rendering.html = IMAGE
This leads to increased test cycle time and generates noise on the flakiness dashboard as tests that were previously not generating results are now showing up with results.
As discussed in bug 72001, the current way GPU tests handle this is by manually adding/removing directories in Tools/Scripts/webkitpy/layout_tests/port/chromium_gpu.py
The chromium_gpu.py method isn't ideal either, and perhaps the non-overridable SKIP isn't either so perhaps we can think of something better here.
--
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
More information about the webkit-unassigned
mailing list