[Webkit-unassigned] [Bug 90121] New: nrwt should support "-failing" baselines/expectations

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=90121

           Summary: nrwt should support "-failing" baselines/expectations
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Keywords: NRWT
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: dpranke at chromium.org
                CC: mjs at apple.com, darin at apple.com, tony at chromium.org,
                    abarth at webkit.org, simon.fraser at apple.com,
                    ojan at chromium.org, rniwa at webkit.org, jacobg at adobe.com


currently there's no good way to distinguish between a baseline file that is believed to produce the "correct" result and a baseline file that is believed to be the "current" (but incorrect) result. It would be nice if we could tell which was which; this is a part of the reason the chromium port has historically suppressed so many test failures.

Of course, we also have a large legacy of existing baselines that could be in either state, so we should come up with some way to accomodate that as well. This has two implications: "-failing" should probably be optional, and there should ideally be some way of distinguishing "-expected correct" from "-expected current but correctness unknown".

We would also need a way to deal with the idea that "-expected correct" is no longer current, i.e., the test changes in such a way that the "-expected" file would need to be updated.

Frankly, it's not clear if the overhead of trying to track this is worth it, but it seems like it might be good to try incrementally (e.g., on a few directories of tests on only a subset of the ports).

Anyone else have thoughts, or proposals for how to implement this?

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.