[Webkit-unassigned] [Bug 86889] New: should we get rid of the SLOW modifier in test_expectations.txt?

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=86889

           Summary: should we get rid of the SLOW modifier in
                    test_expectations.txt?
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: dpranke at chromium.org
                CC: mjs at apple.com, tony at chromium.org, ojan at chromium.org,
                    rniwa at webkit.org


As part of the discussion on webkit-dev in the thread with https://lists.webkit.org/pipermail/webkit-dev/2012-May/020674.html and subsequent discussion on bug 86796, there is a question as to whether we should move the SLOW modifier to the right side of the expectation line, whether its semantics should change, and/or whether we should get rid of it entirely.

1) Suppose you have an expectation with SLOW by itself: bug(dpranke) : foo/bar.html = SLOW

presumably that should imply that the test is expected to pass, but it just takes a while to execute (in the current format, you would have SLOW : foo/bar.html = PASS, which is explicit).

2) Suppose you have SLOW plus an additional outcome keyword: bug(dpranke) : foo/bar.html = SLOW IMAGE

presumably that means that the test is expected to produce a pixel test failure and take a while to execute. It does not represent "either pass slowly or fail the pixel test quickly", nor "run slowly but either pass or fail the pixel test", i.e., this does not make the test flaky. (Of course, we could change the interpretation here)

3) Should it be a warning similar to "unexpected pass" if a test marked SLOW starts to complete quickly? Seems reasonable ...

4) Can we get away with only using SLOW for "SLOW PASS", and say that SLOW failures should not be allowed (since we want to move to a model where any test that fails for longer than a couple hours should get a new expectation/baseline checked in)?

5) Can/should we just get rid of SLOW altogether? If we did this, we would have to adjust the default timeout value for chromium upwards, and we would want to consider what the impact on bot cycle time should be

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.