[Webkit-unassigned] [Bug 127095] New: run-webkit-tests should support assert-only js tests w/o expected.txt files

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=127095

           Summary: run-webkit-tests should support assert-only js tests
                    w/o expected.txt files
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: youennf at gmail.com


Consider merging Blink patches from issue 268711: https://code.google.com/p/chromium/issues/detail?id=268711
Issue description:
//////////////////////////////////////////
This comes up in the context of being able to import and run the W3C's tests, but is applicable to the existing tests in Blink as well.

Many of our JS-based (text-only) tests produce output of the form of lines starting with either "PASS" or "FAIL" (in addition to informational output). 

Programmatically, one can determine if a test is passing by simply asserting that there are no lines that start with "FAIL" in the output. In this case, the -expected.txt is kind of unnecessary. 

In the W3C tests' cases, -expected.txt files don't exist, which means that we need to generate and synthesize false baselines (and then review them) as part of the import. There's > 6000 JS-based tests in the W3C repos, so this is unfortunate :).

I think we should just modify run-webkit-tests to treat a test as passing if an -expected.txt file doesn't exist and the text output isn't a render tree, contains at least one "PASS", and doesn't contain any "FAIL" or "TIMEOUT" lines.

Note that the downside to such an approach is that you can't easily track partial failures (where you pass some assertions but fail others), but we can continue to use -expected.txt for those cases.

Note that of the existing ~21,000 text-only tests in blink, about 9200 of them are tests that contain one or more passing assertions and zero failing assertions. In other words, giving the above definition, we could delete ~9200 expected.txt's and not really miss anything.

Anyone object (to adding support, not deleting the existing -expected.txt, that's a separate issue)?

I've posted a patch indicating roughly how this would work here:

https://codereview.chromium.org/22352002/
//////////////////////////////////////////

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.