[Webkit-unassigned] [Bug 127095] run-webkit-tests should support assert-only js tests w/o expected.txt files

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=127095





--- Comment #2 from youenn fablet <youennf at gmail.com>  2014-01-18 13:38:47 PST ---
(In reply to comment #1)
> I don't think we should do this.  Even if an explicit FAIL wasn't there or all of PASS were present, it's possible for a test to spit out an erroneous line or not run a part of the test.  Having -expected.txt catches these bugs.

I would restrict this approach to testharness.js based tests.
Probably to imported W3C tests initially, this is where it is mostly valuable.

This approach would limit the number of tests to manually check.
If this number is limited, the manual checking(and related expected.txt files) would probably be more reliable.

For those tests, the matching pattern would be something like:
- empty line(s)
- line(s) starting with PASS
- empty line(s)
Any test result that does not match this pattern would require a matching -expected file to pass. This should catch any output that contains spurious line.

To handle the case  of testharness.js based tests that would only partially run, the situation is less clear.
For existing webkit tests, we already have -expected.txt files.
For new webkit tests, a parameter could be added to identify the number of expected tests.
For existing W3C tests, I do not know what can be done on that. Manual checking may not be the best solution anyway. And more tests (even if partially run) seems better than fewer tests.

I would restrict this strategy to imported/w3c tests initially.
If that proves to work, we may envision to apply it to other testharness.js layout tests.
To identify which tests are testharness.js based, testharnessreport.js could be tweaked to ouput a line stating something like "testharness.js based test: ..."

Would that address your concerns?

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.