[Webkit-unassigned] [Bug 127095] run-webkit-tests should support assert-only js tests w/o expected.txt files

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=127095





--- Comment #3 from Ryosuke Niwa <rniwa at webkit.org>  2014-01-18 14:08:58 PST ---
(In reply to comment #2)
> (In reply to comment #1)
> > I don't think we should do this.  Even if an explicit FAIL wasn't there or all of PASS were present, it's possible for a test to spit out an erroneous line or not run a part of the test.  Having -expected.txt catches these bugs.
> 
> I would restrict this approach to testharness.js based tests.
> Probably to imported W3C tests initially, this is where it is mostly valuable.
> 
> This approach would limit the number of tests to manually check.
> If this number is limited, the manual checking(and related expected.txt files) would probably be more reliable.
> 
> For those tests, the matching pattern would be something like:
> - empty line(s)
> - line(s) starting with PASS
> - empty line(s)
> Any test result that does not match this pattern would require a matching -expected file to pass. This should catch any output that contains spurious line.

That doesn't catch the case where one of the test cases didn't get to run.

> To handle the case  of testharness.js based tests that would only partially run, the situation is less clear.
> For existing webkit tests, we already have -expected.txt files.
> For new webkit tests, a parameter could be added to identify the number of expected tests.
> For existing W3C tests, I do not know what can be done on that. Manual checking may not be the best solution anyway. And more tests (even if partially run) seems better than fewer tests.

Checking in -expected.txt doesn't mean we can import less tests.  Also, we can import tests with FAIL expectations as well.  That's how we've been importing tests.

> I would restrict this strategy to imported/w3c tests initially.
> If that proves to work, we may envision to apply it to other testharness.js layout tests.
> To identify which tests are testharness.js based, testharnessreport.js could be tweaked to output a line stating something like "testharness.js based test: ..."

I still strongly object to this approach.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.