[Webkit-unassigned] [Bug 127095] run-webkit-tests should support assert-only js tests w/o expected.txt files

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=127095





--- Comment #5 from Ryosuke Niwa <rniwa at webkit.org>  2014-01-18 16:28:10 PST ---
(In reply to comment #4)
> > > To handle the case  of testharness.js based tests that would only partially run, the situation is less clear.
> > > For existing webkit tests, we already have -expected.txt files.
> > > For new webkit tests, a parameter could be added to identify the number of expected tests.
> > > For existing W3C tests, I do not know what can be done on that. Manual checking may not be the best solution anyway. And more tests (even if partially run) seems better than fewer tests.
> > 
> > Checking in -expected.txt doesn't mean we can import less tests.  Also, we can import tests with FAIL expectations as well.  That's how we've been importing tests.
> 
> Agreed on FAIL expectations. -expected files should be kept for sure.
> 
> Checking in -expected.txt files does not guarantee that all tests are actually run though.
> To guarantee that, one needs to carefully read each test.
> This obviously takes time, hence less imported tests at the end of the day.

We need to do this regardless.  Each test we import needs to be manually reviewed like any other test being added to WebKit.

> When importing tests, I would prefer adding the number of expected tests directly in the HTML file compared to counting the number of PASS in the expected file.

That sounds like an improvement.

> First, tests are more often read than expected files, giving more chances to catch errors in the number of tests.

I disagree. Whenever a test fails, what I'd like to see is the -expected.txt diff.  I don't want to manually examine the test to figure out what might be failing.

> Second, this may be pushed to the W3C repository.

Adding the number of test cases in each test would definitely need to be done in the W3C repository, not in WebKit.

> Would you still object to the approach if the checking of the number of tests is made mandatory?

I still don't think it's a good idea.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.