[Webkit-unassigned] [Bug 127095] run-webkit-tests should support assert-only js tests w/o expected.txt files

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=127095

--- Comment #11 from James Graham <james at hoppipolla.co.uk> ---
FWIW the way that Firefox deals with this is:

* Every test that has an expectation of some sort other than PASS gets a corresponding .ini file with details about the expected result
* All tests must match the expectations in their ini file or PASS, otherwise it's treated as an error.
* When new tests are imported these ini files are autogenerated from a test run using a known-good version of the code.

I strongly disagree with the premise that all imported tests must be given detailed review before they are used. In practice it's highly unlikely that anyone will actually do this work, so as a policy that's exactly equivalent to "we don't run W3C tests". It's also highly unlikely that such a review would catch a meaningful number of issues compared to the cost of doing it. With tests the most important consideration is "is this stable", which reviewers are rather bad at noticing, unless it's doing some very obviously bad things like using setTimeout with short timeouts. Other issues e.g. incorrectness are much easier to spot once the test is actually running.

I agree that the above policy theoretically allows bugs like "test X is expected to run but actually doesn't run". However in practice these are rare, so I am not that concerned. You can of course do better there if you are really worried about it.

I strongly urge that you adopt a process that allows almost-fully automatic imports of the tests from upstream. In Firefox the whole process looks something like:

./mach web-platform-tests-update --sync
git push try
fetchlogs.py try [sha1]
./mach web-platform-tests-update *.log
git rebase -i inbound/tip
git push

That has a couple of manual steps that prevent full automation, but updates are nevertheless trivial and as a consequence we are able to run 95% of the tests with a update latency of about a week.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-unassigned/attachments/20150731/2b365ad0/attachment.html>