[webkit-dev] Coming Soon: new-run-webkit-tests

Eric Seidel eric at webkit.org
Sat Apr 10 20:39:55 PDT 2010

new-run-webkit-tests does not sample tests when they timeout.  timeout
is a perfectly "reasonable" test expectation.  new-run-webkit-tests is
about running all the tests and making sure their behavior matches
what we have in test_expectations.txt.  This is unlike
run-webkit-tests for which the only valid expectation is "PASS".

new-run-webkit-tests makes sense in a world where you want to run all
the tests, but have no prayer of passing them all.  I think most ports
are in this boat.  Possibly even the Mac port these days.

new-run-webkit-tests "always run every test, no matter its expected
outcome" philosophy enables a bunch of neat fringe benefits, including
documenting and tracking flaky tests w/o having them break your
builders or requiring skipping. :)


On Sat, Apr 10, 2010 at 7:49 PM, Alexey Proskuryakov <ap at webkit.org> wrote:
> 10.04.2010, в 17:44, Eric Seidel написал(а):
>> Yes, I think we should keep Chromium's default low timeout.  Having
>> such a low timeout allows new-run-webkit-tests to easily run all flaky
>> tests every time, even ones which occasionally timeout.
> Does new-run-webkit-tests not sample tests that time out? Sampling is so
> slow that is easily outweighs the difference between 3 or 30 seconds that a
> test can run for.
> - WBR, Alexey Proskuryakov

More information about the webkit-dev mailing list