[webkit-dev] Coming Soon: new-run-webkit-tests

[Eric Seidel]

new-run-webkit-tests does not sample tests when they timeout.  timeout
is a perfectly "reasonable" test expectation.  new-run-webkit-tests is
about running all the tests and making sure their behavior matches
what we have in test_expectations.txt.  This is unlike
run-webkit-tests for which the only valid expectation is "PASS".

new-run-webkit-tests makes sense in a world where you want to run all
the tests, but have no prayer of passing them all.  I think most ports
are in this boat.  Possibly even the Mac port these days.

new-run-webkit-tests "always run every test, no matter its expected
outcome" philosophy enables a bunch of neat fringe benefits, including
documenting and tracking flaky tests w/o having them break your
builders or requiring skipping. :)

-eric

On Sat, Apr 10, 2010 at 7:49 PM, Alexey Proskuryakov <ap at webkit.org> wrote:
>
> 10.04.2010, в 17:44, Eric Seidel написал(а):
>
>> Yes, I think we should keep Chromium's default low timeout.  Having
>> such a low timeout allows new-run-webkit-tests to easily run all flaky
>> tests every time, even ones which occasionally timeout.
>
>
> Does new-run-webkit-tests not sample tests that time out? Sampling is so
> slow that is easily outweighs the difference between 3 or 30 seconds that a
> test can run for.
>
> - WBR, Alexey Proskuryakov
>
>