[webkit-dev] Coming Soon: new-run-webkit-tests

[Maciej Stachowiak]

The reason for sampling timeouts is so that you can diagnose what  
happened more easily when you get an unexpected timeout. Same reason  
we save backtraces from crashes. Having some expected timeouts does  
not remove the need to diagnose regressions that manifest as a timeout.



On Apr 10, 2010, at 8:39 PM, Eric Seidel <eric at webkit.org> wrote:

> new-run-webkit-tests does not sample tests when they timeout.  timeout
> is a perfectly "reasonable" test expectation.  new-run-webkit-tests is
> about running all the tests and making sure their behavior matches
> what we have in test_expectations.txt.  This is unlike
> run-webkit-tests for which the only valid expectation is "PASS".
>
> new-run-webkit-tests makes sense in a world where you want to run all
> the tests, but have no prayer of passing them all.  I think most ports
> are in this boat.  Possibly even the Mac port these days.
>
> new-run-webkit-tests "always run every test, no matter its expected
> outcome" philosophy enables a bunch of neat fringe benefits, including
> documenting and tracking flaky tests w/o having them break your
> builders or requiring skipping. :)
>
> -eric
>
> On Sat, Apr 10, 2010 at 7:49 PM, Alexey Proskuryakov <ap at webkit.org>  
> wrote:
>>
>> 10.04.2010, в 17:44, Eric Seidel написал(а):
>>
>>> Yes, I think we should keep Chromium's default low timeout.  Having
>>> such a low timeout allows new-run-webkit-tests to easily run all  
>>> flaky
>>> tests every time, even ones which occasionally timeout.
>>
>>
>> Does new-run-webkit-tests not sample tests that time out? Sampling  
>> is so
>> slow that is easily outweighs the difference between 3 or 30  
>> seconds that a
>> test can run for.
>>
>> - WBR, Alexey Proskuryakov
>>
>>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev