[webkit-dev] Coming Soon: new-run-webkit-tests

[Adam Barth]

https://bugs.webkit.org/show_bug.cgi?id=37402

On Sat, Apr 10, 2010 at 8:55 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> The reason for sampling timeouts is so that you can diagnose what happened
> more easily when you get an unexpected timeout. Same reason we save
> backtraces from crashes. Having some expected timeouts does not remove the
> need to diagnose regressions that manifest as a timeout.
>
>
>
> On Apr 10, 2010, at 8:39 PM, Eric Seidel <eric at webkit.org> wrote:
>
>> new-run-webkit-tests does not sample tests when they timeout.  timeout
>> is a perfectly "reasonable" test expectation.  new-run-webkit-tests is
>> about running all the tests and making sure their behavior matches
>> what we have in test_expectations.txt.  This is unlike
>> run-webkit-tests for which the only valid expectation is "PASS".
>>
>> new-run-webkit-tests makes sense in a world where you want to run all
>> the tests, but have no prayer of passing them all.  I think most ports
>> are in this boat.  Possibly even the Mac port these days.
>>
>> new-run-webkit-tests "always run every test, no matter its expected
>> outcome" philosophy enables a bunch of neat fringe benefits, including
>> documenting and tracking flaky tests w/o having them break your
>> builders or requiring skipping. :)
>>
>> -eric
>>
>> On Sat, Apr 10, 2010 at 7:49 PM, Alexey Proskuryakov <ap at webkit.org>
>> wrote:
>>>
>>> 10.04.2010, в 17:44, Eric Seidel написал(а):
>>>
>>>> Yes, I think we should keep Chromium's default low timeout.  Having
>>>> such a low timeout allows new-run-webkit-tests to easily run all flaky
>>>> tests every time, even ones which occasionally timeout.
>>>
>>>
>>> Does new-run-webkit-tests not sample tests that time out? Sampling is so
>>> slow that is easily outweighs the difference between 3 or 30 seconds that
>>> a
>>> test can run for.
>>>
>>> - WBR, Alexey Proskuryakov
>>>
>>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>