[webkit-dev] Coming Soon: new-run-webkit-tests

[Eric Seidel]

I think it's fine to add some support for sampling "unexpected"
timeouts.  However if the timeout is expected in test_expectations, we
shouldn't bother to sample. :)

Then again, since new-run-webkit-tests is all parallel and stuff we
need not block for the sample like how the current run-webkit-tests
does.

-eric

On Sat, Apr 10, 2010 at 8:57 PM, Adam Barth <abarth at webkit.org> wrote:
> https://bugs.webkit.org/show_bug.cgi?id=37402
>
> On Sat, Apr 10, 2010 at 8:55 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>> The reason for sampling timeouts is so that you can diagnose what happened
>> more easily when you get an unexpected timeout. Same reason we save
>> backtraces from crashes. Having some expected timeouts does not remove the
>> need to diagnose regressions that manifest as a timeout.
>>
>>
>>
>> On Apr 10, 2010, at 8:39 PM, Eric Seidel <eric at webkit.org> wrote:
>>
>>> new-run-webkit-tests does not sample tests when they timeout.  timeout
>>> is a perfectly "reasonable" test expectation.  new-run-webkit-tests is
>>> about running all the tests and making sure their behavior matches
>>> what we have in test_expectations.txt.  This is unlike
>>> run-webkit-tests for which the only valid expectation is "PASS".
>>>
>>> new-run-webkit-tests makes sense in a world where you want to run all
>>> the tests, but have no prayer of passing them all.  I think most ports
>>> are in this boat.  Possibly even the Mac port these days.
>>>
>>> new-run-webkit-tests "always run every test, no matter its expected
>>> outcome" philosophy enables a bunch of neat fringe benefits, including
>>> documenting and tracking flaky tests w/o having them break your
>>> builders or requiring skipping. :)
>>>
>>> -eric
>>>
>>> On Sat, Apr 10, 2010 at 7:49 PM, Alexey Proskuryakov <ap at webkit.org>
>>> wrote:
>>>>
>>>> 10.04.2010, в 17:44, Eric Seidel написал(а):
>>>>
>>>>> Yes, I think we should keep Chromium's default low timeout.  Having
>>>>> such a low timeout allows new-run-webkit-tests to easily run all flaky
>>>>> tests every time, even ones which occasionally timeout.
>>>>
>>>>
>>>> Does new-run-webkit-tests not sample tests that time out? Sampling is so
>>>> slow that is easily outweighs the difference between 3 or 30 seconds that
>>>> a
>>>> test can run for.
>>>>
>>>> - WBR, Alexey Proskuryakov
>>>>
>>>>
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>