[webkit-dev] Coming Soon: new-run-webkit-tests

Eric Seidel eric at webkit.org
Sat Apr 10 21:07:08 PDT 2010

I think it's fine to add some support for sampling "unexpected"
timeouts.  However if the timeout is expected in test_expectations, we
shouldn't bother to sample. :)

Then again, since new-run-webkit-tests is all parallel and stuff we
need not block for the sample like how the current run-webkit-tests


On Sat, Apr 10, 2010 at 8:57 PM, Adam Barth <abarth at webkit.org> wrote:
> https://bugs.webkit.org/show_bug.cgi?id=37402
> On Sat, Apr 10, 2010 at 8:55 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>> The reason for sampling timeouts is so that you can diagnose what happened
>> more easily when you get an unexpected timeout. Same reason we save
>> backtraces from crashes. Having some expected timeouts does not remove the
>> need to diagnose regressions that manifest as a timeout.
>> On Apr 10, 2010, at 8:39 PM, Eric Seidel <eric at webkit.org> wrote:
>>> new-run-webkit-tests does not sample tests when they timeout.  timeout
>>> is a perfectly "reasonable" test expectation.  new-run-webkit-tests is
>>> about running all the tests and making sure their behavior matches
>>> what we have in test_expectations.txt.  This is unlike
>>> run-webkit-tests for which the only valid expectation is "PASS".
>>> new-run-webkit-tests makes sense in a world where you want to run all
>>> the tests, but have no prayer of passing them all.  I think most ports
>>> are in this boat.  Possibly even the Mac port these days.
>>> new-run-webkit-tests "always run every test, no matter its expected
>>> outcome" philosophy enables a bunch of neat fringe benefits, including
>>> documenting and tracking flaky tests w/o having them break your
>>> builders or requiring skipping. :)
>>> -eric
>>> On Sat, Apr 10, 2010 at 7:49 PM, Alexey Proskuryakov <ap at webkit.org>
>>> wrote:
>>>> 10.04.2010, в 17:44, Eric Seidel написал(а):
>>>>> Yes, I think we should keep Chromium's default low timeout.  Having
>>>>> such a low timeout allows new-run-webkit-tests to easily run all flaky
>>>>> tests every time, even ones which occasionally timeout.
>>>> Does new-run-webkit-tests not sample tests that time out? Sampling is so
>>>> slow that is easily outweighs the difference between 3 or 30 seconds that
>>>> a
>>>> test can run for.
>>>> - WBR, Alexey Proskuryakov
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

More information about the webkit-dev mailing list