[webkit-dev] Coming Soon: new-run-webkit-tests

[Ojan Vafai]

Recording to a local file the order in which tests are run on each thread
should be relatively trivial. I filed
https://bugs.webkit.org/show_bug.cgi?id=37396. That will help with flakiness
that is simply order dependent, which I think is most of it. It won't help
with flakiness that has to do with contention for other resources (e.g. ipv6
lookups as in https://bugs.webkit.org/show_bug.cgi?id=37104).

Ojan

On Fri, Apr 9, 2010 at 9:34 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> On Fri, Apr 9, 2010 at 4:43 PM, Brady Eidson <beidson at apple.com> wrote:
> >
> > On Apr 9, 2010, at 4:41 PM, Alexey Proskuryakov wrote:
> >
> >>
> >> On 09.04.2010, at 16:32, Brady Eidson wrote:
> >>
> >>>> - Exposes more flaky tests due to running tests in a non-deterministic
> order.
> >>>
> >>> This sounds like a pro to me.
> >>
> >>
> >> Existing run-webkit-tests can also do that, via a non-default option.
> Unreproducible results with default options do sound like a con.
> >
> > Don't get me wrong - if a test fails reliably due to a specific run
> order, then the tool needs to be able to record the run-order so the failure
> can be explored.
> >
> > But finding more flakey tests by mixing up the run order is - in
> principle - a good thing.
> >
> > I'd forgotten that the current tool can mixup the run-order.
> >
> > We should make the new tool record the order so order-specific failures
> can be explored.
> >
>
> There are two sources of non-determinism.
>
> One, which is easy to deal with, is the result of shuffling the
> testing order. The new script supports this but doesn't use it by
> default.
>
> The other is the result of timing and concurrency issues, and is not
> so easily dealt with. The default code path shards by directory, and
> as a result it is reasonably predictable which tests will run on which
> thread. However, even in this situation you can get contention on the
> filesystem and (much more common) the web server that can produce
> timeouts and other unpredictable events. The
> --experimental-fully-parallel code path makes the situation worse by
> not even being able to easily say that any two tests will or won't be
> executed sequentially by the same thread (the code uses a threadsafe
> Python Queue object with N consumers). We also (I think) see a fair
> amount of contention on the queue itself that contributes to timing
> issues.
>
> -- Dirk
>
> > ~Brady
> >
> >> - WBR, Alexey Proskuryakov
> >>
> >>
> >
> >
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100410/4bd7e518/attachment.html>