[webkit-dev] Coming Soon: new-run-webkit-tests

[Dirk Pranke]

On Fri, Apr 9, 2010 at 4:43 PM, Brady Eidson <beidson at apple.com> wrote:
>
> On Apr 9, 2010, at 4:41 PM, Alexey Proskuryakov wrote:
>
>>
>> On 09.04.2010, at 16:32, Brady Eidson wrote:
>>
>>>> - Exposes more flaky tests due to running tests in a non-deterministic order.
>>>
>>> This sounds like a pro to me.
>>
>>
>> Existing run-webkit-tests can also do that, via a non-default option. Unreproducible results with default options do sound like a con.
>
> Don't get me wrong - if a test fails reliably due to a specific run order, then the tool needs to be able to record the run-order so the failure can be explored.
>
> But finding more flakey tests by mixing up the run order is - in principle - a good thing.
>
> I'd forgotten that the current tool can mixup the run-order.
>
> We should make the new tool record the order so order-specific failures can be explored.
>

There are two sources of non-determinism.

One, which is easy to deal with, is the result of shuffling the
testing order. The new script supports this but doesn't use it by
default.

The other is the result of timing and concurrency issues, and is not
so easily dealt with. The default code path shards by directory, and
as a result it is reasonably predictable which tests will run on which
thread. However, even in this situation you can get contention on the
filesystem and (much more common) the web server that can produce
timeouts and other unpredictable events. The
--experimental-fully-parallel code path makes the situation worse by
not even being able to easily say that any two tests will or won't be
executed sequentially by the same thread (the code uses a threadsafe
Python Queue object with N consumers). We also (I think) see a fair
amount of contention on the queue itself that contributes to timing
issues.

-- Dirk

> ~Brady
>
>> - WBR, Alexey Proskuryakov
>>
>>
>
>