[webkit-dev] run-webkit-tests is moving to parallell testing by default (this weekend)

Dirk Pranke dpranke at chromium.org
Mon Dec 5 14:37:02 PST 2011 

On Mon, Dec 5, 2011 at 1:53 PM, Eric U <ericu at google.com> wrote:
> On Mon, Dec 5, 2011 at 1:01 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>> We never implemented the "general way of marking subdirectories as
>> needing to run serially", but it would be easy to do if we needed to
>> [the 'http' dirs are still special-cased in the code].
>
> Does it also special-case the storage tests that may collide?  I'm
> thinking particularly of the filesystem/filewriter tests, but anything
> that deals with quota may also have issues.
>

It does not, although it would be easy to add such support. I will
also note that by default all of the tests in the same directory are
run sequentially in a single thread, so that may be why it hasn't been
much of an issue.

-- Dirk

>> There is code now (landed a few months ago) to control how many http
>> tests run in parallel separately from the main parallelism flag (so
>> you can run 16 workers but only 2 http tests at a time). I implemented
>> it but never flipped the switch because it was in the middle of Eric
>> flipping the bots over to NRWT in the first place. We should try
>> experimenting with this to see at some point once everything
>> stabilizes otherwise (this is the max_locked_shards() call in
>> manager.py; there's no command line flag).
>>
>> -- Dirk
>>
>> On Mon, Dec 5, 2011 at 11:17 AM, Ojan Vafai <ojan at chromium.org> wrote:
>>> Why is that? I don't know about other ports, but AFAIK, chromium writes to a
>>> mock clipboard and the Apple mac port writes to a local OS clipboard
>>> instance instead of the global one, specifically to avoid copy/paste tests
>>> interacting. Even without running tests in parallel, it's probably a good
>>> idea in order to avoid copy-pastes the developer is doing from affecting the
>>> test run.
>>>
>>> That said, I believe we have a general way of marking subdirectories as
>>> needing to run serially (which is what we do for http) if there are other
>>> reasons we need to.
>>>
>>> On Mon, Dec 5, 2011 at 11:12 AM, David Levin <levin at google.com> wrote:
>>>>
>>>> I believe there are some tests (copy/paste) that it would be very hard to
>>>> fully shard due to how they work.
>>>>
>>>> dave
>>>>
>>>>
>>>>
>>>> On Mon, Dec 5, 2011 at 11:08 AM, Ojan Vafai <ojan at chromium.org> wrote:
>>>>>
>>>>> I looked at one example that didn't exit
>>>>> early: http://build.webkit.org/builders/SnowLeopard%20Intel%20Release%20%28Tests%29/builds/35153/steps/layout-test/logs/stdio
>>>>>
>>>>> In that case, the http tests were the long tail and took 6 minutes longer
>>>>> than all the other tests. We don't split the http tests up because every
>>>>> time we've tried it's caused too much flakiness. It's unclear if the
>>>>> flakiness points to a bug in the test harness (e.g. in how we setup apache)
>>>>> or to bugs in the tests themselves or both. If someone has time to look into
>>>>> this, this is probably the biggest benefit to be found in NRWT runtime when
>>>>> running tests in parallel.
>>>>>
>>>>> FYI, NRWT outputs a log of the runtime after each run:
>>>>>
>>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/9:  4696
>>>>> tests, 1746.63 secs
>>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/8:  1177
>>>>> tests, 1693.47 secs
>>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/3:  1408
>>>>> tests, 2033.51 secs
>>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/2:   941
>>>>> tests, 2119.65 secs
>>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/1:  1121
>>>>> tests, 2041.97 secs
>>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/0:  1453
>>>>> tests, 2515.75 secs
>>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/7:  1189
>>>>> tests, 1731.12 secs
>>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/6:  3556
>>>>> tests, 2114.37 secs
>>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/5:   948
>>>>> tests, 2097.13 secs
>>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/4:  1411
>>>>> tests, 1716.66 secs
>>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/15:   795
>>>>> tests, 2027.16 secs
>>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/14:  1123
>>>>> tests, 1732.72 secs
>>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/13:   425
>>>>> tests, 2021.25 secs
>>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/12:  1175
>>>>> tests, 1710.09 secs
>>>>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO      worker/11:  3462
>>>>> tests, 2096.30 secs
>>>>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO      worker/10:  1449
>>>>> tests, 1722.68 secs
>>>>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO    31120.45
>>>>> cumulative, 1945.03 optimal
>>>>>
>>>>> That shows you that, if we fully sharded all the tests, they would in
>>>>> theory take 1945 seconds to run, but worker/0 (the worker that runs the http
>>>>> tests) took 2515 seconds to run.
>>>>>
>>>>> Ojan
>>>>>
>>>>> On Mon, Dec 5, 2011 at 6:55 AM, Adam Roben <aroben at apple.com> wrote:
>>>>>>
>>>>>> On Dec 2, 2011, at 6:55 PM, Eric Seidel wrote:
>>>>>>
>>>>>> > The SnowLeopard bot went from a 1 hr 4 min (!?!) cycle time, to 38 min
>>>>>> > (still !?!).
>>>>>>
>>>>>> I suspect our Mac test bots could use a dose of RAM. Many of them only
>>>>>> have 3GB, since when you're running tests one by one you don't really need
>>>>>> much more.
>>>>>>
>>>>>> -Adam
>>>>>>
>>>>>> _______________________________________________
>>>>>> webkit-dev mailing list
>>>>>> webkit-dev at lists.webkit.org
>>>>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> webkit-dev mailing list
>>>>> webkit-dev at lists.webkit.org
>>>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>

More information about the webkit-dev mailing list