[webkit-dev] run-webkit-tests is moving to parallell testing by default (this weekend)

Mon Dec 5 13:53:52 PST 2011

On Mon, Dec 5, 2011 at 1:01 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> We never implemented the "general way of marking subdirectories as
> needing to run serially", but it would be easy to do if we needed to
> [the 'http' dirs are still special-cased in the code].

Does it also special-case the storage tests that may collide?  I'm
thinking particularly of the filesystem/filewriter tests, but anything
that deals with quota may also have issues.

> There is code now (landed a few months ago) to control how many http
> tests run in parallel separately from the main parallelism flag (so
> you can run 16 workers but only 2 http tests at a time). I implemented
> it but never flipped the switch because it was in the middle of Eric
> flipping the bots over to NRWT in the first place. We should try
> experimenting with this to see at some point once everything
> stabilizes otherwise (this is the max_locked_shards() call in
> manager.py; there's no command line flag).
>
> -- Dirk
>
> On Mon, Dec 5, 2011 at 11:17 AM, Ojan Vafai <ojan at chromium.org> wrote:
>> Why is that? I don't know about other ports, but AFAIK, chromium writes to a
>> mock clipboard and the Apple mac port writes to a local OS clipboard
>> instance instead of the global one, specifically to avoid copy/paste tests
>> interacting. Even without running tests in parallel, it's probably a good
>> idea in order to avoid copy-pastes the developer is doing from affecting the
>> test run.
>>
>> That said, I believe we have a general way of marking subdirectories as
>> needing to run serially (which is what we do for http) if there are other
>> reasons we need to.
>>
>> On Mon, Dec 5, 2011 at 11:12 AM, David Levin <levin at google.com> wrote:
>>>
>>> I believe there are some tests (copy/paste) that it would be very hard to
>>> fully shard due to how they work.
>>>
>>> dave
>>>
>>>
>>>
>>> On Mon, Dec 5, 2011 at 11:08 AM, Ojan Vafai <ojan at chromium.org> wrote:
>>>>
>>>> I looked at one example that didn't exit
>>>> early: http://build.webkit.org/builders/SnowLeopard%20Intel%20Release%20%28Tests%29/builds/35153/steps/layout-test/logs/stdio
>>>>
>>>> In that case, the http tests were the long tail and took 6 minutes longer
>>>> than all the other tests. We don't split the http tests up because every
>>>> time we've tried it's caused too much flakiness. It's unclear if the
>>>> flakiness points to a bug in the test harness (e.g. in how we setup apache)
>>>> or to bugs in the tests themselves or both. If someone has time to look into
>>>> this, this is probably the biggest benefit to be found in NRWT runtime when
>>>> running tests in parallel.
>>>>
>>>> FYI, NRWT outputs a log of the runtime after each run:
>>>>
>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/9:  4696
>>>> tests, 1746.63 secs
>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/8:  1177
>>>> tests, 1693.47 secs
>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/3:  1408
>>>> tests, 2033.51 secs
>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/2:   941
>>>> tests, 2119.65 secs
>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/1:  1121
>>>> tests, 2041.97 secs
>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/0:  1453
>>>> tests, 2515.75 secs
>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/7:  1189
>>>> tests, 1731.12 secs
>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/6:  3556
>>>> tests, 2114.37 secs
>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/5:   948
>>>> tests, 2097.13 secs
>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/4:  1411
>>>> tests, 1716.66 secs
>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/15:   795
>>>> tests, 2027.16 secs
>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/14:  1123
>>>> tests, 1732.72 secs
>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/13:   425
>>>> tests, 2021.25 secs
>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/12:  1175
>>>> tests, 1710.09 secs
>>>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO      worker/11:  3462
>>>> tests, 2096.30 secs
>>>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO      worker/10:  1449
>>>> tests, 1722.68 secs
>>>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO    31120.45
>>>> cumulative, 1945.03 optimal
>>>>
>>>> That shows you that, if we fully sharded all the tests, they would in
>>>> theory take 1945 seconds to run, but worker/0 (the worker that runs the http
>>>> tests) took 2515 seconds to run.
>>>>
>>>> Ojan
>>>>
>>>> On Mon, Dec 5, 2011 at 6:55 AM, Adam Roben <aroben at apple.com> wrote:
>>>>>
>>>>> On Dec 2, 2011, at 6:55 PM, Eric Seidel wrote:
>>>>>
>>>>> > The SnowLeopard bot went from a 1 hr 4 min (!?!) cycle time, to 38 min
>>>>> > (still !?!).
>>>>>
>>>>> I suspect our Mac test bots could use a dose of RAM. Many of them only
>>>>> have 3GB, since when you're running tests one by one you don't really need
>>>>> much more.
>>>>>
>>>>> -Adam
>>>>>
>>>>> _______________________________________________
>>>>> webkit-dev mailing list
>>>>> webkit-dev at lists.webkit.org
>>>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> webkit-dev mailing list
>>>> webkit-dev at lists.webkit.org
>>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>>
>>>
>>
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>