[webkit-dev] run-webkit-tests is moving to parallell testing by default (this weekend)

[David Levin]

I believe there are some tests (copy/paste) that it would be very hard to
fully shard due to how they work.

dave


On Mon, Dec 5, 2011 at 11:08 AM, Ojan Vafai <ojan at chromium.org> wrote:

> I looked at one example that didn't exit early:
> http://build.webkit.org/builders/SnowLeopard%20Intel%20Release%20%28Tests%29/builds/35153/steps/layout-test/logs/stdio
>
> In that case, the http tests were the long tail and took 6 minutes longer
> than all the other tests. We don't split the http tests up because every
> time we've tried it's caused too much flakiness. It's unclear if the
> flakiness points to a bug in the test harness (e.g. in how we setup apache)
> or to bugs in the tests themselves or both. If someone has time to look
> into this, this is probably the biggest benefit to be found in NRWT runtime
> when running tests in parallel.
>
> FYI, NRWT outputs a log of the runtime after each run:
>
> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/9:  4696 tests, 1746.63 secs
> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/8:  1177 tests, 1693.47 secs
> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/3:  1408 tests, 2033.51 secs
> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/2:   941 tests, 2119.65 secs
> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/1:  1121 tests, 2041.97 secs
> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/0:  1453 tests, 2515.75 secs
> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/7:  1189 tests, 1731.12 secs
> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/6:  3556 tests, 2114.37 secs
> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/5:   948 tests, 2097.13 secs
> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/4:  1411 tests, 1716.66 secs
> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/15:   795 tests, 2027.16 secs
> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/14:  1123 tests, 1732.72 secs
> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/13:   425 tests, 2021.25 secs
> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/12:  1175 tests, 1710.09 secs
> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO      worker/11:  3462 tests, 2096.30 secs
> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO      worker/10:  1449 tests, 1722.68 secs
> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO    31120.45 cumulative, 1945.03 optimal
>
> That shows you that, if we fully sharded all the tests, they would in
> theory take 1945 seconds to run, but worker/0 (the worker that runs the
> http tests) took 2515 seconds to run.
>
> Ojan
>
> On Mon, Dec 5, 2011 at 6:55 AM, Adam Roben <aroben at apple.com> wrote:
>
>> On Dec 2, 2011, at 6:55 PM, Eric Seidel wrote:
>>
>> > The SnowLeopard bot went from a 1 hr 4 min (!?!) cycle time, to 38 min
>> (still !?!).
>>
>> I suspect our Mac test bots could use a dose of RAM. Many of them only
>> have 3GB, since when you're running tests one by one you don't really need
>> much more.
>>
>> -Adam
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20111205/80097aec/attachment.html>