[webkit-dev] run-webkit-tests is moving to parallell testing by default (this weekend)

Dirk Pranke dpranke at chromium.org
Mon Dec 5 15:17:15 PST 2011 

if "http server instance" == "apache child process", then no. We don't
do anything to particularly limit the number of apache children
running or bind them, but given that in the normal case there is only
ever one http test running at a time, you shouldn't see any issues
from contention.

(You would, of course, if we start running multiple http tests in
parallel, so that's something to keep in mind, especially if we shard
by-test instead of by-directory).

-- Dirk

On Mon, Dec 5, 2011 at 3:13 PM, Michael Nordman <michaeln at google.com> wrote:
> Some http tests make use of stateful php scritps with different tests
> utlizing the same scripts in some cases. Does each 'worker' get a dedicated
> http server instance or do they share the same http server?
>
> On Mon, Dec 5, 2011 at 1:01 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> We never implemented the "general way of marking subdirectories as
>> needing to run serially", but it would be easy to do if we needed to
>> [the 'http' dirs are still special-cased in the code].
>>
>> There is code now (landed a few months ago) to control how many http
>> tests run in parallel separately from the main parallelism flag (so
>> you can run 16 workers but only 2 http tests at a time). I implemented
>> it but never flipped the switch because it was in the middle of Eric
>> flipping the bots over to NRWT in the first place. We should try
>> experimenting with this to see at some point once everything
>> stabilizes otherwise (this is the max_locked_shards() call in
>> manager.py; there's no command line flag).
>>
>> -- Dirk
>>
>> On Mon, Dec 5, 2011 at 11:17 AM, Ojan Vafai <ojan at chromium.org> wrote:
>> > Why is that? I don't know about other ports, but AFAIK, chromium writes
>> > to a
>> > mock clipboard and the Apple mac port writes to a local OS clipboard
>> > instance instead of the global one, specifically to avoid copy/paste
>> > tests
>> > interacting. Even without running tests in parallel, it's probably a
>> > good
>> > idea in order to avoid copy-pastes the developer is doing from affecting
>> > the
>> > test run.
>> >
>> > That said, I believe we have a general way of marking subdirectories as
>> > needing to run serially (which is what we do for http) if there are
>> > other
>> > reasons we need to.
>> >
>> > On Mon, Dec 5, 2011 at 11:12 AM, David Levin <levin at google.com> wrote:
>> >>
>> >> I believe there are some tests (copy/paste) that it would be very hard
>> >> to
>> >> fully shard due to how they work.
>> >>
>> >> dave
>> >>
>> >>
>> >>
>> >> On Mon, Dec 5, 2011 at 11:08 AM, Ojan Vafai <ojan at chromium.org> wrote:
>> >>>
>> >>> I looked at one example that didn't exit
>> >>>
>> >>> early: http://build.webkit.org/builders/SnowLeopard%20Intel%20Release%20%28Tests%29/builds/35153/steps/layout-test/logs/stdio
>> >>>
>> >>> In that case, the http tests were the long tail and took 6 minutes
>> >>> longer
>> >>> than all the other tests. We don't split the http tests up because
>> >>> every
>> >>> time we've tried it's caused too much flakiness. It's unclear if the
>> >>> flakiness points to a bug in the test harness (e.g. in how we setup
>> >>> apache)
>> >>> or to bugs in the tests themselves or both. If someone has time to
>> >>> look into
>> >>> this, this is probably the biggest benefit to be found in NRWT runtime
>> >>> when
>> >>> running tests in parallel.
>> >>>
>> >>> FYI, NRWT outputs a log of the runtime after each run:
>> >>>
>> >>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/9:
>> >>>  4696
>> >>> tests, 1746.63 secs
>> >>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/8:
>> >>>  1177
>> >>> tests, 1693.47 secs
>> >>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/3:
>> >>>  1408
>> >>> tests, 2033.51 secs
>> >>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/2:
>> >>> 941
>> >>> tests, 2119.65 secs
>> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/1:
>> >>>  1121
>> >>> tests, 2041.97 secs
>> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/0:
>> >>>  1453
>> >>> tests, 2515.75 secs
>> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/7:
>> >>>  1189
>> >>> tests, 1731.12 secs
>> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/6:
>> >>>  3556
>> >>> tests, 2114.37 secs
>> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/5:
>> >>> 948
>> >>> tests, 2097.13 secs
>> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/4:
>> >>>  1411
>> >>> tests, 1716.66 secs
>> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/15:
>> >>> 795
>> >>> tests, 2027.16 secs
>> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/14:
>> >>>  1123
>> >>> tests, 1732.72 secs
>> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/13:
>> >>> 425
>> >>> tests, 2021.25 secs
>> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/12:
>> >>>  1175
>> >>> tests, 1710.09 secs
>> >>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO      worker/11:
>> >>>  3462
>> >>> tests, 2096.30 secs
>> >>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO      worker/10:
>> >>>  1449
>> >>> tests, 1722.68 secs
>> >>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO    31120.45
>> >>> cumulative, 1945.03 optimal
>> >>>
>> >>> That shows you that, if we fully sharded all the tests, they would in
>> >>> theory take 1945 seconds to run, but worker/0 (the worker that runs
>> >>> the http
>> >>> tests) took 2515 seconds to run.
>> >>>
>> >>> Ojan
>> >>>
>> >>> On Mon, Dec 5, 2011 at 6:55 AM, Adam Roben <aroben at apple.com> wrote:
>> >>>>
>> >>>> On Dec 2, 2011, at 6:55 PM, Eric Seidel wrote:
>> >>>>
>> >>>> > The SnowLeopard bot went from a 1 hr 4 min (!?!) cycle time, to 38
>> >>>> > min
>> >>>> > (still !?!).
>> >>>>
>> >>>> I suspect our Mac test bots could use a dose of RAM. Many of them
>> >>>> only
>> >>>> have 3GB, since when you're running tests one by one you don't really
>> >>>> need
>> >>>> much more.
>> >>>>
>> >>>> -Adam
>> >>>>
>> >>>> _______________________________________________
>> >>>> webkit-dev mailing list
>> >>>> webkit-dev at lists.webkit.org
>> >>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> webkit-dev mailing list
>> >>> webkit-dev at lists.webkit.org
>> >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> >>>
>> >>
>> >
>> >
>> > _______________________________________________
>> > webkit-dev mailing list
>> > webkit-dev at lists.webkit.org
>> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> >
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>

More information about the webkit-dev mailing list