[webkit-dev] run-webkit-tests is moving to parallell testing by default (this weekend)

Mon Dec 5 15:13:26 PST 2011

Some http tests make use of stateful php scritps with different tests
utlizing the same scripts in some cases. Does each 'worker' get a dedicated
http server instance or do they share the same http server?

On Mon, Dec 5, 2011 at 1:01 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> We never implemented the "general way of marking subdirectories as
> needing to run serially", but it would be easy to do if we needed to
> [the 'http' dirs are still special-cased in the code].
>
> There is code now (landed a few months ago) to control how many http
> tests run in parallel separately from the main parallelism flag (so
> you can run 16 workers but only 2 http tests at a time). I implemented
> it but never flipped the switch because it was in the middle of Eric
> flipping the bots over to NRWT in the first place. We should try
> experimenting with this to see at some point once everything
> stabilizes otherwise (this is the max_locked_shards() call in
> manager.py; there's no command line flag).
>
> -- Dirk
>
> On Mon, Dec 5, 2011 at 11:17 AM, Ojan Vafai <ojan at chromium.org> wrote:
> > Why is that? I don't know about other ports, but AFAIK, chromium writes
> to a
> > mock clipboard and the Apple mac port writes to a local OS clipboard
> > instance instead of the global one, specifically to avoid copy/paste
> tests
> > interacting. Even without running tests in parallel, it's probably a good
> > idea in order to avoid copy-pastes the developer is doing from affecting
> the
> > test run.
> >
> > That said, I believe we have a general way of marking subdirectories as
> > needing to run serially (which is what we do for http) if there are other
> > reasons we need to.
> >
> > On Mon, Dec 5, 2011 at 11:12 AM, David Levin <levin at google.com> wrote:
> >>
> >> I believe there are some tests (copy/paste) that it would be very hard
> to
> >> fully shard due to how they work.
> >>
> >> dave
> >>
> >>
> >>
> >> On Mon, Dec 5, 2011 at 11:08 AM, Ojan Vafai <ojan at chromium.org> wrote:
> >>>
> >>> I looked at one example that didn't exit
> >>> early:
> http://build.webkit.org/builders/SnowLeopard%20Intel%20Release%20%28Tests%29/builds/35153/steps/layout-test/logs/stdio
> >>>
> >>> In that case, the http tests were the long tail and took 6 minutes
> longer
> >>> than all the other tests. We don't split the http tests up because
> every
> >>> time we've tried it's caused too much flakiness. It's unclear if the
> >>> flakiness points to a bug in the test harness (e.g. in how we setup
> apache)
> >>> or to bugs in the tests themselves or both. If someone has time to
> look into
> >>> this, this is probably the biggest benefit to be found in NRWT runtime
> when
> >>> running tests in parallel.
> >>>
> >>> FYI, NRWT outputs a log of the runtime after each run:
> >>>
> >>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/9:
>  4696
> >>> tests, 1746.63 secs
> >>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/8:
>  1177
> >>> tests, 1693.47 secs
> >>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/3:
>  1408
> >>> tests, 2033.51 secs
> >>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO       worker/2:
> 941
> >>> tests, 2119.65 secs
> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/1:
>  1121
> >>> tests, 2041.97 secs
> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/0:
>  1453
> >>> tests, 2515.75 secs
> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/7:
>  1189
> >>> tests, 1731.12 secs
> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/6:
>  3556
> >>> tests, 2114.37 secs
> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/5:
> 948
> >>> tests, 2097.13 secs
> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO       worker/4:
>  1411
> >>> tests, 1716.66 secs
> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/15:
> 795
> >>> tests, 2027.16 secs
> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/14:
>  1123
> >>> tests, 1732.72 secs
> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/13:
> 425
> >>> tests, 2021.25 secs
> >>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO      worker/12:
>  1175
> >>> tests, 1710.09 secs
> >>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO      worker/11:
>  3462
> >>> tests, 2096.30 secs
> >>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO      worker/10:
>  1449
> >>> tests, 1722.68 secs
> >>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO    31120.45
> >>> cumulative, 1945.03 optimal
> >>>
> >>> That shows you that, if we fully sharded all the tests, they would in
> >>> theory take 1945 seconds to run, but worker/0 (the worker that runs
> the http
> >>> tests) took 2515 seconds to run.
> >>>
> >>> Ojan
> >>>
> >>> On Mon, Dec 5, 2011 at 6:55 AM, Adam Roben <aroben at apple.com> wrote:
> >>>>
> >>>> On Dec 2, 2011, at 6:55 PM, Eric Seidel wrote:
> >>>>
> >>>> > The SnowLeopard bot went from a 1 hr 4 min (!?!) cycle time, to 38
> min
> >>>> > (still !?!).
> >>>>
> >>>> I suspect our Mac test bots could use a dose of RAM. Many of them only
> >>>> have 3GB, since when you're running tests one by one you don't really
> need
> >>>> much more.
> >>>>
> >>>> -Adam
> >>>>
> >>>> _______________________________________________
> >>>> webkit-dev mailing list
> >>>> webkit-dev at lists.webkit.org
> >>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> webkit-dev mailing list
> >>> webkit-dev at lists.webkit.org
> >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >>>
> >>
> >
> >
> > _______________________________________________
> > webkit-dev mailing list
> > webkit-dev at lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20111205/8a117f78/attachment.html>