[webkit-dev] run-webkit-tests is moving to parallell testing by default (this weekend)

[Dirk Pranke]

On Fri, Dec 2, 2011 at 3:55 PM, Eric Seidel <eric at webkit.org> wrote:
> run-webkit-tests is moving to parallell testing by default (this weekend)
>
> I just moved Mac this afternoon.  The SnowLeopard bot went from a 1 hr
> 4 min (!?!) cycle time, to 38 min (still !?!).
> http://build.webkit.org/builders/SnowLeopard%20Intel%20Debug%20%28Tests%29/builds/3317
> http://build.webkit.org/builders/SnowLeopard%20Intel%20Debug%20%28Tests%29/builds/3318
>
> I will start working on the other bots once I believe the Mac ones to
> be stable.  Let me know if you have any troubles!

I'm not sure we want to flip the switch on them just yet ... if you
look at the log file:

2011-12-02 15:34:44,330 98936 printing.py:462 INFO Tests that timed
out or crashed:
2011-12-02 15:34:44,330 98936 printing.py:462 INFO
java/argument-to-object-type.html took 830.8 seconds
2011-12-02 15:34:44,331 98936 printing.py:462 INFO
fast/frames/lots-of-iframes.html took 733.7 seconds
2011-12-02 15:34:44,331 98936 printing.py:462 INFO
sputnik/Unicode/Unicode_320/S7.6_A3.1.html took 565.3 seconds
2011-12-02 15:34:44,331 98936 printing.py:462 INFO
fast/frames/lots-of-objects.html took 498.0 seconds
2011-12-02 15:34:44,331 98936 printing.py:462 INFO
java/array-return.html took 244.7 seconds
2011-12-02 15:34:44,331 98936 printing.py:462 INFO
inspector/profiler/cpu-profiler-profiling.html took 161.0 seconds
2011-12-02 15:34:44,331 98936 printing.py:462 INFO

some of those tests are taking 10 minutes or more to complete ...
there's clearly one or more bugs here keeping NRWT from timing out DRT
properly. Some are almost certainly in NRWT, but I wonder if there are
things in the o/s or in DRT's implementation that are being serialized
as well?

For contrast, in the single-threaded case, even the slowest test in
the previous run only took 140 seconds.

Even in the absence of bugs, the mac port seems to be running with
timeout values of 35 seconds and 175 seconds for "slow" tests. That
seems awfully slow :)

For comparison, the single-threaded Apple SL Release build took 26
minutes to complete in the most recent not-horribly-broken build
(#35132); the Chromium SL Release took 15 minutes to complete using
two workers (which seems about right, since Chromium's DRT is somewhat
slower than Apple's IIRC). The Chromium SL Debug build also took a
cumulative hour to run, but completed in ~30 minutes with two workers.


-- Dirk

PS: As an aside, we should turn on the "allow multiple http tests in
parallel flag" on at least some of the bots; the Chromium Linux Debug
bot is running on a 48-core machine (IIRC) and most of the workers are
completing in 2-4 minutes but the http tests are running all in one
worker and taking 24 ... with optimal sharding the whole test run
should take about 4 minutes according to the log for that bot:
http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Linux%20%28dbg%29/builds/903/steps/webkit_tests/logs/stdio