[Webkit-unassigned] [Bug 90554] New: [Gtk] 64-bit Release builder is overpowered for testing purposes

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=90554

           Summary: [Gtk] 64-bit Release builder is overpowered for
                    testing purposes
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: zandobersek at gmail.com
                CC: gns at gnome.org, mrobinson at webkit.org,
                    cgarcia at igalia.com, pnormand at igalia.com,
                    svillar at igalia.com


With its 24 cores, the power of the 64-bit Release builder (working on the gtk-linux-slave-2) comes in handy when compiling, but is actually too much when running the layout tests. Here are per-worker running times from a recent test run:

05:42:43.752 19668 Thread timing:
05:42:43.752 19668      worker/22:  1244 tests,  77.82 secs
05:42:43.752 19668      worker/23:  1233 tests,  72.57 secs
05:42:43.752 19668      worker/20:  1111 tests,  71.24 secs
05:42:43.752 19668      worker/21:   872 tests,  87.23 secs
05:42:43.752 19668       worker/9:  1670 tests,  80.52 secs
05:42:43.752 19668       worker/8:   798 tests, 184.16 secs
05:42:43.752 19668       worker/3:  1480 tests,  91.36 secs
05:42:43.752 19668       worker/2:  1456 tests,  75.10 secs
05:42:43.753 19668       worker/1:   556 tests,  80.89 secs
05:42:43.753 19668       worker/0:  1547 tests, 321.38 secs
05:42:43.753 19668       worker/7:   409 tests, 113.01 secs
05:42:43.753 19668       worker/6:   817 tests, 167.93 secs
05:42:43.753 19668       worker/5:  1714 tests,  75.03 secs
05:42:43.753 19668       worker/4:   241 tests, 106.60 secs
05:42:43.753 19668      worker/19:   914 tests,  92.35 secs
05:42:43.753 19668      worker/18:  1657 tests,  71.64 secs
05:42:43.753 19668      worker/17:  1777 tests,  77.16 secs
05:42:43.753 19668      worker/16:  1196 tests, 159.18 secs
05:42:43.753 19668      worker/15:  1288 tests,  70.74 secs
05:42:43.753 19668      worker/14:  1509 tests,  70.83 secs
05:42:43.753 19668      worker/13:  2112 tests,  82.04 secs
05:42:43.753 19668      worker/12:   569 tests,  77.71 secs
05:42:43.753 19668      worker/11:   930 tests,  74.06 secs
05:42:43.753 19668      worker/10:   599 tests,  73.71 secs
05:42:43.753 19668    2454.27 cumulative, 102.26 optimal

worker/0 takes around 5min20sec to complete the testing while only 5 more workers require more than 100 seconds to complete their test runs. worker/0 is the worker running the locked http tests shard, meaning tests from this shard can't be arranged onto other workers. This means that with sufficient number of workers running in parallel, the worker running the http tests (which are the only tests it will run) will be the one taking the most time, limiting the overall run-webkit-tests execution duration.

Currently the http tests still contain two tests that run for half a minute due to inappropriate test failure handling. They'll be skipped, bringing down the worker testing duration for another minute - that's around 260 seconds for that worker.

With those two tests skipped and now around 2394 seconds taking for the complete test suit to run (when tests are run in succession), this means around 2194 seconds for non-http tests. If the per-worker limit is kept around 260 seconds, it turns out only 9 extra workers are required to be run in parallel for the whole test suit to be tested in around 260 seconds (4min20sec - pretty cool).

Also note that there are a few other tests taking 30 seconds to fail, increasing the testing time unnecessarily. Check out the treemap[1] to spot them.

Then again, the GTK port currently skips around 2500 tests and with some of these being unskipped and general increasing number of tests, the cumulative test time will certainly increase.

In debug builds (based on the last successful testing on the builder) the http tests take around 700 seconds but that will be narrowed down to just around 430 seconds. The cumulative test time will then take around 5580 seconds, or 5150 seconds for non-http tests. With the limit set at around 430 seconds, it would take 12 additional workers to be run in parallel to finish the testing around those 430 seconds (7min10sec).

[1] - http://test-results.appspot.com/dashboards/treemap.html#group=%40ToT%20-%20webkit.org&builder=GTK%20Linux%2064-bit%20Release

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.