[webkit-dev] Switching open source Debug bots to building and testing with configuration --force-opt=O3

Mark Lam mark.lam at apple.com
Fri Jun 19 13:03:10 PDT 2020 


> On Jun 19, 2020, at 10:02 AM, Alexey Proskuryakov <ap at webkit.org> wrote:
> 
> 
> 
>> 19 июня 2020 г., в 9:59 AM, Mark Lam <mark.lam at apple.com <mailto:mark.lam at apple.com>> написал(а):
>> 
>> 
>> 
>>> On Jun 19, 2020, at 9:53 AM, Alexey Proskuryakov <ap at webkit.org <mailto:ap at webkit.org>> wrote:
>>> 
>>> 
>>> 
>>>> 18 июня 2020 г., в 9:30 PM, Saam Barati <sbarati at apple.com <mailto:sbarati at apple.com>> написал(а):
>>>> 
>>>> Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.
>>> 
>>> Replying to this point specifically, I wanted to point out that WebKit tests take 2x longer in debug, not 10x longer. JSC tests take 3.6x longer.
>> 
>> Since I collected real data on this last night on the actual bot that runs the JSC stress tests, I’ll share the data here:
>> 
>> Build time
>> A clean build using buid-jsc for a normal Debug build on bot638 takes about 4.5 minutes.
>> A clean build using build-jsc for a --force-opt=O3 Debug build on bot638 takes about 6 minutes.
>> 
>> Test run time
>> Running with a regular Debug build, the test completed in about 4 hours 41 minutes with 1 timeout.
>> Running with a --force-opt=O3 Debug build, the test completed in about 39 minutes with 0 timeouts.
>> 
>> The difference in test run time is 281 minutes vs 29 minutes.  That is a 7.2x ratio, not 3.6x.
> 
> https://build.webkit.org/builders/Apple-Catalina-Debug-JSC-Tests/builds/1080 <https://build.webkit.org/builders/Apple-Catalina-Debug-JSC-Tests/builds/1080> - 4 hrs, 5 secs
> https://build.webkit.org/builders/Apple-Catalina-Release-JSC-Tests/builds/2546 <https://build.webkit.org/builders/Apple-Catalina-Release-JSC-Tests/builds/2546> - 1 hrs, 6 mins, 27 secs
> 
> That's 3.6x.

Some details to consider:

1. https://build.webkit.org/builders/Apple-Catalina-Release-JSC-Tests/builds/2546 <https://build.webkit.org/builders/Apple-Catalina-Release-JSC-Tests/builds/2546> runs on a different bot than https://build.webkit.org/builders/Apple-Catalina-Debug-JSC-Tests/builds/1080 <https://build.webkit.org/builders/Apple-Catalina-Debug-JSC-Tests/builds/1080> .

2. You’re comparing a Release build run vs a Debug build run, not an O3 Debug build vs a regular Debug build.

3. Release builds do less work e.g. all debug ASSERTs are disabled.  Debug only validation code is disabled.

4. Not every file in a Release build is compiled with O3.  If I remember correctly, some are O3, others are not.  None are O0.

5. The collection of JSC tests run for Debug builds and Release builds are slightly different.  For example, Debug builds run 71793 jsc stress tests.  Release builds runs 69950 jsc stress tests.  Hence, the Debug builds run 1843 more tests.

My main point here is that Release builds are not the same as O3 Debug builds: some things are faster, some things are slower.  A Release build test run is at best an approximate representation of how a Debug O3 build test run will behave.  FWIW, my data came from actually running an O3 Debug build and a regular Debug build on the same bot.  

But my number are purely from timing the run of the tests.  If the numbers reported from the bot includes time for any bots scripts in addition to the test run times (I don’t know if there are any), then the total run time will also be different than the numbers I collected.  Feel free to use the 3.6x ratio if you prefer.

About -Og numbers (using the same measurement methology as my O3 Debug and regular Debug runs):

Build time: A clean build using build-jsc for a --force-opt=Og Debug build on bot638 takes about 5.5 minutes.
Test run time: Running with a --force-opt=Og Debug build, the test completed in about 2 hours 19 minute minutes with 0 timeouts.

That's about 3.5x slower than the Debug O3 run.

Mark


> 
> - Alexey
> 
> 
> 
>>>> Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:
>>> 
>>> Last time this was discussed, we talked about -Og, which is specifically designed for the purpose I believe. Where do we stand on understanding and adopting that?
>> 
>> I tried -Og last time after your suggestion, and I remember thinking that the perf was not acceptable back then.  I’ll collect the data again and report back with real number later today.
>> 
>> Mark
>> 
>>> 
>>> - Alexey
>>> 
>>>> - Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
>>>> - O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.
>>>> 
>>>> (And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)
>>>> 
>>>> In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.
>>>> 
>>>> - Saam
>>>> 
>>>>> On Jun 17, 2020, at 9:48 PM, Simon Fraser <simon.fraser at apple.com <mailto:simon.fraser at apple.com>> wrote:
>>>>> 
>>>>> I also object to losing good stack traces for crashes on Debug bots.
>>>>> 
>>>>> Also, I don't think Debug bots should build something different from what I build at my desk.
>>>>> 
>>>>> Simon
>>>>> 
>>>>>> On Jun 17, 2020, at 1:36 PM, Mark Lam <mark.lam at apple.com <mailto:mark.lam at apple.com>> wrote:
>>>>>> 
>>>>>> Hi folks,
>>>>>> 
>>>>>> We're planning to switch the JSC EWS bot and build.webkit.org <http://build.webkit.org/> Debug build and test bots to building with the following set first:
>>>>>> ./Tools/Scripts/set-webkit-configuration --force-opt=O3
>>>>>> 
>>>>>> This means the Debug builds will be built with optimization level forced to O3.
>>>>>> 
>>>>>> Why are we doing this?
>>>>>> 1. So that the JSC EWS will start catching ASSERT failures.
>>>>>> 2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
>>>>>> 3. Tests will run to completion faster and we’ll catch regressions sooner.
>>>>>> 
>>>>>> The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.
>>>>>> 
>>>>>> We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?
>>>>>> 
>>>>>> Thanks.
>>>>>> 
>>>>>> Cheers,
>>>>>> Mark
>>>>>> 
>>>>>> _______________________________________________
>>>>>> webkit-dev mailing list
>>>>>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>>>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>>>>> 
>>>>> _______________________________________________
>>>>> webkit-dev mailing list
>>>>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>>>> _______________________________________________
>>>> webkit-dev mailing list
>>>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>>> 
>>> - Alexey
>>> 
>>> 
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20200619/76591048/attachment.htm>

More information about the webkit-dev mailing list