[webkit-dev] Some text about the B3 compiler

[Ryosuke Niwa]

On Tue, Feb 2, 2016 at 10:42 AM, Carlos Alberto Lopez Perez
<clopez at igalia.com> wrote:
> On 02/02/16 00:27,
- R. Niwa
 wrote:
>> On Mon, Feb 1, 2016 at 7:32 AM, Carlos Alberto Lopez Perez
>> <clopez at igalia.com> wrote:
>>> On 31/01/16 05:16, Filip Pizlo wrote:
>>>> As my original message said, I was wondering if there was any support
>>>> for running some JavaScript tests *in browser*.  run-jsc-stress-tests
>>>> doesn’t support that because it doesn’t know what a browser is.
>>>>
>>>> Some tests, like Kraken, Octane, JetStream, and Speedometer, either
>>>> require a browser to run (like JetStream and Speedometer) or have
>>>> significantly different behavior in the browser than in their
>>>> command-line harnesses (like Kraken and Octane).  If you did have a
>>>> bot that ran these tests in some GTK+ or EFL browser, you’d probably
>>>> catch bugs that testing the JSC shell cannot catch.
>>>>
>>>> -Filip
>>>
>>> Some questions regarding this JSC in-browser benchmarks:
>>>
>>> How does the Apple port runs this? Do you use some script that is
>>> currently available on the WebKit source tree? Are the buildbots running
>>> this tests public?
>>
>> We've been running those benchmarks with
>> http://trac.webkit.org/browser/trunk/Tools/Scripts/run-benchmark
>>
>
> I see.
>
> But this script seems focused on comparing the performance between
> different browsers (safari vs chrome vs firefox) rather than in testing
> and comparing the performance between different revisions of WebKit.

Not at all.  It simply supports running benchmark in other browsers.

> Do you think it makes any difference (from the point of view of
> detecting failures, not from the performance PoV) to run this tests in a
> full-fledged browser like Safari rather than in WebKitTestRunner ?

Yes. There are many browser features that can significantly impact the
real world performance.

> We already have a performance test bot running tests inside WTR.
> And I see that the current set of tests executed on this bot already
> includes Speedometer, and that JetStream and Sunspider are skipped on
> PerformanceTests/Skipped.
>
> So I see some options going forward:
>
>  - Fix the JetStream and Sunspider tests so they can be run as part of
> the current run-perf-tests script that the performance bots execute.

We should use run-benchmark instead since run-benchmark spits out the
JSON file that's compatible with run-pref-tests.

>  - Implement support on the script run-benchmark to run the tests inside
> WTR, and create a new step running this script that will be executed on
> the test bots.

I don't see a point in doing this.   Why is it desirable to run these
benchmarks inside WebKitTestRunner?

>  - Deploy a new bot that runs run-perf-tests on a full-fledged browser
> like Safari or Epiphany.

We should just do this.

> I wonder what you think is the best option or if there is some option
> not viable.
>
> From my PoV, the option #1 has the advantage of reusing the current
> infrastructure that collects and draws performance data at
> https://perf.webkit.org

We have an internal instance of the same dashboard to which we're
reporting results of run-benchmark script.

- R. Niwa