FYI, if you have any complaints or suggestions, please file a bug and block <a href="https://bugs.webkit.org/show_bug.cgi?id=77037">https://bugs.webkit.org/show_bug.cgi?id=77037</a>.<div><br></div><div>Best,<br clear="all">
Ryosuke Niwa<br><font color="#999999">Software Engineer</font><div><font color="#999999">Google Inc.</font></div><div><font color="#999999"><br></font></div><div class="gmail_quote">On Wed, Jan 25, 2012 at 10:16 AM, Ryosuke Niwa <span dir="ltr"><<a href="mailto:rniwa@webkit.org">rniwa@webkit.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">...and Chromium Mac Release (Perf) has been added to <a href="http://build.webkit.org" target="_blank">build.webkit.org</a>:<div>
<a href="http://build.webkit.org/waterfall?show=Chromium%20Mac%20Release%20(Perf)" target="_blank">http://build.webkit.org/waterfall?show=Chromium%20Mac%20Release%20(Perf)</a></div>
<div><br></div><div>See graphs on <a href="http://webkit-perf.appspot.com" target="_blank">http://webkit-perf.appspot.com</a></div><span class="HOEnZb"><font color="#888888"><div><br></div></font></span><div><span class="HOEnZb"><font color="#888888">- Ryosuke<br>
<br></font></span><div class="gmail_quote"><div class="im">On Thu, Jan 19, 2012 at 3:20 PM, Ryosuke Niwa <span dir="ltr"><<a href="mailto:rniwa@webkit.org" target="_blank">rniwa@webkit.org</a>></span> wrote:<br>
</div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi WebKittens,<br><br><b>Executive Summary</b><div><ul><li>I've added <font face="'courier new', monospace">Tools/Scripts/run-perf-test</font>, try out</li>
<li>Please add <font face="'courier new', monospace">--no-timeout</font> and <font face="'courier new', monospace">--timeout</font> options to your DRT</li>
<li>Perf-o-matic coming on <a href="http://webkit-perf.appspot.com" target="_blank">webkit-perf.appspot.com</a>, a clone of <a href="http://graphs.mozilla.org" target="_blank">graphs.mozilla.org</a></li><li>Chromium Mac perf bots coming on <a href="http://build.webkit.org" target="_blank">build.webkit.org</a></li>
<li>Use <font face="'courier new', monospace">PerformanceTests/Parser/resources/runner.js</font> to write new performance tests</li></ul><div><br></div><div><b>Background</b></div><div>We have some performance tests in PerformanceTests but they're not ran by any bots. In fact, there are no performance bots at all on <a href="http://build.webkit.org" target="_blank">build.webkit.org</a>. While <a href="http://build.chromium.org/p/chromium.perf/console" target="_blank">Chromium has perf bots</a>, we can only see progressions and regressions triggered by WebKit changes when Chromium gets a WebKit roll (pulling newer version of WebKit), which happens only a handful times a day. It doesn't scale to the rate at which we're making changes to WebKit and the visibility and the usability of bots are not great for non-Chromium WebKit contributors. Furthermore, Chromium perf bots will not catch JSC progressions and regressions at all.</div>
<div><br></div><div><b>Means to Run Performance Tests</b></div><div>I've added <font face="'courier new', monospace">Tools/Scripts/run-perf-tests</font> to run PerformanceTests in DRT based on the work Ilya Tikhonovsky (loislo) has done for <font face="'courier new', monospace">run-inspector-perf-tests.py</font>. The script aims to run performance tests both locally and on bots similar to the way <font face="'courier new', monospace">run-webkit-tests</font> works and runs on Mac (WebKit1) and Chromium ports. Please try it out and give me a feedback (you can file a bug with "run-perf-tests: " in the summary and cc me).</div>
<div><br></div><div>I didn't merge it into <font face="'courier new', monospace">run-webkit-tests</font> because performance tests don't pass/fail but instead give us some values that fluctuate over time. While Chromium takes an approach to hard-code the rage of acceptable values, such an approach has a high maintenance cost and prone to problems such as having to increase the range periodically as the score slowly degrades over time. Also, as you can see on <a href="http://build.chromium.org/p/chromium.perf/console" target="_blank">Chromium perf bots</a>, the test results tend to fluctuate a lot so hard-coding a tight range of acceptable value is tricky.</div>
<div><br></div><div>Unlike <span style="font-family:'courier new',monospace">run-webkit-tests</span>, <span style="font-family:'courier new',monospace">run-perf-tests</span> doesn't generate any HTML or JSON files to summarize the results by default since only output you get out of performance tests are time took to run tests or scores, which are already reported on stdout. The output of <span style="font-family:'courier new',monospace">run-perf-tests</span> is designed to be compatible with Chromium perf bots but we can easily change that to something more human friendly if people are so inclined. The script optionally generates a JSON file to be used by perf bots.</div>
<div><br></div><div>In order for other ports (e.g. Windows, Qt, GTK, etc...) to support run-perf-tests, simply their respective DRT needs to support <font face="'courier new', monospace">--no-timeout</font> option that disables the watchdog timer. This is necessary as some performance tests take a long time to run. Also, we'll appreciate your help if you could add <font face="'courier new', monospace">--timeout</font> option per <a href="https://bugs.webkit.org/show_bug.cgi?id=76662" target="_blank">https://bugs.webkit.org/show_bug.cgi?id=76662</a> for the code sanity.</div>
<div><br></div><div><b>Adding Performance Bots</b></div><div>In the next couple of days, I'm going to post a patch to add a Chromium Mac Perf bot to <a href="http://build.webkit.org" target="_blank">build.webkit.org</a> (of course, upon appropriate reviews) that runs <span style="font-family:'courier new',monospace">run-perf-tests</span> and uploads a JSON file to <a href="http://webkit-perf.appspot.com" target="_blank">webkit-perf.appspot.com</a>, a clone of <a href="http://graphs.mozilla.org" target="_blank">graphs.mozilla.org</a>.</div>
<div><br></div><div>While we could have adopted Chromium's perf bot output where each slave generates a JSON file with a html front end that loads the JSON, the approach didn't scale well for Chromium when the number of historical values stored on each slave soared and the size of JSON increased proportionally over time. Furthermore, it's hard to compare values between different bots or tests. On the other hand, creating a new front end seemed like a too much work. As such, I've decided to port <a href="https://github.com/mozilla/graphs" target="_blank">Mozilla's Graph Server</a> to WebKit after consulting with tony^work, ojan, and evmar.</div>
<div><br></div><div>While we could have added another dedicated apache server with all nice features Graph Server's native backend provides, the maintenance cost of maintaining such a server seemed too high. Also, Robert Helmer (rhelmer), a Mozilla contributor who is actively working on the Graph Server, told me that Mozilla is planning to replace the backend with a key-value database. Given these circumstances and some experimentations, I wrote our own backend using <a href="http://code.google.com/appengine/" target="_blank">Google App Engine</a> for its low maintenance cost and ease of use; note App Engine is already used by commit-queue and flakiness dashboard.</div>
<div><br></div><div>My work to port the Graph Server is near completion and I expect it to be working in the next couple of days just as I add a Chromium Mac Perf bot. If you're interested in adding new perf bots for your port, please contact me directly and I'll give you a detailed instruction on what needs to happen (it's super trivial but involves giving out or receiving a password).</div>
<div><br></div></div><div><b>How to Write Performance Tests</b></div><div>If you're interested in adding more performance tests (you should be!), then use <a href="http://trac.webkit.org/browser/trunk/PerformanceTests/Parser/html-parser.html" target="_blank">http://trac.webkit.org/browser/trunk/PerformanceTests/Parser/html-parser.html</a> as an example. It uses <a href="http://trac.webkit.org/browser/trunk/PerformanceTests/Parser/resources/runner.js" target="_blank">runner.js</a>, which automatically aggregates results over multiple runs and outputs the results in the preferred format <span style="font-family:'courier new',monospace">run-perf-tests</span> understands.</div>
<div><br></div><div>Since there hadn't been any script to run performance tests, tests in PerformanceTests don't have an uniform output format. As a result, <font face="'courier new', monospace">run-perf-tests</font> only supports running tests in Bindings, Parser, and inspector at the moment. I'd really appreciate your help if you could convert the existing tests to use runner.js to increase the number of performance tests <span style="font-family:'courier new',monospace">run-perf-tests</span> can run or modify <span style="font-family:'courier new',monospace">run-perf-tests</span> so that it can run more tests. Obviously, our goal is to be able to run all tests in PerformanceTests by <span style="font-family:'courier new',monospace">run-perf-tests</span>.</div>
<div><br></div><div>Note Hajime Morita (morrita) has taken initiative on <a href="https://bugs.webkit.org/show_bug.cgi?id=76156" target="_blank">the effort to run Dromaeo in DRT</a>.</div><div><br></div><div>Best regards,</div>
<span><font color="#888888"><div>Ryosuke Niwa<br>
<font color="#999999">Software Engineer</font><div><font color="#999999">Google Inc.</font></div><div><font color="#999999"><br></font></div></div>
</font></span></blockquote></div></div></div><br></div>
</blockquote></div><br></div>