<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[186041] trunk/PerformanceTests</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/186041">186041</a></dd>
<dt>Author</dt> <dd>fpizlo@apple.com</dd>
<dt>Date</dt> <dd>2015-06-27 20:47:00 -0700 (Sat, 27 Jun 2015)</dd>
</dl>
<h3>Log Message</h3>
<pre>[JetStream] Raise the percentile of mandreel-latency and splay-latency
https://bugs.webkit.org/show_bug.cgi?id=146378
Reviewed by Mark Lam.
The current percentile is 95%. When I looked at the sample lists in our GC, it was
clear that the worst 5% samples completely amortize our GC pauses. Our GC pauses can
be quite bad. Clearly, splay-latency is meant to test whether we have an incremental
GC that ensures that you don't have bad worst-case pauses. But 95% is too small,
because it doesn't really capture those pauses. Raising the percentile to above 99%
appears to do the trick. 99.5% or more seems like a good bet. The trade-off there is
just that if we set it too high, then we won't have enough statistics. Doing this very
clearly rewards GCs that are incremental, and punishes GCs that aren't (like ours).
That's what we want, since in the future we want to use this test to guide any
improvements to the worst-case performance of our GC.
The way that the percentile is selected will also affect mandreel-latency. That's a
good thing, because 95% is probably too low for that test as well. That test ends up
with >10k samples. The goal of using 95% in the first place was to get enough samples
to have a stable average. But if we have >10k samples, we can push that percentile up
much higher and still get good statistics while achieving the effect we want - i.e.
getting the worst case.
I don't think that we need to do the same thing for cdjs. That test only takes 200
samples, so 95% means we report the average of the worst 10 samples. That's probably
good enough.
* JetStream/Octane2/base.js: Raise the percentile as described above.
(BenchmarkSuite.prototype.RunSingleBenchmark):
* JetStream/Reference.js: Tweak the reference times to bring the latency tests closer to 100ish on my machine.
* JetStream/create.rb: Bump the version.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkPerformanceTestsChangeLog">trunk/PerformanceTests/ChangeLog</a></li>
<li><a href="#trunkPerformanceTestsJetStreamOctane2basejs">trunk/PerformanceTests/JetStream/Octane2/base.js</a></li>
<li><a href="#trunkPerformanceTestsJetStreamReferencejs">trunk/PerformanceTests/JetStream/Reference.js</a></li>
<li><a href="#trunkPerformanceTestsJetStreamcreaterb">trunk/PerformanceTests/JetStream/create.rb</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkPerformanceTestsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/ChangeLog (186040 => 186041)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/ChangeLog 2015-06-28 03:29:41 UTC (rev 186040)
+++ trunk/PerformanceTests/ChangeLog 2015-06-28 03:47:00 UTC (rev 186041)
</span><span class="lines">@@ -1,3 +1,37 @@
</span><ins>+2015-06-26 Filip Pizlo <fpizlo@apple.com>
+
+ [JetStream] Raise the percentile of mandreel-latency and splay-latency
+ https://bugs.webkit.org/show_bug.cgi?id=146378
+
+ Reviewed by Mark Lam.
+
+ The current percentile is 95%. When I looked at the sample lists in our GC, it was
+ clear that the worst 5% samples completely amortize our GC pauses. Our GC pauses can
+ be quite bad. Clearly, splay-latency is meant to test whether we have an incremental
+ GC that ensures that you don't have bad worst-case pauses. But 95% is too small,
+ because it doesn't really capture those pauses. Raising the percentile to above 99%
+ appears to do the trick. 99.5% or more seems like a good bet. The trade-off there is
+ just that if we set it too high, then we won't have enough statistics. Doing this very
+ clearly rewards GCs that are incremental, and punishes GCs that aren't (like ours).
+ That's what we want, since in the future we want to use this test to guide any
+ improvements to the worst-case performance of our GC.
+
+ The way that the percentile is selected will also affect mandreel-latency. That's a
+ good thing, because 95% is probably too low for that test as well. That test ends up
+ with >10k samples. The goal of using 95% in the first place was to get enough samples
+ to have a stable average. But if we have >10k samples, we can push that percentile up
+ much higher and still get good statistics while achieving the effect we want - i.e.
+ getting the worst case.
+
+ I don't think that we need to do the same thing for cdjs. That test only takes 200
+ samples, so 95% means we report the average of the worst 10 samples. That's probably
+ good enough.
+
+ * JetStream/Octane2/base.js: Raise the percentile as described above.
+ (BenchmarkSuite.prototype.RunSingleBenchmark):
+ * JetStream/Reference.js: Tweak the reference times to bring the latency tests closer to 100ish on my machine.
+ * JetStream/create.rb: Bump the version.
+
</ins><span class="cx"> 2015-06-19 Filip Pizlo <fpizlo@apple.com>
</span><span class="cx">
</span><span class="cx"> Run CDjs as part of JSC stress testing
</span></span></pre></div>
<a id="trunkPerformanceTestsJetStreamOctane2basejs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/JetStream/Octane2/base.js (186040 => 186041)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/JetStream/Octane2/base.js 2015-06-28 03:29:41 UTC (rev 186040)
+++ trunk/PerformanceTests/JetStream/Octane2/base.js 2015-06-28 03:47:00 UTC (rev 186041)
</span><span class="lines">@@ -334,7 +334,7 @@
</span><span class="cx"> if (data.runs < benchmark.minIterations) return data;
</span><span class="cx"> var usec = (data.elapsed * 1000) / data.runs;
</span><span class="cx"> var latencySamples = (benchmark.latencyResult != null) ? benchmark.latencyResult() : [0];
</span><del>- var percentile = 95;
</del><ins>+ var percentile = 99.5;
</ins><span class="cx"> var latency = BenchmarkSuite.AverageAbovePercentile(latencySamples, percentile) * 1000;
</span><span class="cx"> this.NotifyStep(new BenchmarkResult(benchmark, usec, latency));
</span><span class="cx"> return null;
</span></span></pre></div>
<a id="trunkPerformanceTestsJetStreamReferencejs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/JetStream/Reference.js (186040 => 186041)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/JetStream/Reference.js 2015-06-28 03:29:41 UTC (rev 186040)
+++ trunk/PerformanceTests/JetStream/Reference.js 2015-06-28 03:47:00 UTC (rev 186041)
</span><span class="lines">@@ -53,17 +53,17 @@
</span><span class="cx"> "earley-boyer": 2.5795868328750444,
</span><span class="cx"> "regexp-2010": 61.82352941176467,
</span><span class="cx"> "splay": 0.9286376274328075,
</span><del>- "splay-latency": 3.524855736311738,
</del><ins>+ "splay-latency": 40,
</ins><span class="cx"> "navier-stokes": 9.653846153846146,
</span><span class="cx"> "pdfjs": 88.4166666666666,
</span><span class="cx"> "mandreel": 157.14285714285708,
</span><del>- "mandreel-latency": 1.4940079129942474,
</del><ins>+ "mandreel-latency": 10,
</ins><span class="cx"> "gbemu": 135.9999999999998,
</span><span class="cx"> "code-first-load": 2.3249465349251905,
</span><span class="cx"> "box2d": 28.416666666666636,
</span><span class="cx"> "zlib": 887.666666666666,
</span><span class="cx"> "typescript": 1149.9999999999993,
</span><span class="cx"> "lua": 29858,
</span><del>- "cdjs": 14,
</del><ins>+ "cdjs": 20,
</ins><span class="cx"> "geomean": 31.556451704472156,
</span><span class="cx"> });
</span></span></pre></div>
<a id="trunkPerformanceTestsJetStreamcreaterb"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/JetStream/create.rb (186040 => 186041)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/JetStream/create.rb 2015-06-28 03:29:41 UTC (rev 186040)
+++ trunk/PerformanceTests/JetStream/create.rb 2015-06-28 03:47:00 UTC (rev 186041)
</span><span class="lines">@@ -26,7 +26,7 @@
</span><span class="cx"> require "pathname"
</span><span class="cx"> require "shellwords"
</span><span class="cx">
</span><del>-VERSION = "1.1-alpha2"
</del><ins>+VERSION = "1.1-alpha3"
</ins><span class="cx"> DIRECTORY_NAME = "JetStream-#{VERSION}"
</span><span class="cx">
</span><span class="cx"> CDJS_FILES = [
</span></span></pre>
</div>
</div>
</body>
</html>