<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[166700] trunk/Websites/perf.webkit.org</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/166700">166700</a></dd>
<dt>Author</dt> <dd>rniwa@webkit.org</dd>
<dt>Date</dt> <dd>2014-04-03 00:07:02 -0700 (Thu, 03 Apr 2014)</dd>
</dl>
<h3>Log Message</h3>
<pre>WebKitPerfMonitor: Y-axis adjustment is too aggressive
https://bugs.webkit.org/show_bug.cgi?id=130937
Reviewed by Andreas Kling.
Previously, adjusted min. and max. were defined as the two standards deviations away from EWMA of measured
results. This had two major problems:
1. Two standard deviations can be too small to show the confidence interval for results.
2. Sometimes baseline and target can be more than two standards deviations away.
Fixed the bug by completely rewriting the algorithm to compute the interval. Instead of blindly using two
standard deviations as margins, we keep adding quarter the standard deviation on each side until more than 90%
of points lie in the interval or we've expanded 4 standard deviations. Once this condition is met, we reduce
the margin on each side separately to reduce the empty space on either side.
A more rigorous approach would involve computing least squared value of results with respect to intervals
but that seems like an overkill for a simple UI problem; it's also computationally expensive.
* public/index.html:
(Chart..adjustedIntervalForRun): Extracted from computeYAxisBoundsToFitLines.
(Chart..computeYAxisBoundsToFitLines): Compute the min. and max. adjusted intervals out of adjusted intervals
for each runs (current, baseline, and target) so that at least one point from each set of results is shown.
We wouldn't see the difference between measured values versus baseline and target values otherwise.
* public/js/helper-classes.js:
(PerfTestResult.unscaledConfidenceIntervalDelta): Returns the default value if the confidence
interval delta cannot be computed.
(PerfTestResult.isInUnscaledInterval): Added. Returns true iff the confidence intervals lies
within the given interval.
(PerfTestRuns..filteredResults): Extracted from unscaledMeansForAllResults now that PerfTestRuns.min and
PerfTestRuns.max need to use both mean and confidence interval delta for each result.
(PerfTestRuns..unscaledMeansForAllResults):
(PerfTestRuns.min): Take the confidence interval delta into account.
(PerfTestRuns.max): Ditto.
(PerfTestRuns.countResults): Returns the number of results in the given time frame (> minTime).
(PerfTestRuns.countResultsInInterval): Returns the number of results whose confidence interval lie within the
given interval.
(PerfTestRuns.exponentialMovingArithmeticMean): Fixed the typo so that it actually computes the EWMA.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkWebsitesperfwebkitorgChangeLog">trunk/Websites/perf.webkit.org/ChangeLog</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicindexhtml">trunk/Websites/perf.webkit.org/public/index.html</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicjshelperclassesjs">trunk/Websites/perf.webkit.org/public/js/helper-classes.js</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkWebsitesperfwebkitorgChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/ChangeLog (166699 => 166700)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/ChangeLog 2014-04-03 06:21:27 UTC (rev 166699)
+++ trunk/Websites/perf.webkit.org/ChangeLog 2014-04-03 07:07:02 UTC (rev 166700)
</span><span class="lines">@@ -1,3 +1,43 @@
</span><ins>+2014-04-03 Ryosuke Niwa <rniwa@webkit.org>
+
+ WebKitPerfMonitor: Y-axis adjustment is too aggressive
+ https://bugs.webkit.org/show_bug.cgi?id=130937
+
+ Reviewed by Andreas Kling.
+
+ Previously, adjusted min. and max. were defined as the two standards deviations away from EWMA of measured
+ results. This had two major problems:
+ 1. Two standard deviations can be too small to show the confidence interval for results.
+ 2. Sometimes baseline and target can be more than two standards deviations away.
+
+ Fixed the bug by completely rewriting the algorithm to compute the interval. Instead of blindly using two
+ standard deviations as margins, we keep adding quarter the standard deviation on each side until more than 90%
+ of points lie in the interval or we've expanded 4 standard deviations. Once this condition is met, we reduce
+ the margin on each side separately to reduce the empty space on either side.
+
+ A more rigorous approach would involve computing least squared value of results with respect to intervals
+ but that seems like an overkill for a simple UI problem; it's also computationally expensive.
+
+ * public/index.html:
+ (Chart..adjustedIntervalForRun): Extracted from computeYAxisBoundsToFitLines.
+ (Chart..computeYAxisBoundsToFitLines): Compute the min. and max. adjusted intervals out of adjusted intervals
+ for each runs (current, baseline, and target) so that at least one point from each set of results is shown.
+ We wouldn't see the difference between measured values versus baseline and target values otherwise.
+ * public/js/helper-classes.js:
+ (PerfTestResult.unscaledConfidenceIntervalDelta): Returns the default value if the confidence
+ interval delta cannot be computed.
+ (PerfTestResult.isInUnscaledInterval): Added. Returns true iff the confidence intervals lies
+ within the given interval.
+ (PerfTestRuns..filteredResults): Extracted from unscaledMeansForAllResults now that PerfTestRuns.min and
+ PerfTestRuns.max need to use both mean and confidence interval delta for each result.
+ (PerfTestRuns..unscaledMeansForAllResults):
+ (PerfTestRuns.min): Take the confidence interval delta into account.
+ (PerfTestRuns.max): Ditto.
+ (PerfTestRuns.countResults): Returns the number of results in the given time frame (> minTime).
+ (PerfTestRuns.countResultsInInterval): Returns the number of results whose confidence interval lie within the
+ given interval.
+ (PerfTestRuns.exponentialMovingArithmeticMean): Fixed the typo so that it actually computes the EWMA.
+
</ins><span class="cx"> 2014-03-31 Ryosuke Niwa <rniwa@webkit.org>
</span><span class="cx">
</span><span class="cx"> Some CSS tweaks after r166477 and r166479,
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicindexhtml"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/index.html (166699 => 166700)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/index.html 2014-04-03 06:21:27 UTC (rev 166699)
+++ trunk/Websites/perf.webkit.org/public/index.html 2014-04-03 07:07:02 UTC (rev 166700)
</span><span class="lines">@@ -314,25 +314,58 @@
</span><span class="cx"> }
</span><span class="cx"> };
</span><span class="cx">
</span><ins>+ function adjustedIntervalForRun(results, minTime, minRatioToFitInAdjustedInterval) {
+ if (!results)
+ return {min: Number.MAX_VALUE, max: Number.MIN_VALUE};
+ var degreeOfWeightingDecrease = 0.2;
+ var movingAverage = results.exponentialMovingArithmeticMean(minTime, degreeOfWeightingDecrease);
+ var resultsCount = results.countResults(minTime);
+ var adjustmentDelta = results.sampleStandardDeviation(minTime) / 4;
+ var adjustedMin = movingAverage;
+ var adjustedMax = movingAverage;
+ var adjustmentCount;
+ for (adjustmentCount = 0; adjustmentCount < 4 * 4; adjustmentCount++) { // Don't expand beyond 4 standard deviations.
+ adjustedMin -= adjustmentDelta;
+ adjustedMax += adjustmentDelta;
+ if (results.countResultsInInterval(minTime, adjustedMin, adjustedMax) / resultsCount >= minRatioToFitInAdjustedInterval)
+ break;
+ }
+ for (var i = 0; i < adjustmentCount; i++) {
+ if (results.countResultsInInterval(minTime, adjustedMin + adjustmentDelta, adjustedMax) / resultsCount < minRatioToFitInAdjustedInterval)
+ break;
+ adjustedMin += adjustmentDelta;
+ }
+ for (var i = 0; i < adjustmentCount; i++) {
+ if (results.countResultsInInterval(minTime, adjustedMin, adjustedMax - adjustmentDelta) / resultsCount < minRatioToFitInAdjustedInterval)
+ break;
+ adjustedMax -= adjustmentDelta;
+ }
+ return {min: adjustedMin, max: adjustedMax};
+ }
+
</ins><span class="cx"> function computeYAxisBoundsToFitLines(minTime, results, baseline, target) {
</span><del>- var stdevOfAllRuns = results.sampleStandardDeviation(minTime);
- var movingAverage = results.exponentialMovingArithmeticMean(minTime, /* alpha, the degree of weighting decrease */ 0.3);
- var min = results.min(minTime);
- var max = results.max(minTime);
-
</del><ins>+ var minOfAllRuns = results.min(minTime);
+ var maxOfAllRuns = results.max(minTime);
</ins><span class="cx"> if (baseline) {
</span><del>- min = Math.min(min, baseline.min(minTime));
- max = Math.max(max, baseline.max(minTime));
</del><ins>+ minOfAllRuns = Math.min(minOfAllRuns, baseline.min(minTime));
+ maxOfAllRuns = Math.max(maxOfAllRuns, baseline.max(minTime));
</ins><span class="cx"> }
</span><span class="cx"> if (target) {
</span><del>- min = Math.min(min, target.min(minTime));
- max = Math.max(max, target.max(minTime));
</del><ins>+ minOfAllRuns = Math.min(minOfAllRuns, target.min(minTime));
+ maxOfAllRuns = Math.max(maxOfAllRuns, target.max(minTime));
</ins><span class="cx"> }
</span><ins>+ var marginSize = (maxOfAllRuns - minOfAllRuns) * 0.1;
</ins><span class="cx">
</span><del>- var marginSize = (max - min) * 0.1;
- return {min: min - marginSize, max: max + marginSize,
- adjustedMin: Math.min(results.lastResult().mean() - marginSize, Math.max(movingAverage - stdevOfAllRuns * 2, min) - marginSize),
- adjustedMax: Math.max(results.lastResult().mean() + marginSize, Math.min(movingAverage + stdevOfAllRuns * 2, max) + marginSize) };
</del><ins>+ var minRatioToFitInAdjustedInterval = 0.9;
+ var intervalForResults = adjustedIntervalForRun(results, minTime, minRatioToFitInAdjustedInterval);
+ var intervalForBaseline = adjustedIntervalForRun(baseline, minTime, minRatioToFitInAdjustedInterval);
+ var intervalForTarget = adjustedIntervalForRun(target, minTime, minRatioToFitInAdjustedInterval);
+ var adjustedMin = Math.min(intervalForResults.min, intervalForBaseline.min, intervalForTarget.min);
+ var adjustedMax = Math.max(intervalForResults.max, intervalForBaseline.max, intervalForTarget.max);
+ var adjsutedMarginSize = (adjustedMax - adjustedMin) * 0.1;
+ return {min: minOfAllRuns - marginSize, max: maxOfAllRuns + marginSize,
+ adjustedMin: Math.max(minOfAllRuns - marginSize, adjustedMin - adjsutedMarginSize),
+ adjustedMax: Math.min(maxOfAllRuns + marginSize, adjustedMax + adjsutedMarginSize)};
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> function computeStatus(smallerIsBetter, lastResult, baseline, target) {
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicjshelperclassesjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/js/helper-classes.js (166699 => 166700)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/js/helper-classes.js 2014-04-03 06:21:27 UTC (rev 166699)
+++ trunk/Websites/perf.webkit.org/public/js/helper-classes.js 2014-04-03 07:07:02 UTC (rev 166700)
</span><span class="lines">@@ -8,9 +8,17 @@
</span><span class="cx"> this.confidenceIntervalDelta = function () {
</span><span class="cx"> return runs.scalingFactor() * this.unscaledConfidenceIntervalDelta();
</span><span class="cx"> }
</span><del>- this.unscaledConfidenceIntervalDelta = function () {
- return Statistics.confidenceIntervalDelta(0.95, result.iterationCount, result.sum, result.squareSum);
</del><ins>+ this.unscaledConfidenceIntervalDelta = function (defaultValue) {
+ var delta = Statistics.confidenceIntervalDelta(0.95, result.iterationCount, result.sum, result.squareSum);
+ if (isNaN(delta) && defaultValue !== undefined)
+ return defaultValue;
+ return delta;
</ins><span class="cx"> }
</span><ins>+ this.isInUnscaledInterval = function (min, max) {
+ var mean = this.unscaledMean();
+ var delta = this.unscaledConfidenceIntervalDelta(0);
+ return min <= mean - delta && mean + delta <= max;
+ }
</ins><span class="cx"> this.isBetterThan = function(other) { return runs.smallerIsBetter() == (this.mean() < other.mean()); }
</span><span class="cx"> this.relativeDifference = function(other) { return (other.mean() - this.mean()) / this.mean(); }
</span><span class="cx"> this.formattedRelativeDifference = function (other) { return Math.abs(this.relativeDifference(other) * 100).toFixed(2) + '%'; }
</span><span class="lines">@@ -169,25 +177,39 @@
</span><span class="cx"> this.lastResult = function () { return results[results.length - 1]; }
</span><span class="cx"> this.resultAt = function (i) { return results[i]; }
</span><span class="cx">
</span><del>- var unscaledMeansCache;
- var unscaledMeansCacheMinTime;
</del><ins>+ var resultsFilterCache;
+ var resultsFilterCacheMinTime;
+ function filteredResults(minTime) {
+ if (!minTime)
+ return results;
+ if (resultsFilterCacheMinTime != minTime) {
+ resultsFilterCache = results.filter(function (result) { return !minTime || result.build().time() >= minTime; });
+ resultsFilterCacheMinTime = minTime;
+ }
+ return resultsFilterCache;
+ }
+
</ins><span class="cx"> function unscaledMeansForAllResults(minTime) {
</span><del>- if (unscaledMeansCacheMinTime == minTime && unscaledMeansCache)
- return unscaledMeansCache;
- unscaledMeansCache = results.filter(function (result) { return !minTime || result.build().time() >= minTime; })
- .map(function (result) { return result.unscaledMean(); });
- unscaledMeansCacheMinTime = minTime;
- return unscaledMeansCache;
</del><ins>+ return filteredResults(minTime).map(function (result) { return result.unscaledMean(); });
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> this.min = function (minTime) {
</span><del>- return this.scalingFactor() * unscaledMeansForAllResults(minTime)
- .reduce(function (minSoFar, currentMean) { return Math.min(minSoFar, currentMean); }, Number.MAX_VALUE);
</del><ins>+ return this.scalingFactor() * filteredResults(minTime)
+ .reduce(function (minSoFar, result) { return Math.min(minSoFar, result.unscaledMean() - result.unscaledConfidenceIntervalDelta(0)); }, Number.MAX_VALUE);
</ins><span class="cx"> }
</span><span class="cx"> this.max = function (minTime, baselineName) {
</span><del>- return this.scalingFactor() * unscaledMeansForAllResults(minTime)
- .reduce(function (maxSoFar, currentMean) { return Math.max(maxSoFar, currentMean); }, Number.MIN_VALUE);
</del><ins>+ return this.scalingFactor() * filteredResults(minTime)
+ .reduce(function (maxSoFar, result) { return Math.max(maxSoFar, result.unscaledMean() + result.unscaledConfidenceIntervalDelta(0)); }, Number.MIN_VALUE);
</ins><span class="cx"> }
</span><ins>+ this.countResults = function (minTime) {
+ return unscaledMeansForAllResults(minTime).length;
+ }
+ this.countResultsInInterval = function (minTime, min, max) {
+ var unscaledMin = min / this.scalingFactor();
+ var unscaledMax = max / this.scalingFactor();
+ return filteredResults(minTime).reduce(function (count, currentResult) {
+ return count + (currentResult.isInUnscaledInterval(unscaledMin, unscaledMax) ? 1 : 0); }, 0);
+ }
</ins><span class="cx"> this.sampleStandardDeviation = function (minTime) {
</span><span class="cx"> var unscaledMeans = unscaledMeansForAllResults(minTime);
</span><span class="cx"> return this.scalingFactor() * Statistics.sampleStandardDeviation(unscaledMeans.length, Statistics.sum(unscaledMeans), Statistics.squareSum(unscaledMeans));
</span><span class="lines">@@ -196,7 +218,7 @@
</span><span class="cx"> var unscaledMeans = unscaledMeansForAllResults(minTime);
</span><span class="cx"> if (!unscaledMeans.length)
</span><span class="cx"> return NaN;
</span><del>- return this.scalingFactor() * unscaledMeans.reduce(function (movingAverage, currentMean) { return alpha * movingAverage + (1 - alpha) * movingAverage; });
</del><ins>+ return this.scalingFactor() * unscaledMeans.reduce(function (movingAverage, currentMean) { return alpha * currentMean + (1 - alpha) * movingAverage; });
</ins><span class="cx"> }
</span><span class="cx"> this.hasConfidenceInterval = function () { return !isNaN(this.lastResult().unscaledConfidenceIntervalDelta()); }
</span><span class="cx"> var meanPlotCache;
</span></span></pre>
</div>
</div>
</body>
</html>