<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[196372] trunk/PerformanceTests</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/196372">196372</a></dd>
<dt>Author</dt> <dd>jonlee@apple.com</dd>
<dt>Date</dt> <dd>2016-02-10 11:22:08 -0800 (Wed, 10 Feb 2016)</dd>
</dl>
<h3>Log Message</h3>
<pre>Add a ramp controller
https://bugs.webkit.org/show_bug.cgi?id=154028
Provisionally reviewed by Said Abou-Hallawa.
Enhance the graph to include a complexity-fps graph, in addition
to the time graph.
* Animometer/developer.html: Add a ramp option.
* Animometer/resources/debug-runner/animometer.css: Update the style.
* Animometer/resources/strings.js: Flatten the Strings.text constants.
* Animometer/resources/debug-runner/animometer.js:
(ResultsTable.call._addGraphButton): Refactor.
(ResultsTable.call._addTest): Add regression data.
(benchmarkController): Add a form that allows the user to switch between the two forms,
Add a form that allows the user to toggle different data. Hide certain header columns
depending on the selected controller.
* Animometer/resources/debug-runner/graph.js: Add the complexity regressions.
* Animometer/resources/debug-runner/tests.js: Add headers for the ramp results.
* Animometer/resources/runner/animometer.js:
(ResultsTable): If a header is disabled don't include them in _flattenedHeaders.
* Animometer/tests/resources/main.js:
(Controller): Allow options to specify the capacity for sample arrays.
(Regression): A piecewise regression that tries to fit a slope and a flat profile.
(_calculateRegression): Options can fix the slope and bias when calculating the minimal
error. Sweep across the samples in time (which could be backward depending on the controller)
and calculate the intersection point.
(RampController): This controller assumes that the target frame rate is below
58 FPS. It runs in two stages. The first stage quickly determines the order of
magnitude of objects needed to stress the system by the setting the complexity
to increasingly difficult tiers. Perform a series of ramps descending from a
high-water mark of complexity. The complexity needed to reach the target frame
length is done by performing a piecewise regression on each ramp, and track a
running average of these values. For the next ramp, make that running average
the center of the ramp. With a minimum complexity of 0, the high-water mark is
twice that average. The score is based on the highest complexity that can
reach 60 fps.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkPerformanceTestsAnimometerdeveloperhtml">trunk/PerformanceTests/Animometer/developer.html</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesdebugrunneranimometercss">trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.css</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesdebugrunneranimometerjs">trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesdebugrunnergraphjs">trunk/PerformanceTests/Animometer/resources/debug-runner/graph.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesdebugrunnertestsjs">trunk/PerformanceTests/Animometer/resources/debug-runner/tests.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesrunneranimometerjs">trunk/PerformanceTests/Animometer/resources/runner/animometer.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesstringsjs">trunk/PerformanceTests/Animometer/resources/strings.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometertestsresourcesmainjs">trunk/PerformanceTests/Animometer/tests/resources/main.js</a></li>
<li><a href="#trunkPerformanceTestsChangeLog">trunk/PerformanceTests/ChangeLog</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkPerformanceTestsAnimometerdeveloperhtml"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/developer.html (196371 => 196372)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/developer.html 2016-02-10 19:22:07 UTC (rev 196371)
+++ trunk/PerformanceTests/Animometer/developer.html 2016-02-10 19:22:08 UTC (rev 196372)
</span><span class="lines">@@ -46,6 +46,7 @@
</span><span class="cx"> <ul>
</span><span class="cx"> <li><label><input name="adjustment" type="radio" value="step"> Keep at a fixed complexity, then make a big step</label></li>
</span><span class="cx"> <li><label><input name="adjustment" type="radio" value="adaptive" checked> Maintain target FPS</label></li>
</span><ins>+ <li><label><input name="adjustment" type="radio" value="ramp"> Ramp</label></li>
</ins><span class="cx"> </ul>
</span><span class="cx"> </li>
</span><span class="cx"> <li>
</span><span class="lines">@@ -103,6 +104,12 @@
</span><span class="cx"> <h1>Graph:</h1>
</span><span class="cx"> </header>
</span><span class="cx"> <nav>
</span><ins>+ <form name="graph-type">
+ <ul>
+ <li><label><input type="radio" name="graph-type" value="time" checked> Time graph</label></li>
+ <li><label><input type="radio" name="graph-type" value="complexity"> Complexity graph</label></li>
+ </ul>
+ </form>
</ins><span class="cx"> <form name="time-graph-options">
</span><span class="cx"> <ul>
</span><span class="cx"> <li><label><input type="checkbox" name="markers" checked> Markers</label>
</span><span class="lines">@@ -114,8 +121,20 @@
</span><span class="cx"> <span class="rawFPS"></span></li>
</span><span class="cx"> <li><label><input type="checkbox" name="filteredFPS" checked> Filtered FPS</label>
</span><span class="cx"> <span class="filteredFPS"></span></li>
</span><ins>+ <li><label><input type="checkbox" name="regressions" checked> Regressions</label>
</ins><span class="cx"> </ul>
</span><span class="cx"> </form>
</span><ins>+ <form name="complexity-graph-options">
+ <ul class="series">
+ <li><label><input type="checkbox" name="series-raw" checked> Series raw</label></li>
+ <li><label><input type="checkbox" name="series-average" checked> Series average</label></li>
+ </ul>
+ <ul>
+ <li><label><input type="checkbox" name="regression-time-score" checked> Ramp regression score</label>
+ <li><label><input type="checkbox" name="complexity-regression-aggregate-raw" checked> Regression, series raw</label>
+ <li><label><input type="checkbox" name="complexity-regression-aggregate-average" checked> Regression, series average</label>
+ </ul>
+ </form>
</ins><span class="cx"> </nav>
</span><span class="cx"> <p class="score"></p>
</span><span class="cx"> <p class="mean"></p>
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesdebugrunneranimometercss"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.css (196371 => 196372)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.css 2016-02-10 19:22:07 UTC (rev 196371)
+++ trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.css 2016-02-10 19:22:08 UTC (rev 196372)
</span><span class="lines">@@ -356,20 +356,16 @@
</span><span class="cx"> position: absolute;
</span><span class="cx"> top: 1.5em;
</span><span class="cx"> right: 0;
</span><del>- font-size: .8em;
- width: 25%;
</del><ins>+ font-size: .7em;
+ width: 23em;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #test-graph nav ul {
</span><del>- margin: 0 30px 0 0;
</del><ins>+ margin: 0 30px 1em 0;
</ins><span class="cx"> padding: 0;
</span><span class="cx"> list-style: none;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-#test-graph nav ul ul {
- padding-left: 2em;
-}
-
</del><span class="cx"> #test-graph nav li {
</span><span class="cx"> padding: .1em 0;
</span><span class="cx"> }
</span><span class="lines">@@ -403,6 +399,36 @@
</span><span class="cx"> shape-rendering: crispEdges;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+.axis text {
+ fill: #999;
+}
+
+.yLeft.axis text {
+ fill: #7add49;
+}
+.yLeft.axis path,
+.yLeft.axis line {
+ stroke: #7add49;
+}
+.yRight.axis text {
+ fill: #fa4925;
+}
+.yRight.axis path,
+.yRight.axis line {
+ stroke: #fa4925;
+}
+
+.axis.complexity .tick line {
+ stroke: rgba(200, 200, 200, .6);
+ stroke-width: 2px;
+}
+
+.axis.complexity .domain,
+.axis.complexity text {
+ stroke: transparent;
+ fill: transparent;
+}
+
</ins><span class="cx"> .marker line {
</span><span class="cx"> stroke: #5493D6;
</span><span class="cx"> }
</span><span class="lines">@@ -417,7 +443,7 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> .mean.complexity polygon {
</span><del>- fill: hsla(100, 69%, 58%, .1);
</del><ins>+ fill: hsla(100, 69%, 58%, .05);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> .target-fps {
</span><span class="lines">@@ -435,6 +461,15 @@
</span><span class="cx"> fill: hsla(10, 96%, 56%, .1);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+#regressions line {
+ stroke: rgba(200, 200, 200, .8);
+ stroke-width: 2px;
+}
+
+#regressions circle {
+ fill: rgba(200, 200, 200, .8);
+}
+
</ins><span class="cx"> .cursor line {
</span><span class="cx"> stroke: rgb(250, 250, 250);
</span><span class="cx"> stroke-width: 1px;
</span><span class="lines">@@ -471,3 +506,25 @@
</span><span class="cx"> #rawFPS circle {
</span><span class="cx"> fill: rgb(250, 73, 37);
</span><span class="cx"> }
</span><ins>+
+#complexity-graph .regression line {
+ stroke: rgba(253, 253, 253, .8);
+ stroke-width: 2px;
+}
+
+#complexity-graph .regression circle {
+ fill: rgba(253, 253, 253, .8);
+}
+
+#complexity-graph .regression polygon {
+ fill: rgba(253, 253, 253, .05);
+}
+
+#complexity-graph .series.average circle {
+ fill: hsl(170, 96%, 56%);
+}
+
+#complexity-graph .series.average line {
+ stroke: hsla(170, 96%, 56%, .2);
+ stroke-width: 2px;
+}
</ins></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesdebugrunneranimometerjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.js (196371 => 196372)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.js 2016-02-10 19:22:07 UTC (rev 196371)
+++ trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.js 2016-02-10 19:22:08 UTC (rev 196372)
</span><span class="lines">@@ -35,7 +35,7 @@
</span><span class="cx">
</span><span class="cx"> button.addEventListener("click", function() {
</span><span class="cx"> var graphData = {
</span><del>- axes: [Strings.text.experiments.complexity, Strings.text.experiments.frameRate],
</del><ins>+ axes: [Strings.text.complexity, Strings.text.frameRate],
</ins><span class="cx"> samples: data,
</span><span class="cx"> complexityAverageSamples: testResults[Strings.json.complexityAverageSamples],
</span><span class="cx"> averages: {},
</span><span class="lines">@@ -48,6 +48,9 @@
</span><span class="cx">
</span><span class="cx"> [
</span><span class="cx"> Strings.json.score,
</span><ins>+ Strings.json.regressions.timeRegressions,
+ Strings.json.regressions.complexityRegression,
+ Strings.json.regressions.complexityAverageRegression,
</ins><span class="cx"> Strings.json.targetFrameLength
</span><span class="cx"> ].forEach(function(key) {
</span><span class="cx"> if (testResults[key])
</span><span class="lines">@@ -57,7 +60,7 @@
</span><span class="cx"> benchmarkController.showTestGraph(testName, graphData);
</span><span class="cx"> });
</span><span class="cx">
</span><del>- button.textContent = Strings.text.results.graph + "...";
</del><ins>+ button.textContent = Strings.text.graph + "...";
</ins><span class="cx"> },
</span><span class="cx">
</span><span class="cx"> _isNoisyMeasurement: function(jsonExperiment, data, measurement, options)
</span><span class="lines">@@ -105,7 +108,7 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> var td = Utilities.createElement("td", { class: className }, row);
</span><del>- if (header.title == Strings.text.results.graph) {
</del><ins>+ if (header.title == Strings.text.graph) {
</ins><span class="cx"> this._addGraphButton(td, testName, testResults);
</span><span class="cx"> } else if (!("text" in header)) {
</span><span class="cx"> td.textContent = testResults[header.title];
</span><span class="lines">@@ -451,7 +454,9 @@
</span><span class="cx"> initialize: function()
</span><span class="cx"> {
</span><span class="cx"> document.forms["benchmark-options"].addEventListener("change", benchmarkController.onBenchmarkOptionsChanged, true);
</span><ins>+ document.forms["graph-type"].addEventListener("change", benchmarkController.onGraphTypeChanged, true);
</ins><span class="cx"> document.forms["time-graph-options"].addEventListener("change", benchmarkController.onTimeGraphOptionsChanged, true);
</span><ins>+ document.forms["complexity-graph-options"].addEventListener("change", benchmarkController.onComplexityGraphOptionsChanged, true);
</ins><span class="cx"> optionsManager.updateUIFromLocalStorage();
</span><span class="cx"> suitesManager.createElements();
</span><span class="cx"> suitesManager.updateUIFromLocalStorage();
</span><span class="lines">@@ -474,6 +479,12 @@
</span><span class="cx"> {
</span><span class="cx"> var options = optionsManager.updateLocalStorageFromUI();
</span><span class="cx"> var suites = suitesManager.updateLocalStorageFromUI();
</span><ins>+ if (options["adjustment"] == "ramp") {
+ Headers.details[2].disabled = true;
+ } else {
+ Headers.details[3].disabled = true;
+ Headers.details[4].disabled = true;
+ }
</ins><span class="cx"> this._startBenchmark(suites, options, "running-test");
</span><span class="cx"> },
</span><span class="cx">
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesdebugrunnergraphjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/debug-runner/graph.js (196371 => 196372)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/debug-runner/graph.js 2016-02-10 19:22:07 UTC (rev 196371)
+++ trunk/PerformanceTests/Animometer/resources/debug-runner/graph.js 2016-02-10 19:22:08 UTC (rev 196372)
</span><span class="lines">@@ -14,6 +14,14 @@
</span><span class="cx"> this.createTimeGraph(graphData, margins, size);
</span><span class="cx"> this.onTimeGraphOptionsChanged();
</span><span class="cx">
</span><ins>+ var hasComplexityRegression = !!graphData.complexityRegression;
+ this._showOrHideNodes(hasComplexityRegression, "form[name=graph-type]");
+ if (hasComplexityRegression) {
+ document.forms["graph-type"].elements["type"] = "complexity";
+ this.createComplexityGraph(graphData, margins, size);
+ this.onComplexityGraphOptionsChanged();
+ }
+
</ins><span class="cx"> this.onGraphTypeChanged();
</span><span class="cx"> },
</span><span class="cx">
</span><span class="lines">@@ -41,6 +49,165 @@
</span><span class="cx"> .attr("y2", line[3]);
</span><span class="cx"> },
</span><span class="cx">
</span><ins>+ _addRegression: function(data, svg, xScale, yScale)
+ {
+ svg.append("circle")
+ .attr("cx", xScale(data.segment2[1][0]))
+ .attr("cy", yScale(data.segment2[1][1]))
+ .attr("r", 5);
+ this._addRegressionLine(svg, xScale, yScale, data.segment1, data.stdev);
+ this._addRegressionLine(svg, xScale, yScale, data.segment2, data.stdev);
+ },
+
+ createComplexityGraph: function(graphData, margins, size)
+ {
+ var svg = d3.select("#test-graph-data").append("svg")
+ .attr("id", "complexity-graph")
+ .attr("class", "hidden")
+ .attr("width", size.width + margins.left + margins.right)
+ .attr("height", size.height + margins.top + margins.bottom)
+ .append("g")
+ .attr("transform", "translate(" + margins.left + "," + margins.top + ")");
+
+ var xMin = 100000, xMax = 0;
+ if (graphData.timeRegressions) {
+ graphData.timeRegressions.forEach(function(regression) {
+ for (var i = regression.startIndex; i <= regression.endIndex; ++i) {
+ xMin = Math.min(xMin, graphData.samples[i].complexity);
+ xMax = Math.max(xMax, graphData.samples[i].complexity);
+ }
+ });
+ } else {
+ xMin = d3.min(graphData.samples, function(s) { return s.complexity; });
+ xMax = d3.max(graphData.samples, function(s) { return s.complexity; });
+ }
+
+ var xScale = d3.scale.linear()
+ .range([0, size.width])
+ .domain([xMin, xMax]);
+ var yScale = d3.scale.linear()
+ .range([size.height, 0])
+ .domain([1000/20, 1000/60]);
+
+ var xAxis = d3.svg.axis()
+ .scale(xScale)
+ .orient("bottom");
+ var yAxis = d3.svg.axis()
+ .scale(yScale)
+ .tickValues([1000/20, 1000/25, 1000/30, 1000/35, 1000/40, 1000/45, 1000/50, 1000/55, 1000/60])
+ .tickFormat(function(d) { return (1000 / d).toFixed(0); })
+ .orient("left");
+
+ // x-axis
+ svg.append("g")
+ .attr("class", "x axis")
+ .attr("transform", "translate(0," + size.height + ")")
+ .call(xAxis);
+
+ // y-axis
+ svg.append("g")
+ .attr("class", "y axis")
+ .call(yAxis);
+
+ // time-based regression
+ var mean = svg.append("g")
+ .attr("class", "mean complexity");
+ var complexity = graphData.averages[Strings.json.experiments.complexity];
+ this._addRegressionLine(mean, xScale, yScale, [[complexity.average, yScale.domain()[0]], [complexity.average, yScale.domain()[1]]], complexity.stdev, true);
+
+ // regression
+ this._addRegression(graphData.complexityRegression, svg.append("g").attr("class", "regression raw"), xScale, yScale);
+ this._addRegression(graphData.complexityAverageRegression, svg.append("g").attr("class", "regression average"), xScale, yScale);
+
+ var svgGroup = svg.append("g")
+ .attr("class", "series raw");
+ var seriesCounter = 0;
+ graphData.timeRegressions.forEach(function(regression, i) {
+ seriesCounter++;
+ var group = svgGroup.append("g")
+ .attr("class", "series-" + seriesCounter)
+ .attr("fill", "hsl(" + (i / graphData.timeRegressions.length * 360).toFixed(0) + ", 96%, 56%)");
+ group.selectAll("circle")
+ .data(graphData.samples)
+ .enter()
+ .append("circle")
+ .filter(function(d, i) { return i >= regression.startIndex && i <= regression.endIndex; })
+ .attr("cx", function(d) { return xScale(d.complexity); })
+ .attr("cy", function(d) { return yScale(d.frameLength); })
+ .attr("r", 2);
+ });
+
+ group = svg.append("g")
+ .attr("class", "series average")
+ .selectAll("circle")
+ .data(graphData.complexityAverageSamples)
+ .enter();
+ group.append("circle")
+ .attr("cx", function(d) { return xScale(d.complexity); })
+ .attr("cy", function(d) { return yScale(d.frameLength); })
+ .attr("r", 3)
+ group.append("line")
+ .attr("x1", function(d) { return xScale(d.complexity); })
+ .attr("x2", function(d) { return xScale(d.complexity); })
+ .attr("y1", function(d) { return yScale(d.frameLength - d.stdev); })
+ .attr("y2", function(d) { return yScale(d.frameLength + d.stdev); });
+
+ // Cursor
+ var cursorGroup = svg.append("g").attr("class", "cursor hidden");
+ cursorGroup.append("line")
+ .attr("class", "x")
+ .attr("x1", 0)
+ .attr("x2", 0)
+ .attr("y1", yScale(yAxis.scale().domain()[0]) + 10)
+ .attr("y2", yScale(yAxis.scale().domain()[1]));
+ cursorGroup.append("line")
+ .attr("class", "y")
+ .attr("x1", xScale(0) - 10)
+ .attr("x2", xScale(xAxis.scale().domain()[1]))
+ .attr("y1", 0)
+ .attr("y2", 0)
+ cursorGroup.append("text")
+ .attr("class", "label x")
+ .attr("x", 0)
+ .attr("y", yScale(yAxis.scale().domain()[0]) + 15)
+ .attr("baseline-shift", "-100%")
+ .attr("text-anchor", "middle");
+ cursorGroup.append("text")
+ .attr("class", "label y")
+ .attr("x", xScale(0) - 15)
+ .attr("y", 0)
+ .attr("baseline-shift", "-30%")
+ .attr("text-anchor", "end");
+ // Area to handle mouse events
+ var area = svg.append("rect")
+ .attr("fill", "transparent")
+ .attr("x", 0)
+ .attr("y", 0)
+ .attr("width", size.width)
+ .attr("height", size.height);
+
+ area.on("mouseover", function() {
+ document.querySelector("#complexity-graph .cursor").classList.remove("hidden");
+ }).on("mouseout", function() {
+ document.querySelector("#complexity-graph .cursor").classList.add("hidden");
+ }).on("mousemove", function() {
+ var location = d3.mouse(this);
+ var location_domain = [xScale.invert(location[0]), yScale.invert(location[1])];
+ cursorGroup.select("line.x")
+ .attr("x1", location[0])
+ .attr("x2", location[0]);
+ cursorGroup.select("text.x")
+ .attr("x", location[0])
+ .text(location_domain[0].toFixed(1));
+ cursorGroup.select("line.y")
+ .attr("y1", location[1])
+ .attr("y2", location[1]);
+ cursorGroup.select("text.y")
+ .attr("y", location[1])
+ .text((1000 / location_domain[1]).toFixed(1));
+ });
+ },
+
</ins><span class="cx"> createTimeGraph: function(graphData, margins, size)
</span><span class="cx"> {
</span><span class="cx"> var svg = d3.select("#test-graph-data").append("svg")
</span><span class="lines">@@ -60,6 +227,11 @@
</span><span class="cx"> Math.min(d3.min(graphData.samples, function(s) { return s.time; }), 0),
</span><span class="cx"> d3.max(graphData.samples, function(s) { return s.time; })]);
</span><span class="cx"> var complexityMax = d3.max(graphData.samples, function(s) { return s.complexity; });
</span><ins>+ if (graphData.timeRegressions) {
+ complexityMax = Math.max.apply(Math, graphData.timeRegressions.map(function(regression) {
+ return regression.maxComplexity || 0;
+ }));
+ }
</ins><span class="cx">
</span><span class="cx"> var yLeft = d3.scale.linear()
</span><span class="cx"> .range([size.height, 0])
</span><span class="lines">@@ -98,7 +270,7 @@
</span><span class="cx">
</span><span class="cx"> // yLeft-axis
</span><span class="cx"> svg.append("g")
</span><del>- .attr("class", "y axis")
</del><ins>+ .attr("class", "yLeft axis")
</ins><span class="cx"> .attr("fill", "#7ADD49")
</span><span class="cx"> .call(yAxisLeft)
</span><span class="cx"> .append("text")
</span><span class="lines">@@ -112,17 +284,17 @@
</span><span class="cx">
</span><span class="cx"> // yRight-axis
</span><span class="cx"> svg.append("g")
</span><del>- .attr("class", "y axis")
</del><ins>+ .attr("class", "yRight axis")
</ins><span class="cx"> .attr("fill", "#FA4925")
</span><span class="cx"> .attr("transform", "translate(" + size.width + ", 0)")
</span><span class="cx"> .call(yAxisRight)
</span><span class="cx"> .append("text")
</span><span class="cx"> .attr("class", "label")
</span><del>- .attr("transform", "rotate(-90)")
- .attr("y", 6)
</del><ins>+ .attr("x", 9)
+ .attr("y", -20)
</ins><span class="cx"> .attr("fill", "#FA4925")
</span><span class="cx"> .attr("dy", ".71em")
</span><del>- .style("text-anchor", "end")
</del><ins>+ .style("text-anchor", "start")
</ins><span class="cx"> .text(axes[1]);
</span><span class="cx">
</span><span class="cx"> // marks
</span><span class="lines">@@ -209,6 +381,41 @@
</span><span class="cx"> addData("rawFPS", allData, function(d) { return yRight(d.frameLength); }, 1);
</span><span class="cx"> addData("filteredFPS", filteredData, function(d) { return yRight(d.smoothedFrameLength); }, 2);
</span><span class="cx">
</span><ins>+ // regressions
+ var regressionGroup = svg.append("g")
+ .attr("id", "regressions");
+ if (graphData.timeRegressions) {
+ var complexities = [];
+ graphData.timeRegressions.forEach(function (regression) {
+ regressionGroup.append("line")
+ .attr("x1", x(regression.segment1[0][0]))
+ .attr("x2", x(regression.segment1[1][0]))
+ .attr("y1", yRight(regression.segment1[0][1]))
+ .attr("y2", yRight(regression.segment1[1][1]));
+ regressionGroup.append("line")
+ .attr("x1", x(regression.segment2[0][0]))
+ .attr("x2", x(regression.segment2[1][0]))
+ .attr("y1", yRight(regression.segment2[0][1]))
+ .attr("y2", yRight(regression.segment2[1][1]));
+ // inflection point
+ regressionGroup.append("circle")
+ .attr("cx", x(regression.segment1[1][0]))
+ .attr("cy", yLeft(regression.complexity))
+ .attr("r", 5);
+ complexities.push(regression.complexity);
+ });
+ if (complexities.length) {
+ var yLeftComplexities = d3.svg.axis()
+ .scale(yLeft)
+ .tickValues(complexities)
+ .tickSize(10)
+ .orient("left");
+ svg.append("g")
+ .attr("class", "complexity yLeft axis")
+ .call(yLeftComplexities);
+ }
+ }
+
</ins><span class="cx"> // Area to handle mouse events
</span><span class="cx"> var area = svg.append("rect")
</span><span class="cx"> .attr("fill", "transparent")
</span><span class="lines">@@ -291,6 +498,15 @@
</span><span class="cx"> }
</span><span class="cx"> },
</span><span class="cx">
</span><ins>+ onComplexityGraphOptionsChanged: function() {
+ var form = document.forms["complexity-graph-options"].elements;
+ benchmarkController._showOrHideNodes(form["series-raw"].checked, "#complexity-graph .series.raw");
+ benchmarkController._showOrHideNodes(form["series-average"].checked, "#complexity-graph .series.average");
+ benchmarkController._showOrHideNodes(form["regression-time-score"].checked, "#complexity-graph .mean.complexity");
+ benchmarkController._showOrHideNodes(form["complexity-regression-aggregate-raw"].checked, "#complexity-graph .regression.raw");
+ benchmarkController._showOrHideNodes(form["complexity-regression-aggregate-average"].checked, "#complexity-graph .regression.average");
+ },
+
</ins><span class="cx"> onTimeGraphOptionsChanged: function() {
</span><span class="cx"> var form = document.forms["time-graph-options"].elements;
</span><span class="cx"> benchmarkController._showOrHideNodes(form["markers"].checked, ".marker");
</span><span class="lines">@@ -301,11 +517,14 @@
</span><span class="cx"> },
</span><span class="cx">
</span><span class="cx"> onGraphTypeChanged: function() {
</span><ins>+ var form = document.forms["graph-type"].elements;
</ins><span class="cx"> var graphData = document.getElementById("test-graph-data").graphData;
</span><del>- var isTimeSelected = true;
</del><ins>+ var isTimeSelected = form["graph-type"].value == "time";
</ins><span class="cx">
</span><span class="cx"> benchmarkController._showOrHideNodes(isTimeSelected, "#time-graph");
</span><span class="cx"> benchmarkController._showOrHideNodes(isTimeSelected, "form[name=time-graph-options]");
</span><ins>+ benchmarkController._showOrHideNodes(!isTimeSelected, "#complexity-graph");
+ benchmarkController._showOrHideNodes(!isTimeSelected, "form[name=complexity-graph-options]");
</ins><span class="cx">
</span><span class="cx"> var score, mean;
</span><span class="cx"> if (isTimeSelected) {
</span><span class="lines">@@ -326,6 +545,19 @@
</span><span class="cx"> regression.concern.toFixed(2)]);
</span><span class="cx"> }
</span><span class="cx"> mean = mean.join("");
</span><ins>+ } else {
+ score = [
+ "raw: ",
+ graphData.complexityRegression.complexity.toFixed(2),
+ ", average: ",
+ graphData.complexityAverageRegression.complexity.toFixed(2)].join("");
+
+ mean = [
+ "raw: ±",
+ graphData.complexityRegression.stdev.toFixed(2),
+ "ms, average: ±",
+ graphData.complexityAverageRegression.stdev.toFixed(2),
+ "ms"].join("");
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> sectionsManager.setSectionScore("test-graph", score, mean);
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesdebugrunnertestsjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/debug-runner/tests.js (196371 => 196372)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/debug-runner/tests.js 2016-02-10 19:22:07 UTC (rev 196371)
+++ trunk/PerformanceTests/Animometer/resources/debug-runner/tests.js 2016-02-10 19:22:08 UTC (rev 196372)
</span><span class="lines">@@ -1,10 +1,10 @@
</span><span class="cx"> Utilities.extendObject(Headers, {
</span><span class="cx"> details: [
</span><span class="cx"> {
</span><del>- title: Strings.text.results.graph
</del><ins>+ title: Strings.text.graph
</ins><span class="cx"> },
</span><span class="cx"> {
</span><del>- title: Strings.text.experiments.complexity,
</del><ins>+ title: Strings.text.complexity,
</ins><span class="cx"> children:
</span><span class="cx"> [
</span><span class="cx"> {
</span><span class="lines">@@ -32,7 +32,7 @@
</span><span class="cx"> ]
</span><span class="cx"> },
</span><span class="cx"> {
</span><del>- title: Strings.text.experiments.frameRate,
</del><ins>+ title: Strings.text.frameRate,
</ins><span class="cx"> children:
</span><span class="cx"> [
</span><span class="cx"> {
</span><span class="lines">@@ -65,6 +65,50 @@
</span><span class="cx"> }
</span><span class="cx"> ]
</span><span class="cx"> },
</span><ins>+ {
+ title: Strings.text.mergedRawComplexity,
+ children:
+ [
+ {
+ text: function(data) {
+ return data[Strings.json.regressions.complexityRegression][Strings.json.regressions.complexity].toFixed(2);
+ },
+ className: "average"
+ },
+ {
+ text: function(data) {
+ return [
+ "± ",
+ data[Strings.json.regressions.complexityRegression][Strings.json.measurements.stdev].toFixed(2),
+ "ms"
+ ].join("");
+ },
+ className: "stdev"
+ }
+ ]
+ },
+ {
+ title: Strings.text.mergedAverageComplexity,
+ children:
+ [
+ {
+ text: function(data) {
+ return data[Strings.json.regressions.complexityAverageRegression][Strings.json.regressions.complexity].toFixed(2);
+ },
+ className: "average"
+ },
+ {
+ text: function(data) {
+ return [
+ "± ",
+ data[Strings.json.regressions.complexityAverageRegression][Strings.json.measurements.stdev].toFixed(2),
+ "ms"
+ ].join("");
+ },
+ className: "stdev"
+ }
+ ]
+ },
</ins><span class="cx"> ]
</span><span class="cx"> })
</span><span class="cx">
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesrunneranimometerjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/runner/animometer.js (196371 => 196372)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/runner/animometer.js 2016-02-10 19:22:07 UTC (rev 196371)
+++ trunk/PerformanceTests/Animometer/resources/runner/animometer.js 2016-02-10 19:22:08 UTC (rev 196372)
</span><span class="lines">@@ -70,12 +70,19 @@
</span><span class="cx">
</span><span class="cx"> this._flattenedHeaders = [];
</span><span class="cx"> this._headers.forEach(function(header) {
</span><ins>+ if (header.disabled)
+ return;
+
</ins><span class="cx"> if (header.children)
</span><span class="cx"> this._flattenedHeaders = this._flattenedHeaders.concat(header.children);
</span><span class="cx"> else
</span><span class="cx"> this._flattenedHeaders.push(header);
</span><span class="cx"> }, this);
</span><span class="cx">
</span><ins>+ this._flattenedHeaders = this._flattenedHeaders.filter(function (header) {
+ return !header.disabled;
+ });
+
</ins><span class="cx"> this.clear();
</span><span class="cx"> }, {
</span><span class="cx">
</span><span class="lines">@@ -90,8 +97,11 @@
</span><span class="cx"> var row = Utilities.createElement("tr", {}, thead);
</span><span class="cx">
</span><span class="cx"> this._headers.forEach(function (header) {
</span><ins>+ if (header.disabled)
+ return;
+
</ins><span class="cx"> var th = Utilities.createElement("th", {}, row);
</span><del>- if (header.title != Strings.text.results.graph)
</del><ins>+ if (header.title != Strings.text.graph)
</ins><span class="cx"> th.textContent = header.title;
</span><span class="cx"> if (header.children)
</span><span class="cx"> th.colSpan = header.children.length;
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesstringsjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/strings.js (196371 => 196372)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/strings.js 2016-02-10 19:22:07 UTC (rev 196371)
+++ trunk/PerformanceTests/Animometer/resources/strings.js 2016-02-10 19:22:08 UTC (rev 196372)
</span><span class="lines">@@ -4,27 +4,16 @@
</span><span class="cx"> score: "Score",
</span><span class="cx"> samples: "Samples",
</span><span class="cx">
</span><del>- experiments: {
- complexity: "Complexity",
- frameRate: "FPS"
- },
-
- measurements: {
- average: "Avg.",
- concern: "W.5%",
- stdev: "Std.",
- percent: "%"
- },
-
- results: {
- results: "Results",
- graph: "Graph",
- json: "JSON"
- }
</del><ins>+ complexity: "Complexity",
+ frameRate: "FPS",
+ mergedRawComplexity: "Merged raw",
+ mergedAverageComplexity: "Merged average",
+ graph: "Graph"
</ins><span class="cx"> },
</span><span class="cx"> json: {
</span><span class="cx"> score: "score",
</span><span class="cx"> samples: "samples",
</span><ins>+ complexityAverageSamples: "complexityAverageSamples",
</ins><span class="cx"> marks: "marks",
</span><span class="cx">
</span><span class="cx"> targetFrameLength: "targetFrameLength",
</span><span class="lines">@@ -36,6 +25,19 @@
</span><span class="cx"> frameRate: "frameRate"
</span><span class="cx"> },
</span><span class="cx">
</span><ins>+ regressions: {
+ timeRegressions: "timeRegressions",
+ complexity: "complexity",
+ maxComplexity: "maxComplexity",
+ startIndex: "startIndex",
+ endIndex: "endIndex",
+
+ complexityRegression: "complexityRegression",
+ complexityAverageRegression: "complexityAverageRegression",
+ segment1: "segment1",
+ segment2: "segment2"
+ },
+
</ins><span class="cx"> measurements: {
</span><span class="cx"> average: "average",
</span><span class="cx"> concern: "concern",
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometertestsresourcesmainjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/tests/resources/main.js (196371 => 196372)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/tests/resources/main.js 2016-02-10 19:22:07 UTC (rev 196371)
+++ trunk/PerformanceTests/Animometer/tests/resources/main.js 2016-02-10 19:22:08 UTC (rev 196372)
</span><span class="lines">@@ -6,7 +6,8 @@
</span><span class="cx"> this._startTimestamp = 0;
</span><span class="cx"> this._endTimestamp = options["test-interval"];
</span><span class="cx"> // Default data series: timestamp, complexity, estimatedFrameLength
</span><del>- this._sampler = new Sampler(options["series-count"] || 3, (60 * options["test-interval"] / 1000), this);
</del><ins>+ var sampleSize = options["sample-capacity"] || (60 * options["test-interval"] / 1000);
+ this._sampler = new Sampler(options["series-count"] || 3, sampleSize, this);
</ins><span class="cx"> this._marks = {};
</span><span class="cx">
</span><span class="cx"> this._frameLengthEstimator = new SimpleKalmanEstimator(options["kalman-process-error"], options["kalman-measurement-error"]);
</span><span class="lines">@@ -258,6 +259,421 @@
</span><span class="cx"> }
</span><span class="cx"> });
</span><span class="cx">
</span><ins>+Regression = Utilities.createClass(
+ function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
+ {
+ var slope = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
+ shouldClip: true,
+ s1: 1000/60,
+ t1: 0
+ });
+ var flat = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
+ shouldClip: true,
+ t1: 0,
+ t2: 0
+ });
+ var desired;
+ if (slope.error < flat.error)
+ desired = slope;
+ else
+ desired = flat;
+
+ this.startIndex = Math.min(startIndex, endIndex);
+ this.endIndex = Math.max(startIndex, endIndex);
+
+ this.complexity = desired.complexity;
+ this.s1 = desired.s1;
+ this.t1 = desired.t1;
+ this.s2 = desired.s2;
+ this.t2 = desired.t2;
+ this.error = desired.error;
+ }, {
+
+ // A generic two-segment piecewise regression calculator. Based on Kundu/Ubhaya
+ //
+ // Minimize sum of (y - y')^2
+ // where y = s1 + t1*x
+ // y = s2 + t2*x
+ // y' = s1 + t1*x' = s2 + t2*x' if x_0 <= x' <= x_n
+ //
+ // Allows for fixing s1, t1, s2, t2
+ //
+ // x is assumed to be complexity, y is frame length. Can be used for pure complexity-FPS
+ // analysis or for ramp controllers since complexity monotonically decreases with time.
+ _calculateRegression: function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
+ {
+ var iterationDirection = endIndex > startIndex ? 1 : -1;
+ var lowComplexity = getComplexity(samples, startIndex);
+ var highComplexity = getComplexity(samples, endIndex);
+ var a1 = 0, b1 = 0, c1 = 0, d1 = 0, h1 = 0, k1 = 0;
+ var a2 = 0, b2 = 0, c2 = 0, d2 = 0, h2 = 0, k2 = 0;
+
+ // Iterate from low to high complexity
+ for (var i = startIndex; iterationDirection * (endIndex - i) > -1; i += iterationDirection) {
+ var x = getComplexity(samples, i);
+ var y = getFrameLength(samples, i);
+ a2 += 1;
+ b2 += x;
+ c2 += x * x;
+ d2 += y;
+ h2 += y * x;
+ k2 += y * y;
+ }
+
+ var s1_best, t1_best, s2_best, t2_best, x_best, error_best, x_prime;
+
+ function setBest(s1, t1, s2, t2, error, x_prime, x)
+ {
+ s1_best = s1;
+ t1_best = t1;
+ s2_best = s2;
+ t2_best = t2;
+ error_best = error;
+ if (!options.shouldClip || (x_prime >= lowComplexity && x_prime <= highComplexity))
+ x_best = x_prime;
+ else {
+ // Discontinuous piecewise regression
+ x_best = x;
+ }
+ }
+
+ // Iterate from startIndex to endIndex - 1, inclusive
+ for (var i = startIndex; iterationDirection * (endIndex - i) > 0; i += iterationDirection) {
+ var x = getComplexity(samples, i);
+ var y = getFrameLength(samples, i);
+ var xx = x * x;
+ var yx = y * x;
+ var yy = y * y;
+ // a1, b1, etc. is sum from startIndex to i, inclusive
+ a1 += 1;
+ b1 += x;
+ c1 += xx;
+ d1 += y;
+ h1 += yx;
+ k1 += yy;
+ // a2, b2, etc. is sum from i+1 to endIndex, inclusive
+ a2 -= 1;
+ b2 -= x;
+ c2 -= xx;
+ d2 -= y;
+ h2 -= yx;
+ k2 -= yy;
+
+ var A = c1*d1 - b1*h1;
+ var B = a1*h1 - b1*d1;
+ var C = a1*c1 - b1*b1;
+ var D = c2*d2 - b2*h2;
+ var E = a2*h2 - b2*d2;
+ var F = a2*c2 - b2*b2;
+ var s1 = options.s1 !== undefined ? options.s1 : (A / C);
+ var t1 = options.t1 !== undefined ? options.t1 : (B / C);
+ var s2 = options.s2 !== undefined ? options.s2 : (D / F);
+ var t2 = options.t2 !== undefined ? options.t2 : (E / F);
+ // Assumes that the two segments meet
+ var x_prime = (s1 - s2) / (t2 - t1);
+
+ var error1 = (k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) || 0;
+ var error2 = (k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) || 0;
+
+ if (i == startIndex) {
+ setBest(s1, t1, s2, t2, error1 + error2, x_prime, x);
+ continue;
+ }
+
+ // Projected point is not between this and the next sample
+ if (x_prime > getComplexity(samples, i + iterationDirection) || x_prime < x) {
+ // Calculate lambda, which divides the weight of this sample between the two lines
+
+ // These values remove the influence of this sample
+ var I = c1 - 2*b1*x + a1*xx;
+ var H = C - I;
+ var G = A + B*x - C*y;
+
+ var J = D + E*x - F*y;
+ var K = c2 - 2*b2*x + a2*xx;
+
+ var lambda = (G*F + G*K - H*J) / (I*J + G*K);
+ if (lambda > 0 && lambda < 1) {
+ var lambda1 = 1 - lambda;
+ s1 = options.s1 !== undefined ? options.s1 : ((A - lambda1*(-h1*x + d1*xx + c1*y - b1*yx)) / (C - lambda1*I));
+ t1 = options.t1 !== undefined ? options.t1 : ((B - lambda1*(h1 - d1*x - b1*y + a1*yx)) / (C - lambda1*I));
+ s2 = options.s2 !== undefined ? options.s2 : ((D + lambda1*(-h2*x + d2*xx + c2*y - b2*yx)) / (F + lambda1*K));
+ t2 = options.t2 !== undefined ? options.t2 : ((E + lambda1*(h2 - d2*x - b2*y + a2*yx)) / (F + lambda1*K));
+ x_prime = (s1 - s2) / (t2 - t1);
+
+ error1 = ((k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) - lambda1 * Math.pow(y - (s1 + t1*x), 2)) || 0;
+ error2 = ((k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) + lambda1 * Math.pow(y - (s2 + t2*x), 2)) || 0;
+ }
+ }
+
+ if (error1 + error2 < error_best)
+ setBest(s1, t1, s2, t2, error1 + error2, x_prime, x);
+ }
+
+ return {
+ complexity: x_best,
+ s1: s1_best,
+ t1: t1_best,
+ s2: s2_best,
+ t2: t2_best,
+ error: error_best
+ };
+ }
+});
+
+RampController = Utilities.createSubclass(Controller,
+ function(benchmark, options)
+ {
+ // The tier warmup takes at most 5 seconds
+ options["sample-capacity"] = (options["test-interval"] / 1000 + 5) * 60;
+ Controller.call(this, benchmark, options);
+
+ // Initially start with a tier test to find the bounds
+ // The number of objects in a tier test is 10^|_tier|
+ this._tier = 0;
+ // The timestamp is first set after the first interval completes
+ this._tierStartTimestamp = 0;
+ // If the engine can handle the tier's complexity at 60 FPS, test for a short
+ // period, then move on to the next tier
+ this._tierFastTestLength = 250;
+ // If the engine is under stress, let the test run a little longer to let
+ // the measurement settle
+ this._tierSlowTestLength = 750;
+ this._maximumComplexity = 0;
+ this._minimumTier = 0;
+
+ // After the tier range is determined, figure out the number of ramp iterations
+ var minimumRampLength = 3000;
+ var totalRampIterations = Math.max(1, Math.floor(this._endTimestamp / minimumRampLength));
+ // Give a little extra room to run since the ramps won't be exactly this length
+ this._rampLength = Math.floor((this._endTimestamp - totalRampIterations * this._intervalLength) / totalRampIterations);
+ this._rampWarmupLength = 200;
+ this._rampDidWarmup = false;
+ this._rampRegressions = [];
+
+ // Add some tolerance; frame lengths shorter than this are considered to be @ 60 fps
+ this._fps60Threshold = 1000/58;
+ // We are looking for the complexity that will get us at least as slow this threshold
+ this._fpsLowestThreshold = 1000/30;
+
+ this._finishedTierSampling = false;
+ this._startedRamps = false;
+ this._complexityPrime = new Experiment;
+ }, {
+
+ start: function(startTimestamp, stage)
+ {
+ Controller.prototype.start.call(this, startTimestamp, stage);
+ this._rampStartTimestamp = 0;
+ },
+
+ didFinishInterval: function(timestamp, stage, intervalAverageFrameLength)
+ {
+ if (!this._finishedTierSampling) {
+ if (this._tierStartTimestamp > 0 && timestamp < this._tierStartTimestamp + this._tierFastTestLength)
+ return;
+
+ var currentComplexity = stage.complexity();
+ var currentFrameLength = this._frameLengthEstimator.estimate;
+ if (currentFrameLength < this._fpsLowestThreshold) {
+ var isAnimatingAt60FPS = currentFrameLength < this._fps60Threshold;
+ var hasFinishedSlowTierTest = timestamp > this._tierStartTimestamp + this._tierSlowTestLength;
+
+ // We're measuring at 60 fps, so quickly move on to the next tier, or
+ // we've slower than 60 fps, but we've let this tier run long enough to
+ // get an estimate
+ if (currentFrameLength < this._fps60Threshold || timestamp > this._tierStartTimestamp + this._tierSlowTestLength) {
+ this._lastComplexity = currentComplexity;
+ this._lastFrameLength = currentFrameLength;
+
+ this._tierStartTimestamp = timestamp;
+ this._tier += .5;
+ var nextTierComplexity = Math.round(Math.pow(10, this._tier));
+ this.mark("Complexity: " + nextTierComplexity, timestamp);
+
+ stage.tune(nextTierComplexity - currentComplexity);
+ }
+ return;
+ } else if (timestamp < this._tierStartTimestamp + this._tierSlowTestLength)
+ return;
+
+ this._finishedTierSampling = true;
+ // Extend the test length so that the full test length is made of the ramps
+ this._endTimestamp += timestamp;
+ this.mark(Strings.json.samplingStartTimeOffset, timestamp);
+
+ // Sometimes this last tier will drop the frame length well below the threshold
+ // Avoid going down that far since it means fewer measurements are taken in the 60 fps area
+ // Interpolate a maximum complexity that gets us around the lowest threshold
+ this._maximumComplexity = Math.floor(this._lastComplexity + (this._fpsLowestThreshold - this._lastFrameLength) / (currentFrameLength - this._lastFrameLength) * (currentComplexity - this._lastComplexity));
+ stage.tune(this._maximumComplexity - currentComplexity);
+ this._rampStartTimestamp = timestamp;
+ this._rampDidWarmup = false;
+ this.isFrameLengthEstimatorEnabled = false;
+ this._intervalCount = 0;
+ return;
+ }
+
+ if ((timestamp - this._rampStartTimestamp) < this._rampWarmupLength)
+ return;
+
+ if (this._rampDidWarmup)
+ return;
+
+ this._rampDidWarmup = true;
+ this._currentRampLength = this._rampStartTimestamp + this._rampLength - timestamp;
+ // Start timestamp represents start of ramp down, after warm up
+ this._rampStartTimestamp = timestamp;
+ this._rampStartIndex = this._sampler.sampleCount;
+ },
+
+ tune: function(timestamp, stage, didFinishInterval)
+ {
+ if (!didFinishInterval || !this._rampDidWarmup)
+ return;
+
+ var progress = (timestamp - this._rampStartTimestamp) / this._currentRampLength;
+
+ if (progress < 1) {
+ stage.tune(Math.round((1 - progress) * this._maximumComplexity) - stage.complexity());
+ return;
+ }
+
+ var regression = new Regression(this._sampler.samples, this._getComplexity, this._getFrameLength,
+ this._sampler.sampleCount - 1, this._rampStartIndex);
+ this._rampRegressions.push(regression);
+
+ this._complexityPrime.sample(regression.complexity);
+ this._maximumComplexity = Math.max(5, Math.round(this._complexityPrime.mean() * 2));
+
+ // Next ramp
+ this._rampDidWarmup = false;
+ // Start timestamp represents start of ramp iteration and warm up
+ this._rampStartTimestamp = timestamp;
+ stage.tune(this._maximumComplexity - stage.complexity());
+ },
+
+ _getComplexity: function(samples, i) {
+ return samples[1][i];
+ },
+
+ _getFrameLength: function(samples, i) {
+ return samples[0][i] - samples[0][i - 1];
+ },
+
+ processSamples: function(results)
+ {
+ Controller.prototype.processSamples.call(this, results);
+
+ // Have samplingTimeOffset represent time 0
+ var startTimestamp = this._marks[Strings.json.samplingStartTimeOffset].time;
+ results[Strings.json.samples].forEach(function(sample) {
+ sample.time -= startTimestamp;
+ });
+ for (var markName in results[Strings.json.marks]) {
+ results[Strings.json.marks][markName].time -= startTimestamp;
+ }
+
+ var samples = results[Strings.json.samples];
+ results[Strings.json.regressions.timeRegressions] = [];
+ var complexityRegressionSamples = [];
+ var timeComplexityScore = new Experiment;
+ this._rampRegressions.forEach(function(ramp) {
+ var startIndex = ramp.startIndex, endIndex = ramp.endIndex;
+ var startTime = samples[startIndex].time, endTime = samples[endIndex].time;
+ var startComplexity = samples[startIndex].complexity, endComplexity = samples[endIndex].complexity;
+
+ timeComplexityScore.sample(ramp.complexity);
+
+ var regression = {};
+ results[Strings.json.regressions.timeRegressions].push(regression);
+
+ var percentage = (ramp.complexity - startComplexity) / (endComplexity - startComplexity);
+ var inflectionTime = startTime + percentage * (endTime - startTime);
+
+ regression[Strings.json.regressions.segment1] = [
+ [startTime, ramp.s2 + ramp.t2 * startComplexity],
+ [inflectionTime, ramp.s2 + ramp.t2 * ramp.complexity]
+ ];
+ regression[Strings.json.regressions.segment2] = [
+ [inflectionTime, ramp.s1 + ramp.t1 * ramp.complexity],
+ [endTime, ramp.s1 + ramp.t1 * endComplexity]
+ ];
+ regression[Strings.json.regressions.complexity] = ramp.complexity;
+ regression[Strings.json.regressions.maxComplexity] = Math.max(startComplexity, endComplexity);
+ regression[Strings.json.regressions.startIndex] = startIndex;
+ regression[Strings.json.regressions.endIndex] = endIndex;
+
+ for (var j = startIndex; j <= endIndex; ++j)
+ complexityRegressionSamples.push(samples[j]);
+ });
+
+ // Aggregate all of the ramps into one big dataset and calculate a regression from this
+ complexityRegressionSamples.sort(function(a, b) {
+ return a.complexity - b.complexity;
+ });
+
+ // Samples averaged based on complexity
+ results[Strings.json.complexityAverageSamples] = [];
+ var currentComplexity = -1;
+ var experimentAtComplexity;
+ function addSample() {
+ results[Strings.json.complexityAverageSamples].push({
+ complexity: currentComplexity,
+ frameLength: experimentAtComplexity.mean(),
+ stdev: experimentAtComplexity.standardDeviation(),
+ });
+ }
+ complexityRegressionSamples.forEach(function(sample) {
+ if (sample.complexity != currentComplexity) {
+ if (currentComplexity > -1)
+ addSample();
+
+ currentComplexity = sample.complexity;
+ experimentAtComplexity = new Experiment;
+ }
+ experimentAtComplexity.sample(sample.frameLength);
+ });
+ // Finish off the last one
+ addSample();
+
+ function calculateRegression(samples, key) {
+ var complexityRegression = new Regression(
+ samples,
+ function (samples, i) { return samples[i].complexity; },
+ function (samples, i) { return samples[i].frameLength; },
+ 0, samples.length - 1
+ );
+ var minComplexity = samples[0].complexity;
+ var maxComplexity = samples[samples.length - 1].complexity;
+ var regression = {};
+ results[key] = regression;
+ regression[Strings.json.regressions.segment1] = [
+ [minComplexity, complexityRegression.s1 + complexityRegression.t1 * minComplexity],
+ [complexityRegression.complexity, complexityRegression.s1 + complexityRegression.t1 * complexityRegression.complexity]
+ ];
+ regression[Strings.json.regressions.segment2] = [
+ [complexityRegression.complexity, complexityRegression.s2 + complexityRegression.t2 * complexityRegression.complexity],
+ [maxComplexity, complexityRegression.s2 + complexityRegression.t2 * maxComplexity]
+ ];
+ regression[Strings.json.regressions.complexity] = complexityRegression.complexity;
+ regression[Strings.json.measurements.stdev] = Math.sqrt(complexityRegression.error / samples.length);
+ }
+
+ calculateRegression(complexityRegressionSamples, Strings.json.regressions.complexityRegression);
+ calculateRegression(results[Strings.json.complexityAverageSamples], Strings.json.regressions.complexityAverageRegression);
+
+ // Frame rate experiment result is unneeded
+ delete results[Strings.json.experiments.frameRate];
+
+ results[Strings.json.score] = timeComplexityScore.mean();
+ results[Strings.json.experiments.complexity] = {};
+ results[Strings.json.experiments.complexity][Strings.json.measurements.average] = timeComplexityScore.mean();
+ results[Strings.json.experiments.complexity][Strings.json.measurements.stdev] = timeComplexityScore.standardDeviation();
+ results[Strings.json.experiments.complexity][Strings.json.measurements.percent] = timeComplexityScore.percentage();
+ }
+});
+
</ins><span class="cx"> Stage = Utilities.createClass(
</span><span class="cx"> function()
</span><span class="cx"> {
</span><span class="lines">@@ -435,9 +851,11 @@
</span><span class="cx"> this._controller = new StepController(this, options);
</span><span class="cx"> break;
</span><span class="cx"> case "adaptive":
</span><del>- default:
</del><span class="cx"> this._controller = new AdaptiveController(this, options);
</span><span class="cx"> break;
</span><ins>+ case "ramp":
+ this._controller = new RampController(this, options);
+ break;
</ins><span class="cx"> }
</span><span class="cx"> }, {
</span><span class="cx">
</span></span></pre></div>
<a id="trunkPerformanceTestsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/ChangeLog (196371 => 196372)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/ChangeLog 2016-02-10 19:22:07 UTC (rev 196371)
+++ trunk/PerformanceTests/ChangeLog 2016-02-10 19:22:08 UTC (rev 196372)
</span><span class="lines">@@ -1,5 +1,45 @@
</span><span class="cx"> 2016-02-08 Jon Lee <jonlee@apple.com>
</span><span class="cx">
</span><ins>+ Add a ramp controller
+ https://bugs.webkit.org/show_bug.cgi?id=154028
+
+ Provisionally reviewed by Said Abou-Hallawa.
+
+ Enhance the graph to include a complexity-fps graph, in addition
+ to the time graph.
+
+ * Animometer/developer.html: Add a ramp option.
+ * Animometer/resources/debug-runner/animometer.css: Update the style.
+ * Animometer/resources/strings.js: Flatten the Strings.text constants.
+ * Animometer/resources/debug-runner/animometer.js:
+ (ResultsTable.call._addGraphButton): Refactor.
+ (ResultsTable.call._addTest): Add regression data.
+ (benchmarkController): Add a form that allows the user to switch between the two forms,
+ Add a form that allows the user to toggle different data. Hide certain header columns
+ depending on the selected controller.
+ * Animometer/resources/debug-runner/graph.js: Add the complexity regressions.
+ * Animometer/resources/debug-runner/tests.js: Add headers for the ramp results.
+ * Animometer/resources/runner/animometer.js:
+ (ResultsTable): If a header is disabled don't include them in _flattenedHeaders.
+ * Animometer/tests/resources/main.js:
+ (Controller): Allow options to specify the capacity for sample arrays.
+ (Regression): A piecewise regression that tries to fit a slope and a flat profile.
+ (_calculateRegression): Options can fix the slope and bias when calculating the minimal
+ error. Sweep across the samples in time (which could be backward depending on the controller)
+ and calculate the intersection point.
+ (RampController): This controller assumes that the target frame rate is below
+ 58 FPS. It runs in two stages. The first stage quickly determines the order of
+ magnitude of objects needed to stress the system by the setting the complexity
+ to increasingly difficult tiers. Perform a series of ramps descending from a
+ high-water mark of complexity. The complexity needed to reach the target frame
+ length is done by performing a piecewise regression on each ramp, and track a
+ running average of these values. For the next ramp, make that running average
+ the center of the ramp. With a minimum complexity of 0, the high-water mark is
+ twice that average. The score is based on the highest complexity that can
+ reach 60 fps.
+
+2016-02-08 Jon Lee <jonlee@apple.com>
+
</ins><span class="cx"> Address Said's comments on the benchmark, and do some clean up.
</span><span class="cx">
</span><span class="cx"> * Animometer/developer.html:
</span></span></pre>
</div>
</div>
</body>
</html>