<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[196288] trunk/PerformanceTests</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/196288">196288</a></dd>
<dt>Author</dt> <dd>jonlee@apple.com</dd>
<dt>Date</dt> <dd>2016-02-08 19:25:39 -0800 (Mon, 08 Feb 2016)</dd>
</dl>
<h3>Log Message</h3>
<pre>Update how the benchmark is run
https://bugs.webkit.org/show_bug.cgi?id=153960
Provisionally reviewed by Said Abou-Hallawa.
Introduce the notion of a Controller. It is responsible for recording, updating,
and processing the statistics and complexity of the benchmark. This allows
plugging in different Controllers.
This strips most of the functionality from Animator and BenchmarkState, so fold
what's left into Benchmark. Now, Benchmarks only own a stage and a controller, but
are responsible for driving the animation loop.
Rewrite Animator._shouldRequestAnotherFrame into two different Controllers. One
maintains a fixed complexity, and the other adapts the complexity to meet a
fixed FPS.
Fix the Kalman estimator to be modeled on a scalar variable with no model.
* Animometer/tests/resources/main.js: Remove BenchmarkState and Animator, and
replace it with a Controller. Add a FixedController and refactor the previous controller
to an AdaptiveController.
(Controller): Controllers own the estimator and the sampler. When a new frame is
displayed, the animation loop calls update(). The estimator and sampler record
stats, then tune. Samplers can track multiple series of data. The basic controller
tracks timestamp, complexity, and estimated frame rate.
The Kalman estimation is based on the frame length rather than the frame
rate. Because FPS is inverse proportional to frame length, in the case where the measured
frame length is very small, the FPS ends up being a wildly large number (in the order of
600-1000 "FPS"), and it pulls the estimator up drastically enough that it takes a while
for it to settle back down. Using frame length reduces the impact of these spikes.
Converging the estimation takes enough time to avoid initializing it immediately
when the benchmark starts. Instead, the benchmark runs for a brief period of time (100ms)
before running it in earnest. Allow controllers an opportunity to set the complexity
before starting recording.
When the benchmark is complete, the controller has an opportunity to process
the samples. The default implementation calculates the raw FPS based on the time
difference of the samples, and calculates the complexity score. This is moved from
Benchmark.processSamples.
(Controller): Initialize timestamps. These are at first relative to the start of the
benchmark, but are offset by the absolute start time during start(). By default maintain
3 data series, but subclasses can override.
(start): Calls recordFirstSample() for subclasses to override if needed.
(recordFirstSample): For basic controller, start sampling at the beginning.
(update): Update the frame length estimator and sample.
(shouldStop): Checks that the time is before _endTimestamp.
(results): Returns the processed samples.
(processSamples): Iterate through the sample data and collate them. Include scores.
(FixedComplexityController): Controller that tunes the stage to the desired complexity
prior to starting, and keeps it at that complexity.
(AdaptiveController): Have the estimator estimate the interval frame rate instead of the
raw frame rate.
The previous version of this controller ignored the frame that came after the
adjustment. The raw FPS show that whatever noise the scene change adds is negligible
compared to the noise of the system overall. Stop ignoring that frame and include all
frames in the measurements.
(Benchmark): Remove dependency on animator, and instantiate a runner based on what is
selected. Most of the loop's functionality is in Controller, so remove here.
(Benchmark.run): Remove start() since it is only called from run(), and fold it in here.
(Benchmark._animateLoop): Fold in from Animator.animateLoop. Let the benchmark run for
a brief period before calling Controller.start().
* Animometer/tests/resources/math.js: Fix the Kalman estimator. The filter estimates
a scalar variable, and makes basic assumptions regarding the model. As a result
none of the linear algebra classes are needed, so remove Matrix, Vector3, and Matrix3.
(SimpleKalmanEstimator): Calculate the gain based on the provided process and
measurement errors.
(KalmanEstimator): Deleted.
(IdentityEstimator): Deleted.
(PIDController): Refactor to use the Utilities.createClass() helper.
The Kalman filter algorithm is explained here http://greg.czerniak.info/guides/kalman1/.
The state, represented by a scalar, is the estimated frame length. There is no user
transition of the state, and the state is the same as the measurement. With this model,
the estimation error converges, so calculate the gain ahead of time.
* Animometer/developer.html: Remove fixed-after-warmup since it is not useful.
Replace the option to toggle the estimator, and make it possible to customize the
estimator's error parameters. Show raw FPS by default, and remove interval FPS,
which will be shown instead of the filtered raw FPS.
* Animometer/resources/debug-runner/animometer.css: Put the header behind the graph.
Remove #intervalFPS rules; move the color to #filteredFPS.
* Animometer/resources/debug-runner/graph.js:
(updateGraphData): Update the hr style to force the layout to be calculated
correctly. Change the tick format to be in terms of seconds, since the timestamps
are in milliseconds. Remove interval data.
* Animometer/resources/runner/animometer.js:
(window.benchmarkController.startBenchmark): Set Kalman parameters.
* Animometer/resources/runner/benchmark-runner.js:
(_runBenchmarkAndRecordResults): When a benchmark completes, expect it to return
the final data, rather than passing a sampler from the controller. This avoids
needing to expose the sampler variable in the benchmark.
* Animometer/tests/resources/sampler.js:
(process): Move the setting of the target frame rate to AdaptiveController.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkPerformanceTestsAnimometerdeveloperhtml">trunk/PerformanceTests/Animometer/developer.html</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesdebugrunneranimometercss">trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.css</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesdebugrunnergraphjs">trunk/PerformanceTests/Animometer/resources/debug-runner/graph.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesextensionsjs">trunk/PerformanceTests/Animometer/resources/extensions.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesrunneranimometerjs">trunk/PerformanceTests/Animometer/resources/runner/animometer.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometerresourcesrunnerbenchmarkrunnerjs">trunk/PerformanceTests/Animometer/resources/runner/benchmark-runner.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometertestsresourcesmainjs">trunk/PerformanceTests/Animometer/tests/resources/main.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometertestsresourcesmathjs">trunk/PerformanceTests/Animometer/tests/resources/math.js</a></li>
<li><a href="#trunkPerformanceTestsAnimometertestsresourcessamplerjs">trunk/PerformanceTests/Animometer/tests/resources/sampler.js</a></li>
<li><a href="#trunkPerformanceTestsChangeLog">trunk/PerformanceTests/ChangeLog</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkPerformanceTestsAnimometerdeveloperhtml"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/developer.html (196287 => 196288)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/developer.html 2016-02-09 03:04:20 UTC (rev 196287)
+++ trunk/PerformanceTests/Animometer/developer.html 2016-02-09 03:25:39 UTC (rev 196288)
</span><span class="lines">@@ -44,8 +44,7 @@
</span><span class="cx"> <li>
</span><span class="cx"> <h3>Adjusting the test complexity:</h3>
</span><span class="cx"> <ul>
</span><del>- <li><label><input name="adjustment" type="radio" value="fixed"> Keep constant</label></li>
- <li><label><input name="adjustment" type="radio" value="fixed-after-warmup"> Keep constant after warmup to target FPS</label></li>
</del><ins>+ <li><label><input name="adjustment" type="radio" value="fixed"> Keep at a fixed complexity</label></li>
</ins><span class="cx"> <li><label><input name="adjustment" type="radio" value="adaptive" checked> Maintain target FPS</label></li>
</span><span class="cx"> </ul>
</span><span class="cx"> </li>
</span><span class="lines">@@ -53,7 +52,11 @@
</span><span class="cx"> <label>Target frame rate: <input type="number" id="frame-rate" value="50"> FPS</label>
</span><span class="cx"> </li>
</span><span class="cx"> <li>
</span><del>- <label><input type="checkbox" id="estimated-frame-rate" checked> Filter frame rate calculation</label>
</del><ins>+ <h3>Kalman filter estimated error:</h3>
+ <ul>
+ <li><label>Process error (Q): <input type="number" id="kalman-process-error" value="1"></label></li>
+ <li><label>Measurement error (R): <input type="number" id="kalman-measurement-error" value="4"></label></li>
+ </ul>
</ins><span class="cx"> </li>
</span><span class="cx"> </ul>
</span><span class="cx"> </form>
</span><span class="lines">@@ -100,12 +103,10 @@
</span><span class="cx"> <li><label><input type="checkbox" name="averages" checked> Averages</label></li>
</span><span class="cx"> <li><label><input type="checkbox" name="complexity" checked> Complexity</label>
</span><span class="cx"> <span class="complexity"></span></li>
</span><del>- <li><label><input type="checkbox" name="rawFPS"> Raw FPS</label>
</del><ins>+ <li><label><input type="checkbox" name="rawFPS" checked> Raw FPS</label>
</ins><span class="cx"> <span class="rawFPS"></span></li>
</span><span class="cx"> <li><label><input type="checkbox" name="filteredFPS" checked> Filtered FPS</label>
</span><span class="cx"> <span class="filteredFPS"></span></li>
</span><del>- <li><label><input type="checkbox" name="intervalFPS"> Average FPS per sample interval</label>
- <span class="intervalFPS"></span></li>
</del><span class="cx"> </ul>
</span><span class="cx"> </form>
</span><span class="cx"> </nav>
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesdebugrunneranimometercss"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.css (196287 => 196288)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.css 2016-02-09 03:04:20 UTC (rev 196287)
+++ trunk/PerformanceTests/Animometer/resources/debug-runner/animometer.css 2016-02-09 03:25:39 UTC (rev 196288)
</span><span class="lines">@@ -349,6 +349,7 @@
</span><span class="cx"> #test-graph-data {
</span><span class="cx"> flex: 1 1 auto;
</span><span class="cx"> align-self: stretch;
</span><ins>+ z-index: 1;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #test-graph nav {
</span><span class="lines">@@ -441,12 +442,12 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #filteredFPS path {
</span><del>- stroke: rgba(250, 73, 37, .7);
- stroke-width: 2px;
</del><ins>+ stroke: hsla(30, 96%, 56%, .7);
+ stroke-width: 1px;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #filteredFPS circle {
</span><del>- fill: rgb(250, 73, 37);
</del><ins>+ fill: hsl(30, 96%, 56%);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #rawFPS path {
</span><span class="lines">@@ -457,12 +458,3 @@
</span><span class="cx"> #rawFPS circle {
</span><span class="cx"> fill: rgb(250, 73, 37);
</span><span class="cx"> }
</span><del>-
-#intervalFPS path {
- stroke: hsla(30, 96%, 56%, .7);
- stroke-width: 1px;
-}
-
-#intervalFPS circle {
- fill: hsl(30, 96%, 56%);
-}
</del></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesdebugrunnergraphjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/debug-runner/graph.js (196287 => 196288)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/debug-runner/graph.js 2016-02-09 03:04:20 UTC (rev 196287)
+++ trunk/PerformanceTests/Animometer/resources/debug-runner/graph.js 2016-02-09 03:25:39 UTC (rev 196288)
</span><span class="lines">@@ -1,9 +1,13 @@
</span><span class="cx"> Utilities.extendObject(window.benchmarkController, {
</span><ins>+ layoutCounter: 0,
+
</ins><span class="cx"> updateGraphData: function(graphData)
</span><span class="cx"> {
</span><span class="cx"> var element = document.getElementById("test-graph-data");
</span><span class="cx"> element.innerHTML = "";
</span><del>- var margins = new Insets(10, 30, 30, 40);
</del><ins>+ document.querySelector("hr").style.width = this.layoutCounter++ + "px";
+
+ var margins = new Insets(30, 30, 30, 40);
</ins><span class="cx"> var size = Point.elementClientSize(element).subtract(margins.size);
</span><span class="cx">
</span><span class="cx"> var svg = d3.select("#test-graph-data").append("svg")
</span><span class="lines">@@ -29,7 +33,8 @@
</span><span class="cx"> // Axes
</span><span class="cx"> var xAxis = d3.svg.axis()
</span><span class="cx"> .scale(x)
</span><del>- .orient("bottom");
</del><ins>+ .orient("bottom")
+ .tickFormat(function(d) { return (d/1000).toFixed(0); });
</ins><span class="cx"> var yAxisLeft = d3.svg.axis()
</span><span class="cx"> .scale(yLeft)
</span><span class="cx"> .orient("left");
</span><span class="lines">@@ -127,9 +132,6 @@
</span><span class="cx"> var filteredData = graphData.samples.filter(function (sample) {
</span><span class="cx"> return "smoothedFPS" in sample;
</span><span class="cx"> });
</span><del>- var intervalData = graphData.samples.filter(function (sample) {
- return "intervalFPS" in sample;
- });
</del><span class="cx">
</span><span class="cx"> function addData(name, data, yCoordinateCallback, pointRadius, omitLine) {
</span><span class="cx"> var svgGroup = svg.append("g").attr("id", name);
</span><span class="lines">@@ -156,7 +158,6 @@
</span><span class="cx"> addData("complexity", allData, function(d) { return yLeft(d.complexity); }, 2);
</span><span class="cx"> addData("rawFPS", allData, function(d) { return yRight(d.fps); }, 1);
</span><span class="cx"> addData("filteredFPS", filteredData, function(d) { return yRight(d.smoothedFPS); }, 2);
</span><del>- addData("intervalFPS", intervalData, function(d) { return yRight(d.intervalFPS); }, 2);
</del><span class="cx">
</span><span class="cx"> // Area to handle mouse events
</span><span class="cx"> var area = svg.append("rect")
</span><span class="lines">@@ -167,7 +168,7 @@
</span><span class="cx"> .attr("height", size.y);
</span><span class="cx">
</span><span class="cx"> var timeBisect = d3.bisector(function(d) { return d.time; }).right;
</span><del>- var statsToHighlight = ["complexity", "rawFPS", "filteredFPS", "intervalFPS"];
</del><ins>+ var statsToHighlight = ["complexity", "rawFPS", "filteredFPS"];
</ins><span class="cx"> area.on("mouseover", function() {
</span><span class="cx"> document.getElementById("cursor").classList.remove("hidden");
</span><span class="cx"> document.querySelector("#test-graph nav").classList.remove("hide-data");
</span><span class="lines">@@ -189,7 +190,7 @@
</span><span class="cx"> .attr("x2", cursor_x)
</span><span class="cx"> .attr("y2", yRight(cursor_y));
</span><span class="cx">
</span><del>- document.querySelector("#test-graph nav .time").textContent = data.time.toFixed(4) + "s (" + index + ")";
</del><ins>+ document.querySelector("#test-graph nav .time").textContent = (data.time / 1000).toFixed(4) + "s (" + index + ")";
</ins><span class="cx"> statsToHighlight.forEach(function(name) {
</span><span class="cx"> var element = document.querySelector("#test-graph nav ." + name);
</span><span class="cx"> var content = "";
</span><span class="lines">@@ -209,12 +210,6 @@
</span><span class="cx"> data_y = yRight(data.smoothedFPS);
</span><span class="cx"> }
</span><span class="cx"> break;
</span><del>- case "intervalFPS":
- if ("intervalFPS" in data) {
- content = data.intervalFPS.toFixed(2);
- data_y = yRight(data.intervalFPS);
- }
- break;
</del><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> element.textContent = content;
</span><span class="lines">@@ -250,6 +245,5 @@
</span><span class="cx"> showOrHideNodes(form["complexity"].checked, "#complexity");
</span><span class="cx"> showOrHideNodes(form["rawFPS"].checked, "#rawFPS");
</span><span class="cx"> showOrHideNodes(form["filteredFPS"].checked, "#filteredFPS");
</span><del>- showOrHideNodes(form["intervalFPS"].checked, "#intervalFPS");
</del><span class="cx"> }
</span><span class="cx"> });
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesextensionsjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/extensions.js (196287 => 196288)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/extensions.js 2016-02-09 03:04:20 UTC (rev 196287)
+++ trunk/PerformanceTests/Animometer/resources/extensions.js 2016-02-09 03:25:39 UTC (rev 196288)
</span><span class="lines">@@ -173,7 +173,7 @@
</span><span class="cx"> {
</span><span class="cx"> return "x = " + this.x + ", y = " + this.y;
</span><span class="cx"> },
</span><del>-
</del><ins>+
</ins><span class="cx"> add: function(other)
</span><span class="cx"> {
</span><span class="cx"> if(isNaN(other.x))
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesrunneranimometerjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/runner/animometer.js (196287 => 196288)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/runner/animometer.js 2016-02-09 03:04:20 UTC (rev 196287)
+++ trunk/PerformanceTests/Animometer/resources/runner/animometer.js 2016-02-09 03:25:39 UTC (rev 196288)
</span><span class="lines">@@ -307,7 +307,8 @@
</span><span class="cx"> "display": "minimal",
</span><span class="cx"> "adjustment": "adaptive",
</span><span class="cx"> "frame-rate": 50,
</span><del>- "estimated-frame-rate": true
</del><ins>+ "kalman-process-error": 1,
+ "kalman-measurement-error": 4
</ins><span class="cx"> };
</span><span class="cx"> this._startBenchmark(Suites, options, "test-container");
</span><span class="cx"> },
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometerresourcesrunnerbenchmarkrunnerjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/resources/runner/benchmark-runner.js (196287 => 196288)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/resources/runner/benchmark-runner.js 2016-02-09 03:04:20 UTC (rev 196287)
+++ trunk/PerformanceTests/Animometer/resources/runner/benchmark-runner.js 2016-02-09 03:25:39 UTC (rev 196288)
</span><span class="lines">@@ -94,9 +94,9 @@
</span><span class="cx"> Utilities.extendObject(options, contentWindow.Utilities.parseParameters());
</span><span class="cx">
</span><span class="cx"> var benchmark = new contentWindow.benchmarkClass(options);
</span><del>- benchmark.run().then(function(sampler) {
</del><ins>+ benchmark.run().then(function(results) {
</ins><span class="cx"> var samplers = self._suitesSamplers[suite.name] || {};
</span><del>- samplers[test.name] = sampler.process(options);
</del><ins>+ samplers[test.name] = results;
</ins><span class="cx"> self._suitesSamplers[suite.name] = samplers;
</span><span class="cx">
</span><span class="cx"> if (self._client && self._client.didRunTest)
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometertestsresourcesmainjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/tests/resources/main.js (196287 => 196288)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/tests/resources/main.js 2016-02-09 03:04:20 UTC (rev 196287)
+++ trunk/PerformanceTests/Animometer/tests/resources/main.js 2016-02-09 03:25:39 UTC (rev 196288)
</span><span class="lines">@@ -1,53 +1,181 @@
</span><del>-function BenchmarkState(testInterval)
-{
- this._currentTimeOffset = 0;
- this._stageInterval = testInterval / BenchmarkState.stages.FINISHED;
-}
</del><span class="cx">
</span><del>-// The enum values and the messages should be in the same order
-BenchmarkState.stages = {
- WARMING: 0,
- SAMPLING: 1,
- FINISHED: 2,
-}
</del><ins>+Controller = Utilities.createClass(
+ function(testLength, benchmark, seriesCount)
+ {
+ // Initialize timestamps relative to the start of the benchmark
+ // In start() the timestamps are offset by the start timestamp
+ this._startTimestamp = 0;
+ this._endTimestamp = testLength;
+ // Default data series: timestamp, complexity, estimatedFrameLength
+ this._sampler = new Sampler(seriesCount || 3, 60 * testLength, this);
+ this._estimator = new SimpleKalmanEstimator(benchmark.options["kalman-process-error"], benchmark.options["kalman-measurement-error"]);
</ins><span class="cx">
</span><del>-BenchmarkState.prototype =
-{
- _timeOffset: function(stage)
</del><ins>+ this.initialComplexity = 0;
+ }, {
+
+ start: function(stage, startTimestamp)
</ins><span class="cx"> {
</span><del>- return stage * this._stageInterval;
</del><ins>+ this._startTimestamp = startTimestamp;
+ this._endTimestamp += startTimestamp;
+ this.recordFirstSample(stage, startTimestamp);
</ins><span class="cx"> },
</span><span class="cx">
</span><del>- _message: function(stage, timeOffset)
</del><ins>+ recordFirstSample: function(stage, startTimestamp)
</ins><span class="cx"> {
</span><del>- if (stage == BenchmarkState.stages.FINISHED)
- return BenchmarkState.stages.messages[stage];
</del><ins>+ this._sampler.record(startTimestamp, stage.complexity(), -1);
+ this._sampler.mark(Strings.json.samplingTimeOffset, { time: 0 });
+ },
</ins><span class="cx">
</span><del>- return BenchmarkState.stages.messages[stage] + "... ("
- + Math.floor((timeOffset - this._timeOffset(stage)) / 1000) + "/"
- + Math.floor((this._timeOffset(stage + 1) - this._timeOffset(stage)) / 1000) + ")";
</del><ins>+ update: function(stage, timestamp)
+ {
+ this._estimator.sample(timestamp - this._sampler.samples[0][this._sampler.sampleCount - 1]);
+ this._sampler.record(timestamp, stage.complexity(), 1000 / this._estimator.estimate);
</ins><span class="cx"> },
</span><span class="cx">
</span><del>- update: function(currentTimeOffset)
</del><ins>+ shouldStop: function(timestamp)
</ins><span class="cx"> {
</span><del>- this._currentTimeOffset = currentTimeOffset;
</del><ins>+ return timestamp > this._endTimestamp;
</ins><span class="cx"> },
</span><span class="cx">
</span><del>- samplingTimeOffset: function()
</del><ins>+ results: function()
</ins><span class="cx"> {
</span><del>- return this._timeOffset(BenchmarkState.stages.SAMPLING);
</del><ins>+ return this._sampler.process();
</ins><span class="cx"> },
</span><span class="cx">
</span><del>- currentStage: function()
</del><ins>+ processSamples: function(results)
</ins><span class="cx"> {
</span><del>- for (var stage = BenchmarkState.stages.WARMING; stage < BenchmarkState.stages.FINISHED; ++stage) {
- if (this._currentTimeOffset < this._timeOffset(stage + 1))
- return stage;
</del><ins>+ var complexityExperiment = new Experiment;
+ var smoothedFPSExperiment = new Experiment;
+
+ var samples = this._sampler.samples;
+
+ var samplingIndex = 0;
+ var samplingMark = this._sampler.marks[Strings.json.samplingTimeOffset];
+ if (samplingMark) {
+ samplingIndex = samplingMark.index;
+ results[Strings.json.samplingTimeOffset] = samplingMark.time;
</ins><span class="cx"> }
</span><del>- return BenchmarkState.stages.FINISHED;
</del><ins>+
+ results[Strings.json.samples] = samples[0].map(function(timestamp, i) {
+ var result = {
+ // Represent time in seconds
+ time: timestamp - this._startTimestamp,
+ complexity: samples[1][i]
+ };
+
+ // time offsets represented as FPS
+ if (i == 0)
+ result.fps = 60;
+ else
+ result.fps = 1000 / (timestamp - samples[0][i - 1]);
+
+ if (samples[2][i] != -1)
+ result.smoothedFPS = samples[2][i];
+
+ // Don't start adding data to the experiments until we reach the sampling timestamp
+ if (i >= samplingIndex) {
+ complexityExperiment.sample(result.complexity);
+ if (result.smoothedFPS && result.smoothedFPS != -1)
+ smoothedFPSExperiment.sample(result.smoothedFPS);
+ }
+
+ return result;
+ }, this);
+
+ results[Strings.json.score] = complexityExperiment.score(Experiment.defaults.CONCERN);
+
+ var complexityResults = {};
+ results[Strings.json.experiments.complexity] = complexityResults;
+ complexityResults[Strings.json.measurements.average] = complexityExperiment.mean();
+ complexityResults[Strings.json.measurements.concern] = complexityExperiment.concern(Experiment.defaults.CONCERN);
+ complexityResults[Strings.json.measurements.stdev] = complexityExperiment.standardDeviation();
+ complexityResults[Strings.json.measurements.percent] = complexityExperiment.percentage();
+
+ var smoothedFPSResults = {};
+ results[Strings.json.experiments.frameRate] = smoothedFPSResults;
+ smoothedFPSResults[Strings.json.measurements.average] = smoothedFPSExperiment.mean();
+ smoothedFPSResults[Strings.json.measurements.concern] = smoothedFPSExperiment.concern(Experiment.defaults.CONCERN);
+ smoothedFPSResults[Strings.json.measurements.stdev] = smoothedFPSExperiment.standardDeviation();
+ smoothedFPSResults[Strings.json.measurements.percent] = smoothedFPSExperiment.percentage();
</ins><span class="cx"> }
</span><del>-}
</del><ins>+});
</ins><span class="cx">
</span><ins>+FixedComplexityController = Utilities.createSubclass(Controller,
+ function(testInterval, benchmark)
+ {
+ Controller.call(this, testInterval, benchmark);
+ this.initialComplexity = benchmark.options["complexity"];
+ }
+);
+
+AdaptiveController = Utilities.createSubclass(Controller,
+ function(testInterval, benchmark)
+ {
+ // Data series: timestamp, complexity, estimatedIntervalFrameLength
+ Controller.call(this, testInterval, benchmark);
+
+ // All tests start at 0, so we expect to see 60 fps quickly.
+ this._samplingTimestamp = testInterval / 2;
+ this._startedSampling = false;
+ this._targetFrameRate = benchmark.options["frame-rate"];
+ this._pid = new PIDController(this._targetFrameRate);
+
+ this._intervalFrameCount = 0;
+ this._numberOfFramesToMeasurePerInterval = 4;
+ }, {
+
+ start: function(stage, startTimestamp)
+ {
+ Controller.prototype.start.call(this, stage, startTimestamp);
+
+ this._samplingTimestamp += startTimestamp;
+ this._intervalTimestamp = startTimestamp;
+ },
+
+ recordFirstSample: function(stage, startTimestamp)
+ {
+ this._sampler.record(startTimestamp, stage.complexity(), -1);
+ },
+
+ update: function(stage, timestamp)
+ {
+ if (!this._startedSampling && timestamp > this._samplingTimestamp) {
+ this._startedSampling = true;
+ this._sampler.mark(Strings.json.samplingTimeOffset, {
+ time: this._samplingTimestamp - this._startTimestamp
+ });
+ }
+
+ // Start the work for the next frame.
+ ++this._intervalFrameCount;
+
+ if (this._intervalFrameCount < this._numberOfFramesToMeasurePerInterval) {
+ this._sampler.record(timestamp, stage.complexity(), -1);
+ return;
+ }
+
+ // Adjust the test to reach the desired FPS.
+ var intervalLength = timestamp - this._intervalTimestamp;
+ this._estimator.sample(intervalLength / this._numberOfFramesToMeasurePerInterval);
+ var intervalEstimatedFrameRate = 1000 / this._estimator.estimate;
+ var tuneValue = -this._pid.tune(timestamp - this._startTimestamp, intervalLength, intervalEstimatedFrameRate);
+ tuneValue = tuneValue > 0 ? Math.floor(tuneValue) : Math.ceil(tuneValue);
+ stage.tune(tuneValue);
+
+ this._sampler.record(timestamp, stage.complexity(), intervalEstimatedFrameRate);
+
+ // Start the next interval.
+ this._intervalFrameCount = 0;
+ this._intervalTimestamp = timestamp;
+ },
+
+ processSamples: function(results)
+ {
+ Controller.prototype.processSamples.call(this, results);
+ results[Strings.json.targetFPS] = this._targetFrameRate;
+ }
+});
+
</ins><span class="cx"> Stage = Utilities.createClass(
</span><span class="cx"> function()
</span><span class="cx"> {
</span><span class="lines">@@ -180,100 +308,26 @@
</span><span class="cx"> }
</span><span class="cx"> });
</span><span class="cx">
</span><del>-Animator = Utilities.createClass(
- function()
- {
- this._intervalFrameCount = 0;
- this._numberOfFramesToMeasurePerInterval = 3;
- this._referenceTime = 0;
- this._currentTimeOffset = 0;
- }, {
-
- initialize: function(benchmark)
- {
- this._benchmark = benchmark;
-
- // Use Kalman filter to get a more non-fluctuating frame rate.
- if (benchmark.options["estimated-frame-rate"])
- this._estimator = new KalmanEstimator(60);
- else
- this._estimator = new IdentityEstimator;
- },
-
- get benchmark()
- {
- return this._benchmark;
- },
-
- _intervalTimeDelta: function()
- {
- return this._currentTimeOffset - this._startTimeOffset;
- },
-
- _shouldRequestAnotherFrame: function()
- {
- // Cadence is number of frames to measure, then one more frame to adjust the scene, and drop
- var currentTime = performance.now();
-
- if (!this._referenceTime)
- this._referenceTime = currentTime;
-
- this._currentTimeOffset = currentTime - this._referenceTime;
-
- if (!this._intervalFrameCount)
- this._startTimeOffset = this._currentTimeOffset;
-
- // Start the work for the next frame.
- ++this._intervalFrameCount;
-
- // Drop _dropFrameCount frames and measure the average of _measureFrameCount frames.
- if (this._intervalFrameCount <= this._numberOfFramesToMeasurePerInterval) {
- this._benchmark.record(this._currentTimeOffset, -1, -1);
- return true;
- }
-
- // Get the average FPS of _measureFrameCount frames over intervalTimeDelta.
- var intervalTimeDelta = this._intervalTimeDelta();
- var intervalFrameRate = 1000 / (intervalTimeDelta / this._numberOfFramesToMeasurePerInterval);
- var estimatedIntervalFrameRate = this._estimator.estimate(intervalFrameRate);
- // Record the complexity of the frame we just rendered. The next frame's time respresents the adjusted
- // complexity
- this._benchmark.record(this._currentTimeOffset, estimatedIntervalFrameRate, intervalFrameRate);
-
- // Adjust the test to reach the desired FPS.
- var shouldContinueRunning = this._benchmark.update(this._currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate);
-
- // Start the next drop/measure cycle.
- this._intervalFrameCount = 0;
-
- // If result is false, no more requestAnimationFrame() will be invoked.
- return shouldContinueRunning;
- },
-
- animateLoop: function()
- {
- if (this._shouldRequestAnotherFrame()) {
- this._benchmark.stage.animate(this._intervalTimeDelta());
- requestAnimationFrame(this.animateLoop.bind(this));
- }
- }
-});
-
</del><span class="cx"> Benchmark = Utilities.createClass(
</span><span class="cx"> function(stage, options)
</span><span class="cx"> {
</span><span class="cx"> this._options = options;
</span><ins>+ this._animateLoop = this._animateLoop.bind(this);
</ins><span class="cx">
</span><span class="cx"> this._stage = stage;
</span><span class="cx"> this._stage.initialize(this);
</span><del>- this._animator = new Animator();
- this._animator.initialize(this);
</del><span class="cx">
</span><del>- this._recordInterval = 200;
- this._isSampling = false;
- this._controller = new PIDController(this._options["frame-rate"]);
- this._sampler = new Sampler(4, 60 * this._options["test-interval"], this);
- this._state = new BenchmarkState(this._options["test-interval"] * 1000);
</del><ins>+ var testIntervalMilliseconds = options["test-interval"] * 1000;
+ switch (options["adjustment"])
+ {
+ case "fixed":
+ this._controller = new FixedComplexityController(testIntervalMilliseconds, this);
+ break;
+ case "adaptive":
+ default:
+ this._controller = new AdaptiveController(testIntervalMilliseconds, this);
+ break;
+ }
</ins><span class="cx"> }, {
</span><span class="cx">
</span><span class="cx"> get options()
</span><span class="lines">@@ -286,84 +340,19 @@
</span><span class="cx"> return this._stage;
</span><span class="cx"> },
</span><span class="cx">
</span><del>- get animator()
- {
- return this._animator;
- },
-
- // Called from the load event listener or from this.run().
- start: function()
- {
- this._animator.animateLoop();
- },
-
- // Called from the animator to adjust the complexity of the test.
- update: function(currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate)
- {
- this._state.update(currentTimeOffset);
-
- var stage = this._state.currentStage();
- if (stage == BenchmarkState.stages.FINISHED) {
- this._stage.clear();
- return false;
- }
-
- if (stage == BenchmarkState.stages.SAMPLING && !this._isSampling) {
- this._sampler.mark(Strings.json.samplingTimeOffset, {
- time: this._state.samplingTimeOffset() / 1000
- });
- this._isSampling = true;
- }
-
- var tuneValue = 0;
- if (this._options["adjustment"] == "fixed") {
- if (this._options["complexity"]) {
- // this._stage.tune(0) returns the current complexity of the test.
- tuneValue = this._options["complexity"] - this._stage.tune(0);
- }
- }
- else if (!(this._isSampling && this._options["adjustment"] == "fixed-after-warmup")) {
- // The relationship between frameRate and test complexity is inverse-proportional so we
- // need to use the negative of PIDController.tune() to change the complexity of the test.
- tuneValue = -this._controller.tune(currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate);
- tuneValue = tuneValue > 0 ? Math.floor(tuneValue) : Math.ceil(tuneValue);
- }
-
- this._stage.tune(tuneValue);
-
- if (typeof this._recordTimeOffset == "undefined")
- this._recordTimeOffset = currentTimeOffset;
-
- var stage = this._state.currentStage();
- if (stage != BenchmarkState.stages.FINISHED && currentTimeOffset < this._recordTimeOffset + this._recordInterval)
- return true;
-
- this._recordTimeOffset = currentTimeOffset;
- return true;
- },
-
- record: function(currentTimeOffset, estimatedFrameRate, intervalFrameRate)
- {
- // If the frame rate is -1 it means we are still recording for this sample
- this._sampler.record(currentTimeOffset, this.stage.complexity(), estimatedFrameRate, intervalFrameRate);
- },
-
</del><span class="cx"> run: function()
</span><span class="cx"> {
</span><span class="cx"> return this.waitUntilReady().then(function() {
</span><del>- this.start();
- var promise = new SimplePromise;
- var resolveWhenFinished = function() {
- if (typeof this._state != "undefined" && (this._state.currentStage() == BenchmarkState.stages.FINISHED))
- return promise.resolve(this._sampler);
- setTimeout(resolveWhenFinished, 50);
- }.bind(this);
-
- resolveWhenFinished();
- return promise;
</del><ins>+ this._finishPromise = new SimplePromise;
+ this._previousTimestamp = performance.now();
+ this._didWarmUp = false;
+ this._stage.tune(this._controller.initialComplexity - this._stage.complexity());
+ this._animateLoop();
+ return this._finishPromise;
</ins><span class="cx"> }.bind(this));
</span><span class="cx"> },
</span><span class="cx">
</span><ins>+ // Subclasses should override this if they have setup to do prior to commencing.
</ins><span class="cx"> waitUntilReady: function()
</span><span class="cx"> {
</span><span class="cx"> var promise = new SimplePromise;
</span><span class="lines">@@ -371,55 +360,30 @@
</span><span class="cx"> return promise;
</span><span class="cx"> },
</span><span class="cx">
</span><del>- processSamples: function(results)
</del><ins>+ _animateLoop: function()
</ins><span class="cx"> {
</span><del>- var complexity = new Experiment;
- var smoothedFPS = new Experiment;
- var samplingIndex = 0;
</del><ins>+ this._currentTimestamp = performance.now();
</ins><span class="cx">
</span><del>- var samplingMark = this._sampler.marks[Strings.json.samplingTimeOffset];
- if (samplingMark) {
- samplingIndex = samplingMark.index;
- results[Strings.json.samplingTimeOffset] = samplingMark.time;
- }
-
- results[Strings.json.samples] = this._sampler.samples[0].map(function(d, i) {
- var result = {
- // time offsets represented as seconds
- time: d/1000,
- complexity: this._sampler.samples[1][i]
- };
-
- // time offsets represented as FPS
- if (i == 0)
- result.fps = 60;
- else
- result.fps = 1000 / (d - this._sampler.samples[0][i - 1]);
-
- var smoothedFPSresult = this._sampler.samples[2][i];
- if (smoothedFPSresult != -1) {
- result.smoothedFPS = smoothedFPSresult;
- result.intervalFPS = this._sampler.samples[3][i];
</del><ins>+ if (!this._didWarmUp) {
+ if (this._currentTimestamp - this._previousTimestamp >= 100) {
+ this._didWarmUp = true;
+ this._controller.start(this._stage, this._currentTimestamp);
+ this._previousTimestamp = this._currentTimestamp;
</ins><span class="cx"> }
</span><span class="cx">
</span><del>- if (i >= samplingIndex) {
- complexity.sample(result.complexity);
- if (smoothedFPSresult != -1) {
- smoothedFPS.sample(smoothedFPSresult);
- }
- }
</del><ins>+ this._stage.animate(0);
+ requestAnimationFrame(this._animateLoop);
+ return;
+ }
</ins><span class="cx">
</span><del>- return result;
- }, this);
</del><ins>+ this._controller.update(this._stage, this._currentTimestamp);
+ if (this._controller.shouldStop(this._currentTimestamp)) {
+ this._finishPromise.resolve(this._controller.results());
+ return;
+ }
</ins><span class="cx">
</span><del>- results[Strings.json.score] = complexity.score(Experiment.defaults.CONCERN);
- [complexity, smoothedFPS].forEach(function(experiment, index) {
- var jsonExperiment = !index ? Strings.json.experiments.complexity : Strings.json.experiments.frameRate;
- results[jsonExperiment] = {};
- results[jsonExperiment][Strings.json.measurements.average] = experiment.mean();
- results[jsonExperiment][Strings.json.measurements.concern] = experiment.concern(Experiment.defaults.CONCERN);
- results[jsonExperiment][Strings.json.measurements.stdev] = experiment.standardDeviation();
- results[jsonExperiment][Strings.json.measurements.percent] = experiment.percentage();
- });
</del><ins>+ this._stage.animate(this._currentTimestamp - this._previousTimestamp);
+ this._previousTimestamp = this._currentTimestamp;
+ requestAnimationFrame(this._animateLoop);
</ins><span class="cx"> }
</span><span class="cx"> });
</span></span></pre></div>
<a id="trunkPerformanceTestsAnimometertestsresourcesmathjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/tests/resources/math.js (196287 => 196288)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/tests/resources/math.js 2016-02-09 03:04:20 UTC (rev 196287)
+++ trunk/PerformanceTests/Animometer/tests/resources/math.js 2016-02-09 03:25:39 UTC (rev 196288)
</span><span class="lines">@@ -1,226 +1,41 @@
</span><del>-var Matrix =
-{
- init: function(m, n, v)
- {
- return Array(m * n).fill(v);
- },
</del><ins>+SimpleKalmanEstimator = Utilities.createClass(
+ function(processError, measurementError) {
+ this._initialized = false;
</ins><span class="cx">
</span><del>- zeros: function(m, n)
- {
- return Matrix.init(m, n, 0);
- },
</del><ins>+ var error = .5 * (Math.sqrt(processError * processError + 4 * processError * measurementError) - processError);
+ this._gain = error / (error + measurementError);
+ }, {
</ins><span class="cx">
</span><del>- ones: function(m, n)
</del><ins>+ sample: function(newMeasurement)
</ins><span class="cx"> {
</span><del>- return Matrix.init(m, n, 1);
- },
-
- identity: function(n)
- {
- var out = new Matrix.zeros(n, n);
- for (var i = 0; i < n; ++i)
- out[i * n + i] = 1;
- return out;
- },
-
- str: function(A, n, m)
- {
- var out = (m > 1 && n > 1 ? "Matrix[" + n + ", " + m : "Vector[" + m * n) + "] = [";
- for (var i = 0; i < n * m; ++i) {
- out += A[i];
- if (i < n * m - 1)
- out += ", ";
</del><ins>+ if (!this._initialized) {
+ this._initialized = true;
+ this._estimatedMeasurement = newMeasurement;
+ return;
</ins><span class="cx"> }
</span><del>- return out + "]";
- },
</del><span class="cx">
</span><del>- pos: function(m, i, j)
- {
- return m * i + j;
</del><ins>+ this._estimatedMeasurement = this._estimatedMeasurement + this._gain * (newMeasurement - this._estimatedMeasurement);
</ins><span class="cx"> },
</span><span class="cx">
</span><del>- add: function(A, B, n, m)
</del><ins>+ get estimate()
</ins><span class="cx"> {
</span><del>- var out = Matrix.zeros(n, m);
- for (var i = 0; i < n * m; ++i)
- out[i] = A[i] + B[i];
- return out;
- },
-
- subtract: function(A, B, n, m)
- {
- var out = Matrix.zeros(n, m);
- for (var i = 0; i < n * m; ++i)
- out[i] = A[i] - B[i];
- return out;
- },
-
- scale: function(s, A, n, m)
- {
- var out = Matrix.zeros(n, m);
- for (var i = 0; i < n * m; ++i)
- out[i] = s * A[i];
- return out;
- },
-
- transpose: function(A, n, m)
- {
- var out = Matrix.zeros(m, n);
- for (var i = 0; i < n; ++i) {
- for (var j = 0; j < m; ++j)
- out[Matrix.pos(n, i, j)] = A[Matrix.pos(m, j, i)];
- }
- return out;
- },
-
- multiply: function(A, B, n, m, p)
- {
- var out = Matrix.zeros(n, p);
- for (var i = 0; i < n; ++i) {
- for (var j = 0; j < p; ++j) {
- for (var k = 0; k < m; ++k) {
- out[Matrix.pos(p, i, j)] += A[Matrix.pos(m, i, k)] * B[Matrix.pos(p, k, j)];
- }
- }
- }
- return out;
</del><ins>+ return this._estimatedMeasurement;
</ins><span class="cx"> }
</span><del>-}
</del><ins>+});
</ins><span class="cx">
</span><del>-var Vector3 =
-{
- zeros: function()
</del><ins>+PIDController = Utilities.createClass(
+ function(ysp)
</ins><span class="cx"> {
</span><del>- return Matrix.zeros(1, 3);
- },
</del><ins>+ this._ysp = ysp;
+ this._out = 0;
</ins><span class="cx">
</span><del>- ones: function()
- {
- return Matrix.ones(1, 3);
- },
</del><ins>+ this._Kp = 0;
+ this._stage = PIDController.stages.WARMING;
</ins><span class="cx">
</span><del>- str: function(v)
- {
- return Matrix.str(v, 1, 3);
- },
</del><ins>+ this._eold = 0;
+ this._I = 0;
+ }, {
</ins><span class="cx">
</span><del>- add: function(v, w)
- {
- return Matrix.add(v, w, 1, 3);
- },
-
- subtract: function(v, w)
- {
- return Matrix.subtract(v, w, 1, 3);
- },
-
- scale: function(s, v)
- {
- return Matrix.scale(s, v, 1, 3);
- },
-
- multiplyMatrix3: function(v, A)
- {
- return Matrix.multiply(v, A, 1, 3, 3);
- },
-
- multiplyVector3: function(v, w)
- {
- var out = 0;
- for (var i = 0; i < 3; ++i)
- out += v[i] * w[i];
- return out;
- }
-}
-
-var Matrix3 =
-{
- zeros: function()
- {
- return Matrix.zeros(3, 3);
- },
-
- identity: function()
- {
- return Matrix.identity(3, 3);
- },
-
- str: function(A)
- {
- return Matrix.str(A, 3, 3);
- },
-
- pos: function(i, j)
- {
- return Matrix.pos(3, i, j);
- },
-
- add: function(A, B)
- {
- return Matrix.add(A, B, 3, 3);
- },
-
- subtract: function(A, B)
- {
- return Matrix.subtract(A, B, 3, 3);
- },
-
- scale: function(s, A)
- {
- return Matrix.scale(s, A, 3, 3);
- },
-
- transpose: function(A)
- {
- return Matrix.transpose(A, 3, 3);
- },
-
- multiplyMatrix3: function(A, B)
- {
- return Matrix.multiply(A, B, 3, 3, 3);
- },
-
- multiplyVector3: function(A, v)
- {
- return Matrix.multiply(A, v, 3, 3, 1);
- }
-}
-
-function PIDController(ysp)
-{
- this._ysp = ysp;
- this._out = 0;
-
- this._Kp = 0;
- this._stage = PIDController.stages.WARMING;
-
- this._eold = 0;
- this._I = 0;
-}
-
-// This enum will be used to tell whether the system output (or the controller input)
-// is moving towards the set-point or away from it.
-PIDController.yPositions = {
- BEFORE_SETPOINT: 0,
- AFTER_SETPOINT: 1
-}
-
-// The Ziegler–Nichols method for is used tuning the PID controller. The workflow of
-// the tuning is split into four stages. The first two stages determine the values
-// of the PID controller gains. During these two stages we return the proportional
-// term only. The third stage is used to determine the min-max values of the
-// saturation actuator. In the last stage back-calculation and tracking are applied
-// to avoid integrator windup. During the last two stages, we return a PID control
-// value.
-PIDController.stages = {
- WARMING: 0, // Increase the value of the Kp until the system output reaches ysp.
- OVERSHOOT: 1, // Measure the oscillation period and the overshoot value
- UNDERSHOOT: 2, // Return PID value and measure the undershoot value
- SATURATE: 3 // Return PID value and apply back-calculation and tracking.
-}
-
-PIDController.prototype =
-{
</del><span class="cx"> // Determines whether the current y is
</span><span class="cx"> // before ysp => (below ysp if ysp > y0) || (above ysp if ysp < y0)
</span><span class="cx"> // after ysp => (above ysp if ysp > y0) || (below ysp if ysp < y0)
</span><span class="lines">@@ -410,7 +225,7 @@
</span><span class="cx"> return u;
</span><span class="cx"> },
</span><span class="cx">
</span><del>- // Called from the benchmark to tune its test. It uses Ziegler–Nichols method
</del><ins>+ // Called from the benchmark to tune its test. It uses Ziegler-Nichols method
</ins><span class="cx"> // to calculate the controller parameters. It then returns a PID tuning value.
</span><span class="cx"> tune: function(t, h, y)
</span><span class="cx"> {
</span><span class="lines">@@ -426,66 +241,27 @@
</span><span class="cx"> // Apply back-calculation and tracking to avoid integrator windup
</span><span class="cx"> return this._saturate(v, e);
</span><span class="cx"> }
</span><del>-}
</del><ins>+});
</ins><span class="cx">
</span><del>-function KalmanEstimator(initX)
-{
- // Initialize state transition matrix.
- this._matA = Matrix3.identity();
- this._matA[Matrix3.pos(0, 2)] = 1;
</del><ins>+Utilities.extendObject(PIDController, {
+ // This enum will be used to tell whether the system output (or the controller input)
+ // is moving towards the set-point or away from it.
+ yPositions: {
+ BEFORE_SETPOINT: 0,
+ AFTER_SETPOINT: 1
+ },
</ins><span class="cx">
</span><del>- // Initialize measurement matrix.
- this._vecH = Vector3.zeros();
- this._vecH[0] = 1;
-
- this._matQ = Matrix3.identity();
- this._R = 1000;
-
- // Initial state conditions.
- this._vecX_est = Vector3.zeros();
- this._vecX_est[0] = initX;
- this._matP_est = Matrix3.zeros();
-}
-
-KalmanEstimator.prototype =
-{
- estimate: function(current)
- {
- // Project the state ahead
- // X_prd(k) = A * X_est(k-1)
- var vecX_prd = Matrix3.multiplyVector3(this._matA, this._vecX_est);
-
- // Project the error covariance ahead
- // P_prd(k) = A * P_est(k-1) * A' + Q
- var matP_prd = Matrix3.add(Matrix3.multiplyMatrix3(Matrix3.multiplyMatrix3(this._matA, this._matP_est), Matrix3.transpose(this._matA)), this._matQ);
-
- // Compute Kalman gain
- // B = H * P_prd(k)';
- // S = B * H' + R;
- // K(k) = (S \ B)';
- var vecB = Vector3.multiplyMatrix3(this._vecH, Matrix3.transpose(matP_prd));
- var S = Vector3.multiplyVector3(vecB, this._vecH) + this._R;
- var vecGain = Vector3.scale(1/S, vecB);
-
- // Update the estimate via z(k)
- // X_est(k) = x_prd + K(k) * (z(k) - H * X_prd(k));
- this._vecX_est = Vector3.add(vecX_prd, Vector3.scale(current - Vector3.multiplyVector3(this._vecH, vecX_prd), vecGain));
-
- // Update the error covariance
- // P_est(k) = P_prd(k) - K(k) * H * P_prd(k);
- this._matP_est = Matrix3.subtract(matP_prd, Matrix3.scale(Vector3.multiplyVector3(vecGain, this._vecH), matP_prd));
-
- // Compute the estimated measurement.
- // y = H * X_est(k);
- return Vector3.multiplyVector3(this._vecH, this._vecX_est);
</del><ins>+ // The Ziegler-Nichols method for is used tuning the PID controller. The workflow of
+ // the tuning is split into four stages. The first two stages determine the values
+ // of the PID controller gains. During these two stages we return the proportional
+ // term only. The third stage is used to determine the min-max values of the
+ // saturation actuator. In the last stage back-calculation and tracking are applied
+ // to avoid integrator windup. During the last two stages, we return a PID control
+ // value.
+ stages: {
+ WARMING: 0, // Increase the value of the Kp until the system output reaches ysp.
+ OVERSHOOT: 1, // Measure the oscillation period and the overshoot value
+ UNDERSHOOT: 2, // Return PID value and measure the undershoot value
+ SATURATE: 3 // Return PID value and apply back-calculation and tracking.
</ins><span class="cx"> }
</span><del>-}
-
-function IdentityEstimator() {}
-IdentityEstimator.prototype =
-{
- estimate: function(current)
- {
- return current;
- }
-};
</del><ins>+});
</ins></span></pre></div>
<a id="trunkPerformanceTestsAnimometertestsresourcessamplerjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/Animometer/tests/resources/sampler.js (196287 => 196288)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/Animometer/tests/resources/sampler.js 2016-02-09 03:04:20 UTC (rev 196287)
+++ trunk/PerformanceTests/Animometer/tests/resources/sampler.js 2016-02-09 03:25:39 UTC (rev 196288)
</span><span class="lines">@@ -81,13 +81,10 @@
</span><span class="cx"> this.marks[comment] = data;
</span><span class="cx"> },
</span><span class="cx">
</span><del>- process: function(options)
</del><ins>+ process: function()
</ins><span class="cx"> {
</span><span class="cx"> var results = {};
</span><span class="cx">
</span><del>- if (options["adjustment"] == "adaptive")
- results[Strings.json.targetFPS] = +options["frame-rate"];
-
</del><span class="cx"> // Remove unused capacity
</span><span class="cx"> this.samples = this.samples.map(function(array) {
</span><span class="cx"> return array.slice(0, this.sampleCount);
</span></span></pre></div>
<a id="trunkPerformanceTestsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/ChangeLog (196287 => 196288)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/ChangeLog 2016-02-09 03:04:20 UTC (rev 196287)
+++ trunk/PerformanceTests/ChangeLog 2016-02-09 03:25:39 UTC (rev 196288)
</span><span class="lines">@@ -1,3 +1,105 @@
</span><ins>+2016-02-07 Jon Lee <jonlee@apple.com>
+
+ Update how the benchmark is run
+ https://bugs.webkit.org/show_bug.cgi?id=153960
+
+ Provisionally reviewed by Said Abou-Hallawa.
+
+ Introduce the notion of a Controller. It is responsible for recording, updating,
+ and processing the statistics and complexity of the benchmark. This allows
+ plugging in different Controllers.
+
+ This strips most of the functionality from Animator and BenchmarkState, so fold
+ what's left into Benchmark. Now, Benchmarks only own a stage and a controller, but
+ are responsible for driving the animation loop.
+
+ Rewrite Animator._shouldRequestAnotherFrame into two different Controllers. One
+ maintains a fixed complexity, and the other adapts the complexity to meet a
+ fixed FPS.
+
+ Fix the Kalman estimator to be modeled on a scalar variable with no model.
+
+ * Animometer/tests/resources/main.js: Remove BenchmarkState and Animator, and
+ replace it with a Controller. Add a FixedController and refactor the previous controller
+ to an AdaptiveController.
+
+ (Controller): Controllers own the estimator and the sampler. When a new frame is
+ displayed, the animation loop calls update(). The estimator and sampler record
+ stats, then tune. Samplers can track multiple series of data. The basic controller
+ tracks timestamp, complexity, and estimated frame rate.
+ The Kalman estimation is based on the frame length rather than the frame
+ rate. Because FPS is inverse proportional to frame length, in the case where the measured
+ frame length is very small, the FPS ends up being a wildly large number (in the order of
+ 600-1000 "FPS"), and it pulls the estimator up drastically enough that it takes a while
+ for it to settle back down. Using frame length reduces the impact of these spikes.
+ Converging the estimation takes enough time to avoid initializing it immediately
+ when the benchmark starts. Instead, the benchmark runs for a brief period of time (100ms)
+ before running it in earnest. Allow controllers an opportunity to set the complexity
+ before starting recording.
+ When the benchmark is complete, the controller has an opportunity to process
+ the samples. The default implementation calculates the raw FPS based on the time
+ difference of the samples, and calculates the complexity score. This is moved from
+ Benchmark.processSamples.
+
+ (Controller): Initialize timestamps. These are at first relative to the start of the
+ benchmark, but are offset by the absolute start time during start(). By default maintain
+ 3 data series, but subclasses can override.
+ (start): Calls recordFirstSample() for subclasses to override if needed.
+ (recordFirstSample): For basic controller, start sampling at the beginning.
+ (update): Update the frame length estimator and sample.
+ (shouldStop): Checks that the time is before _endTimestamp.
+ (results): Returns the processed samples.
+ (processSamples): Iterate through the sample data and collate them. Include scores.
+
+ (FixedComplexityController): Controller that tunes the stage to the desired complexity
+ prior to starting, and keeps it at that complexity.
+
+ (AdaptiveController): Have the estimator estimate the interval frame rate instead of the
+ raw frame rate.
+ The previous version of this controller ignored the frame that came after the
+ adjustment. The raw FPS show that whatever noise the scene change adds is negligible
+ compared to the noise of the system overall. Stop ignoring that frame and include all
+ frames in the measurements.
+
+ (Benchmark): Remove dependency on animator, and instantiate a runner based on what is
+ selected. Most of the loop's functionality is in Controller, so remove here.
+ (Benchmark.run): Remove start() since it is only called from run(), and fold it in here.
+ (Benchmark._animateLoop): Fold in from Animator.animateLoop. Let the benchmark run for
+ a brief period before calling Controller.start().
+
+ * Animometer/tests/resources/math.js: Fix the Kalman estimator. The filter estimates
+ a scalar variable, and makes basic assumptions regarding the model. As a result
+ none of the linear algebra classes are needed, so remove Matrix, Vector3, and Matrix3.
+ (SimpleKalmanEstimator): Calculate the gain based on the provided process and
+ measurement errors.
+ (KalmanEstimator): Deleted.
+ (IdentityEstimator): Deleted.
+ (PIDController): Refactor to use the Utilities.createClass() helper.
+
+ The Kalman filter algorithm is explained here http://greg.czerniak.info/guides/kalman1/.
+ The state, represented by a scalar, is the estimated frame length. There is no user
+ transition of the state, and the state is the same as the measurement. With this model,
+ the estimation error converges, so calculate the gain ahead of time.
+
+ * Animometer/developer.html: Remove fixed-after-warmup since it is not useful.
+ Replace the option to toggle the estimator, and make it possible to customize the
+ estimator's error parameters. Show raw FPS by default, and remove interval FPS,
+ which will be shown instead of the filtered raw FPS.
+ * Animometer/resources/debug-runner/animometer.css: Put the header behind the graph.
+ Remove #intervalFPS rules; move the color to #filteredFPS.
+ * Animometer/resources/debug-runner/graph.js:
+ (updateGraphData): Update the hr style to force the layout to be calculated
+ correctly. Change the tick format to be in terms of seconds, since the timestamps
+ are in milliseconds. Remove interval data.
+ * Animometer/resources/runner/animometer.js:
+ (window.benchmarkController.startBenchmark): Set Kalman parameters.
+ * Animometer/resources/runner/benchmark-runner.js:
+ (_runBenchmarkAndRecordResults): When a benchmark completes, expect it to return
+ the final data, rather than passing a sampler from the controller. This avoids
+ needing to expose the sampler variable in the benchmark.
+ * Animometer/tests/resources/sampler.js:
+ (process): Move the setting of the target frame rate to AdaptiveController.
+
</ins><span class="cx"> 2016-02-06 Jon Lee <jonlee@apple.com>
</span><span class="cx">
</span><span class="cx"> Code clean up: Move Rotater function closer to Stage static methods.
</span></span></pre>
</div>
</div>
</body>
</html>