<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[194738] trunk/Websites/perf.webkit.org</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/194738">194738</a></dd>
<dt>Author</dt> <dd>rniwa@webkit.org</dd>
<dt>Date</dt> <dd>2016-01-07 15:47:14 -0800 (Thu, 07 Jan 2016)</dd>
</dl>
<h3>Log Message</h3>
<pre>Perf dashboard should automatically add aggregators
https://bugs.webkit.org/show_bug.cgi?id=152818
Reviewed by Chris Dumez.
When an aggregator entry is missing in aggregators table, automatically insert it in /api/report.
In a very early version of the perf dashboard, we had the ability to define a custom aggregator
in an admin page. In practice, nobody used or needed this feature so we got rid of it even before
the dashboard was landed into WebKit repository. This patch cleans up that mess.
* run-tests.js:
(main): Added the filtering capability.
(TestEnvironment): Expose the config JSON in the test environment.
* public/include/report-processor.php:
(ReportProcessor): Renamed name_to_aggregator now that it only contains ID.
(ReportProcessor::__construct): No longer fetches the aggregator table. An equivalent work is done
in newly added ensure_aggregators.
(ReportProcessor::process): Calls ensure_aggregators which populates name_to_aggregator_id.
(ReportProcessor::ensure_aggregators): Added. Add the builtin aggregators: Arithmetic, Geometric,
Harmonic, and Total.
(TestRunsGenerator): Renamed name_to_aggregator now that it only contains ID.
(TestRunsGenerator::__construct):
(TestRunsGenerator::add_aggregated_metric): Don't include aggregator_definition here since it's
never used now that all the aggregations are done natively in PHP.
(TestRunsGenerator::$aggregators): Added. We don't include SquareSum since it's only used for
computing run_square_sum_cache in test_runs table and it's useless elsewhere.
(TestRunsGenerator::aggregate_values): Add a comment about that.
* tests/api-report.js: Updated a test case to reflect the change.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkWebsitesperfwebkitorgChangeLog">trunk/Websites/perf.webkit.org/ChangeLog</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicincludereportprocessorphp">trunk/Websites/perf.webkit.org/public/include/report-processor.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgruntestsjs">trunk/Websites/perf.webkit.org/run-tests.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgtestsapireportjs">trunk/Websites/perf.webkit.org/tests/api-report.js</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkWebsitesperfwebkitorgChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/ChangeLog (194737 => 194738)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/ChangeLog 2016-01-07 23:44:27 UTC (rev 194737)
+++ trunk/Websites/perf.webkit.org/ChangeLog 2016-01-07 23:47:14 UTC (rev 194738)
</span><span class="lines">@@ -1,3 +1,37 @@
</span><ins>+2016-01-07 Ryosuke Niwa <rniwa@webkit.org>
+
+ Perf dashboard should automatically add aggregators
+ https://bugs.webkit.org/show_bug.cgi?id=152818
+
+ Reviewed by Chris Dumez.
+
+ When an aggregator entry is missing in aggregators table, automatically insert it in /api/report.
+
+ In a very early version of the perf dashboard, we had the ability to define a custom aggregator
+ in an admin page. In practice, nobody used or needed this feature so we got rid of it even before
+ the dashboard was landed into WebKit repository. This patch cleans up that mess.
+
+ * run-tests.js:
+ (main): Added the filtering capability.
+ (TestEnvironment): Expose the config JSON in the test environment.
+
+ * public/include/report-processor.php:
+ (ReportProcessor): Renamed name_to_aggregator now that it only contains ID.
+ (ReportProcessor::__construct): No longer fetches the aggregator table. An equivalent work is done
+ in newly added ensure_aggregators.
+ (ReportProcessor::process): Calls ensure_aggregators which populates name_to_aggregator_id.
+ (ReportProcessor::ensure_aggregators): Added. Add the builtin aggregators: Arithmetic, Geometric,
+ Harmonic, and Total.
+ (TestRunsGenerator): Renamed name_to_aggregator now that it only contains ID.
+ (TestRunsGenerator::__construct):
+ (TestRunsGenerator::add_aggregated_metric): Don't include aggregator_definition here since it's
+ never used now that all the aggregations are done natively in PHP.
+ (TestRunsGenerator::$aggregators): Added. We don't include SquareSum since it's only used for
+ computing run_square_sum_cache in test_runs table and it's useless elsewhere.
+ (TestRunsGenerator::aggregate_values): Add a comment about that.
+
+ * tests/api-report.js: Updated a test case to reflect the change.
+
</ins><span class="cx"> 2016-01-06 Ryosuke Niwa <rniwa@webkit.org>
</span><span class="cx">
</span><span class="cx"> Perf dashboard JSON API should fail gracefully when postgres is down
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicincludereportprocessorphp"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/include/report-processor.php (194737 => 194738)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/include/report-processor.php 2016-01-07 23:44:27 UTC (rev 194737)
+++ trunk/Websites/perf.webkit.org/public/include/report-processor.php 2016-01-07 23:47:14 UTC (rev 194738)
</span><span class="lines">@@ -4,19 +4,13 @@
</span><span class="cx">
</span><span class="cx"> class ReportProcessor {
</span><span class="cx"> private $db;
</span><del>- private $name_to_aggregator;
</del><ins>+ private $name_to_aggregator_id;
</ins><span class="cx"> private $report_id;
</span><span class="cx"> private $runs;
</span><span class="cx">
</span><span class="cx"> function __construct($db) {
</span><span class="cx"> $this->db = $db;
</span><del>- $this->name_to_aggregator = array();
- $aggregator_table = $db->fetch_table('aggregators');
- if ($aggregator_table) {
- foreach ($aggregator_table as $aggregator_row) {
- $this->name_to_aggregator[$aggregator_row['aggregator_name']] = $aggregator_row;
- }
- }
</del><ins>+ $this->name_to_aggregator_id = array();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> private function exit_with_error($message, $details = NULL) {
</span><span class="lines">@@ -78,7 +72,9 @@
</span><span class="cx"> if (!$existing_report_id)
</span><span class="cx"> $this->store_report($report, $build_data);
</span><span class="cx">
</span><del>- $this->runs = new TestRunsGenerator($this->db, $this->name_to_aggregator, $this->report_id);
</del><ins>+ $this->ensure_aggregators();
+
+ $this->runs = new TestRunsGenerator($this->db, $this->name_to_aggregator_id, $this->report_id);
</ins><span class="cx"> $this->recursively_ensure_tests($report['tests']);
</span><span class="cx">
</span><span class="cx"> $this->runs->aggregate();
</span><span class="lines">@@ -113,6 +109,15 @@
</span><span class="cx"> $this->exit_with_error('FailedToStoreRunReport');
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ private function ensure_aggregators() {
+ foreach (TestRunsGenerator::$aggregators as $name) {
+ $id = $this->db->select_or_insert_row('aggregators', 'aggregator', array('name' => $name));
+ if (!$id)
+ $this->exit_with_error('FailedToInsertAggregator', array('aggregator' => $name));
+ $this->name_to_aggregator_id[$name] = $id;
+ }
+ }
+
</ins><span class="cx"> private function resolve_build_id($build_data, $revisions, $build_request_id) {
</span><span class="cx"> // FIXME: This code has a race condition. See <rdar://problem/15876303>.
</span><span class="cx"> $results = $this->db->query_and_fetch_all("SELECT build_id, build_slave FROM builds
</span><span class="lines">@@ -209,15 +214,15 @@
</span><span class="cx">
</span><span class="cx"> class TestRunsGenerator {
</span><span class="cx"> private $db;
</span><del>- private $name_to_aggregator;
</del><ins>+ private $name_to_aggregator_id;
</ins><span class="cx"> private $report_id;
</span><span class="cx"> private $metrics_to_aggregate;
</span><span class="cx"> private $parent_to_values;
</span><span class="cx"> private $values_to_commit;
</span><span class="cx">
</span><del>- function __construct($db, $name_to_aggregator, $report_id) {
</del><ins>+ function __construct($db, $name_to_aggregator_id, $report_id) {
</ins><span class="cx"> $this->db = $db;
</span><del>- $this->name_to_aggregator = $name_to_aggregator or array();
</del><ins>+ $this->name_to_aggregator_id = $name_to_aggregator_id;
</ins><span class="cx"> $this->report_id = $report_id;
</span><span class="cx"> $this->metrics_to_aggregate = array();
</span><span class="cx"> $this->parent_to_values = array();
</span><span class="lines">@@ -232,10 +237,11 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> function add_aggregated_metric($parent_id, $test_id, $test_name, $metric_name, $aggregator_name, $level) {
</span><del>- array_key_exists($aggregator_name, $this->name_to_aggregator) or $this->exit_with_error('AggregatorNotFound', array('name' => $aggregator_name));
</del><ins>+ array_key_exists($aggregator_name, $this->name_to_aggregator_id)
+ or $this->exit_with_error('AggregatorNotFound', array('name' => $aggregator_name));
</ins><span class="cx">
</span><span class="cx"> $metric_id = $this->db->select_or_insert_row('test_metrics', 'metric', array('name' => $metric_name,
</span><del>- 'test' => $test_id, 'aggregator' => $this->name_to_aggregator[$aggregator_name]['aggregator_id']));
</del><ins>+ 'test' => $test_id, 'aggregator' => $this->name_to_aggregator_id[$aggregator_name]));
</ins><span class="cx"> if (!$metric_id)
</span><span class="cx"> $this->exit_with_error('FailedToAddAggregatedMetric', array('name' => $metric_name, 'test' => $test_id, 'aggregator' => $aggregator_name));
</span><span class="cx">
</span><span class="lines">@@ -246,7 +252,6 @@
</span><span class="cx"> 'test_name' => $test_name,
</span><span class="cx"> 'metric_name' => $metric_name,
</span><span class="cx"> 'aggregator' => $aggregator_name,
</span><del>- 'aggregator_definition' => $this->name_to_aggregator[$aggregator_name]['aggregator_definition'],
</del><span class="cx"> 'level' => $level));
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -355,6 +360,8 @@
</span><span class="cx"> return array('values' => $values_by_iterations, 'group_sizes' => $group_sizes);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ static public $aggregators = array('Arithmetic', 'Geometric', 'Harmonic', 'Total');
+
</ins><span class="cx"> private function aggregate_values($aggregator, $values) {
</span><span class="cx"> switch ($aggregator) {
</span><span class="cx"> case 'Arithmetic':
</span><span class="lines">@@ -365,7 +372,7 @@
</span><span class="cx"> return count($values) / array_sum(array_map(function ($x) { return 1 / $x; }, $values));
</span><span class="cx"> case 'Total':
</span><span class="cx"> return array_sum($values);
</span><del>- case 'SquareSum':
</del><ins>+ case 'SquareSum': # This aggregator is only used internally to compute run_square_sum_cache in test_runs table.
</ins><span class="cx"> return array_sum(array_map(function ($x) { return $x * $x; }, $values));
</span><span class="cx"> default:
</span><span class="cx"> $this->exit_with_error('UnknownAggregator', array('aggregator' => $aggregator));
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgruntestsjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/run-tests.js (194737 => 194738)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/run-tests.js 2016-01-07 23:44:27 UTC (rev 194737)
+++ trunk/Websites/perf.webkit.org/run-tests.js 2016-01-07 23:47:14 UTC (rev 194738)
</span><span class="lines">@@ -88,8 +88,9 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>-function main() {
</del><ins>+function main(argv) {
</ins><span class="cx"> var client = connect(true);
</span><ins>+ var filter = argv[2];
</ins><span class="cx"> confirmUserWantsDatabaseToBeInitializedIfNeeded(client, function (error, shouldContinue) {
</span><span class="cx"> if (error)
</span><span class="cx"> console.error(error);
</span><span class="lines">@@ -110,7 +111,7 @@
</span><span class="cx"> var testCaseQueue = new SerializedTaskQueue();
</span><span class="cx"> var testFileQueue = new SerializedTaskQueue();
</span><span class="cx"> fs.readdirSync(pathToTests()).map(function (testFile) {
</span><del>- if (!testFile.match(/.js$/))
</del><ins>+ if (!testFile.match(/.js$/) || (filter && testFile.indexOf(filter) != 0))
</ins><span class="cx"> return;
</span><span class="cx"> testFileQueue.addTask(function (error, callback) {
</span><span class="cx"> var testContent = fs.readFileSync(pathToTests(testFile), 'utf-8');
</span><span class="lines">@@ -282,6 +283,8 @@
</span><span class="cx"> return hash.digest('hex');
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ this.config = config;
+
</ins><span class="cx"> this.notifyDone = function () { currentTestContext.done(); }
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -334,4 +337,4 @@
</span><span class="cx"> process.stdout.write(this.description() + ': ');
</span><span class="cx"> }
</span><span class="cx">
</span><del>-main();
</del><ins>+main(process.argv);
</ins></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtestsapireportjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/tests/api-report.js (194737 => 194738)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tests/api-report.js 2016-01-07 23:44:27 UTC (rev 194737)
+++ trunk/Websites/perf.webkit.org/tests/api-report.js 2016-01-07 23:47:14 UTC (rev 194738)
</span><span class="lines">@@ -372,12 +372,12 @@
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><del>- it("should reject when aggregators are missing", function () {
</del><ins>+ it("should not reject when aggregators are missing", function () {
</ins><span class="cx"> addBuilder(reportWithTwoLevelsOfAggregations, function () {
</span><span class="cx"> postJSON('/api/report/', reportWithTwoLevelsOfAggregations, function (response) {
</span><span class="cx"> assert.equal(response.statusCode, 200);
</span><del>- assert.equal(JSON.parse(response.responseText)['status'], 'AggregatorNotFound');
- assert.equal(JSON.parse(response.responseText)['failureStored'], true);
</del><ins>+ assert.equal(JSON.parse(response.responseText)['status'], 'OK');
+ assert.equal(JSON.parse(response.responseText)['failureStored'], false);
</ins><span class="cx"> notifyDone();
</span><span class="cx"> });
</span><span class="cx"> });
</span></span></pre>
</div>
</div>
</body>
</html>