<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[183232] trunk/Websites/perf.webkit.org</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/183232">183232</a></dd>
<dt>Author</dt> <dd>rniwa@webkit.org</dd>
<dt>Date</dt> <dd>2015-04-23 18:16:37 -0700 (Thu, 23 Apr 2015)</dd>
</dl>
<h3>Log Message</h3>
<pre>Perf dashboard should automatically detect regressions
https://bugs.webkit.org/show_bug.cgi?id=141443
Reviewed by Anders Carlsson.
Added a node.js script detect-changes.js to detect potential regressions and progressions
on the graphs tracked on v2 dashboards.
* init-database.sql: Added analysis_strategies table and task_segmentation and task_test_range
columns to analysis_tasks to keep the segmentation and test range selection strategies used
to create an analysis task.
* public/api/analysis-tasks.php:
(format_task): Include task_segmentation and analysis_tasks in the results.
* public/include/json-header.php:
(remote_user_name): Returns null when the privileged API is authenticated as a slave instead
of a CSRF prevention token.
(should_authenticate_as_slave): Added.
(ensure_privileged_api_data_and_token_or_slave): Added. Authenticate as a slave if slaveName
and slavePassword are specified. Since detect-changes.js and other slaves are not susceptible
to a CSRF attack, we don't need to check a CSRF token.
* public/privileged-api/create-analysis-task.php:
(main): Use ensure_privileged_api_data_and_token_or_slave to let detect-changes.js create new
analysis task. Also add or find segmentation and test range selection strategies if specified.
* public/privileged-api/create-test-group.php:
(main): Use ensure_privileged_api_data_and_token_or_slave.
* public/privileged-api/generate-csrf-token.php:
* public/v2/app.js:
(App.Pane._computeMovingAverageAndOutliers): _executeStrategy has been moved to Statistics.
* public/v2/data.js: Export Measurement, RunsData, TimeSeries. Used in detect-changes.js.
(Array.prototype.find): Added a polyfill to be used in node.js.
(RunsData.fetchRuns):
(RunsData.pathForFetchingRuns): Extracted from fetchRuns. Used in detect-changes.js.
(RunsData.createRunsDataInResponse): Extracted from App.Manifest._formatFetchedData to use it
in detect-changes.js.
(RunsData.unitFromMetricName): Ditto.
(RunsData.isSmallerBetter): Ditto.
(RunsData.prototype._timeSeriesByTimeInternal): Added secondaryTime to sort points when commit
times are identical.
(TimeSeries): When commit times are identical, order points based on build time. This is needed
for when we trigger two builds at two different OS versions with the same WebKit revision since
OS versions don't change the commit times.
(TimeSeries.prototype.findPointByIndex): Added.
(TimeSeries.prototype.rawValues): Added.
* public/v2/js/statistics.js:
(Statistics.TestRangeSelectionStrategies.[0]): Use the 99% two-sided probability as claimed in the
description of this strategy instead of the default probability. Also fixed a bug that debugging
code was referring to non-existent variables.
(Statistics.executeStrategy): Moved from App.Pane (app.js).
* public/v2/manifest.js:
(App.Manifest._formatFetchedData): Various code has been extracted into RunsData in data.js to be
used in detect-changes.js.
* tools/detect-changes.js: Added. The script fetches the manifest JSON, analyzes each graph in
the v2 dashboards, and creates an analysis task for the latest regression or progression detected.
It also schedules an A/B testing if possible and notifies another server; e.g. to send an email.
(main): Loads the settings JSON specified in the argument.
(fetchManifestAndAnalyzeData): The main loop that periodically wakes up to do the analysis.
(mapInOrder): Executes callback sequentially (i.e. blocking) on each item in the array.
(configurationsForTesting): Finds every (platform, metric) pair to analyze in the v2 dashbaords,
and computes various values for when statistically significant changes are detected later.
(analyzeConfiguration): Finds potential regressions and progression in the last X days where X
is the specified maximum number of days using the specified strategies. Sort the resultant ranges
in chronological order and create a new analysis task for the very last change we detected. We'll
eventually create an analysis task for all detected changes since we're repeating the analysis in
fetchManifestAndAnalyzeData after some time.
(computeRangesForTesting): Fetch measured values and compute ranges to test using the specified
segmentation and test range selection strategies. Once ranges are found, find overlapping analysis
tasks as they need to be filtered out in analyzeConfiguration to avoid creating multiple analysis
tasks for the same range (e.g. humans may create one before the script gets to do it).
(createAnalysisTaskAndNotify): Create a new analysis task for the specified range, trigger an A/B
testing if available, and notify another server with a HTML message as specified.
(findStrategyByLabel):
(changeTypeForRange): A change is a regression if values are getting larger in a smaller-is-better
test or values are getting smaller in a larger-is-better test and vice versa.
(summarizeRange): Create a human readable string that summarizes the change detected. e.g.
"Potential 3.2% regression detected between 2015-04-20 12:00 and 17:00".
(formatTimeRange):
(getJSON):
(postJSON):
(postNotification): Recursively replaces $title and $massage in the specified JSON template.
(instantiateNotificationTemplate):
(fetchJSON):</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkWebsitesperfwebkitorgChangeLog">trunk/Websites/perf.webkit.org/ChangeLog</a></li>
<li><a href="#trunkWebsitesperfwebkitorginitdatabasesql">trunk/Websites/perf.webkit.org/init-database.sql</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicapianalysistasksphp">trunk/Websites/perf.webkit.org/public/api/analysis-tasks.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicincludejsonheaderphp">trunk/Websites/perf.webkit.org/public/include/json-header.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicprivilegedapicreateanalysistaskphp">trunk/Websites/perf.webkit.org/public/privileged-api/create-analysis-task.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicprivilegedapicreatetestgroupphp">trunk/Websites/perf.webkit.org/public/privileged-api/create-test-group.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicprivilegedapigeneratecsrftokenphp">trunk/Websites/perf.webkit.org/public/privileged-api/generate-csrf-token.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicv2appjs">trunk/Websites/perf.webkit.org/public/v2/app.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicv2datajs">trunk/Websites/perf.webkit.org/public/v2/data.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicv2jsstatisticsjs">trunk/Websites/perf.webkit.org/public/v2/js/statistics.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicv2manifestjs">trunk/Websites/perf.webkit.org/public/v2/manifest.js</a></li>
</ul>
<h3>Added Paths</h3>
<ul>
<li><a href="#trunkWebsitesperfwebkitorgtoolsdetectchangesjs">trunk/Websites/perf.webkit.org/tools/detect-changes.js</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkWebsitesperfwebkitorgChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/ChangeLog (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/ChangeLog 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/ChangeLog 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -1,3 +1,97 @@
</span><ins>+2015-04-23 Ryosuke Niwa <rniwa@webkit.org>
+
+ Perf dashboard should automatically detect regressions
+ https://bugs.webkit.org/show_bug.cgi?id=141443
+
+ Reviewed by Anders Carlsson.
+
+ Added a node.js script detect-changes.js to detect potential regressions and progressions
+ on the graphs tracked on v2 dashboards.
+
+ * init-database.sql: Added analysis_strategies table and task_segmentation and task_test_range
+ columns to analysis_tasks to keep the segmentation and test range selection strategies used
+ to create an analysis task.
+
+ * public/api/analysis-tasks.php:
+ (format_task): Include task_segmentation and analysis_tasks in the results.
+
+ * public/include/json-header.php:
+ (remote_user_name): Returns null when the privileged API is authenticated as a slave instead
+ of a CSRF prevention token.
+ (should_authenticate_as_slave): Added.
+ (ensure_privileged_api_data_and_token_or_slave): Added. Authenticate as a slave if slaveName
+ and slavePassword are specified. Since detect-changes.js and other slaves are not susceptible
+ to a CSRF attack, we don't need to check a CSRF token.
+
+ * public/privileged-api/create-analysis-task.php:
+ (main): Use ensure_privileged_api_data_and_token_or_slave to let detect-changes.js create new
+ analysis task. Also add or find segmentation and test range selection strategies if specified.
+
+ * public/privileged-api/create-test-group.php:
+ (main): Use ensure_privileged_api_data_and_token_or_slave.
+
+ * public/privileged-api/generate-csrf-token.php:
+
+ * public/v2/app.js:
+ (App.Pane._computeMovingAverageAndOutliers): _executeStrategy has been moved to Statistics.
+
+ * public/v2/data.js: Export Measurement, RunsData, TimeSeries. Used in detect-changes.js.
+ (Array.prototype.find): Added a polyfill to be used in node.js.
+ (RunsData.fetchRuns):
+ (RunsData.pathForFetchingRuns): Extracted from fetchRuns. Used in detect-changes.js.
+ (RunsData.createRunsDataInResponse): Extracted from App.Manifest._formatFetchedData to use it
+ in detect-changes.js.
+ (RunsData.unitFromMetricName): Ditto.
+ (RunsData.isSmallerBetter): Ditto.
+ (RunsData.prototype._timeSeriesByTimeInternal): Added secondaryTime to sort points when commit
+ times are identical.
+ (TimeSeries): When commit times are identical, order points based on build time. This is needed
+ for when we trigger two builds at two different OS versions with the same WebKit revision since
+ OS versions don't change the commit times.
+ (TimeSeries.prototype.findPointByIndex): Added.
+ (TimeSeries.prototype.rawValues): Added.
+
+ * public/v2/js/statistics.js:
+ (Statistics.TestRangeSelectionStrategies.[0]): Use the 99% two-sided probability as claimed in the
+ description of this strategy instead of the default probability. Also fixed a bug that debugging
+ code was referring to non-existent variables.
+ (Statistics.executeStrategy): Moved from App.Pane (app.js).
+
+ * public/v2/manifest.js:
+ (App.Manifest._formatFetchedData): Various code has been extracted into RunsData in data.js to be
+ used in detect-changes.js.
+
+ * tools/detect-changes.js: Added. The script fetches the manifest JSON, analyzes each graph in
+ the v2 dashboards, and creates an analysis task for the latest regression or progression detected.
+ It also schedules an A/B testing if possible and notifies another server; e.g. to send an email.
+ (main): Loads the settings JSON specified in the argument.
+ (fetchManifestAndAnalyzeData): The main loop that periodically wakes up to do the analysis.
+ (mapInOrder): Executes callback sequentially (i.e. blocking) on each item in the array.
+ (configurationsForTesting): Finds every (platform, metric) pair to analyze in the v2 dashbaords,
+ and computes various values for when statistically significant changes are detected later.
+ (analyzeConfiguration): Finds potential regressions and progression in the last X days where X
+ is the specified maximum number of days using the specified strategies. Sort the resultant ranges
+ in chronological order and create a new analysis task for the very last change we detected. We'll
+ eventually create an analysis task for all detected changes since we're repeating the analysis in
+ fetchManifestAndAnalyzeData after some time.
+ (computeRangesForTesting): Fetch measured values and compute ranges to test using the specified
+ segmentation and test range selection strategies. Once ranges are found, find overlapping analysis
+ tasks as they need to be filtered out in analyzeConfiguration to avoid creating multiple analysis
+ tasks for the same range (e.g. humans may create one before the script gets to do it).
+ (createAnalysisTaskAndNotify): Create a new analysis task for the specified range, trigger an A/B
+ testing if available, and notify another server with a HTML message as specified.
+ (findStrategyByLabel):
+ (changeTypeForRange): A change is a regression if values are getting larger in a smaller-is-better
+ test or values are getting smaller in a larger-is-better test and vice versa.
+ (summarizeRange): Create a human readable string that summarizes the change detected. e.g.
+ "Potential 3.2% regression detected between 2015-04-20 12:00 and 17:00".
+ (formatTimeRange):
+ (getJSON):
+ (postJSON):
+ (postNotification): Recursively replaces $title and $massage in the specified JSON template.
+ (instantiateNotificationTemplate):
+ (fetchJSON):
+
</ins><span class="cx"> 2015-04-20 Ryosuke Niwa <rniwa@webkit.org>
</span><span class="cx">
</span><span class="cx"> Perf dashboard should have UI to set status on analysis tasks
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorginitdatabasesql"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/init-database.sql (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/init-database.sql 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/init-database.sql 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -174,11 +174,17 @@
</span><span class="cx"> report_failure varchar(64),
</span><span class="cx"> report_failure_details text);
</span><span class="cx">
</span><ins>+CREATE TABLE analysis_strategies (
+ strategy_id serial PRIMARY KEY,
+ strategy_name varchar(64) NOT NULL);
+
</ins><span class="cx"> CREATE TYPE analysis_task_result_type as ENUM ('progression', 'regression', 'unchanged', 'inconclusive');
</span><span class="cx"> CREATE TABLE analysis_tasks (
</span><span class="cx"> task_id serial PRIMARY KEY,
</span><span class="cx"> task_name varchar(256) NOT NULL,
</span><span class="cx"> task_author varchar(256),
</span><ins>+ task_segmentation integer REFERENCES analysis_strategies,
+ task_test_range integer REFERENCES analysis_strategies,
</ins><span class="cx"> task_created_at timestamp NOT NULL DEFAULT (CURRENT_TIMESTAMP AT TIME ZONE 'UTC'),
</span><span class="cx"> task_platform integer REFERENCES platforms NOT NULL,
</span><span class="cx"> task_metric integer REFERENCES test_metrics NOT NULL,
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicapianalysistasksphp"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/api/analysis-tasks.php (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/api/analysis-tasks.php 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/api/analysis-tasks.php 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -81,6 +81,8 @@
</span><span class="cx"> 'id' => $task_row['task_id'],
</span><span class="cx"> 'name' => $task_row['task_name'],
</span><span class="cx"> 'author' => $task_row['task_author'],
</span><ins>+ 'segmentationStrategy' => $task_row['task_segmentation'],
+ 'testRangeStragegy' => $task_row['task_test_range'],
</ins><span class="cx"> 'createdAt' => strtotime($task_row['task_created_at']) * 1000,
</span><span class="cx"> 'platform' => $task_row['task_platform'],
</span><span class="cx"> 'metric' => $task_row['task_metric'],
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicincludejsonheaderphp"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/include/json-header.php (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/include/json-header.php 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/include/json-header.php 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -96,10 +96,23 @@
</span><span class="cx"> return $data;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-function remote_user_name() {
- return array_get($_SERVER, 'REMOTE_USER');
</del><ins>+function remote_user_name($data) {
+ return should_authenticate_as_slave($data) ? NULL : array_get($_SERVER, 'REMOTE_USER');
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+function should_authenticate_as_slave($data) {
+ return array_key_exists('slaveName', $data) && array_key_exists('slavePassword', $data);
+}
+
+function ensure_privileged_api_data_and_token_or_slave($db) {
+ $data = ensure_privileged_api_data();
+ if (should_authenticate_as_slave($data))
+ verify_slave($db, $data);
+ else if (!verify_token(array_get($data, 'token')))
+ exit_with_error('InvalidToken');
+ return $data;
+}
+
</ins><span class="cx"> function compute_token() {
</span><span class="cx"> if (!array_key_exists('CSRFSalt', $_COOKIE) || !array_key_exists('CSRFExpiration', $_COOKIE))
</span><span class="cx"> return NULL;
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicprivilegedapicreateanalysistaskphp"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/privileged-api/create-analysis-task.php (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/privileged-api/create-analysis-task.php 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/privileged-api/create-analysis-task.php 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -3,26 +3,48 @@
</span><span class="cx"> require_once('../include/json-header.php');
</span><span class="cx">
</span><span class="cx"> function main() {
</span><del>- $data = ensure_privileged_api_data_and_token();
</del><ins>+ $db = connect();
+ $data = ensure_privileged_api_data_and_token_or_slave($db);
</ins><span class="cx">
</span><del>- $author = remote_user_name();
</del><ins>+ $author = remote_user_name($data);
</ins><span class="cx"> $name = array_get($data, 'name');
</span><span class="cx"> $start_run_id = array_get($data, 'startRun');
</span><span class="cx"> $end_run_id = array_get($data, 'endRun');
</span><span class="cx">
</span><ins>+ $segmentation_name = array_get($data, 'segmentationStrategy');
+ $test_range_name = array_get($data, 'testRangeStrategy');
+
</ins><span class="cx"> if (!$name)
</span><span class="cx"> exit_with_error('MissingName', array('name' => $name));
</span><span class="cx"> $range = array('startRunId' => $start_run_id, 'endRunId' => $end_run_id);
</span><span class="cx"> if (!$start_run_id || !$end_run_id)
</span><span class="cx"> exit_with_error('MissingRange', $range);
</span><span class="cx">
</span><del>- $db = connect();
</del><span class="cx"> $start_run = ensure_row_by_id($db, 'test_runs', 'run', $start_run_id, 'InvalidStartRun', $range);
</span><span class="cx"> $end_run = ensure_row_by_id($db, 'test_runs', 'run', $end_run_id, 'InvalidEndRun', $range);
</span><span class="cx">
</span><span class="cx"> $config = ensure_config_from_runs($db, $start_run, $end_run);
</span><span class="cx">
</span><span class="cx"> $db->begin_transaction();
</span><ins>+
+ $segmentation_id = NULL;
+ if ($segmentation_name) {
+ $segmentation_id = $db->select_or_insert_row('analysis_strategies', 'strategy', array('name' => $segmentation_name));
+ if (!$segmentation_id) {
+ $db->rollback_transaction();
+ exit_with_error('CannotFindOrInsertSegmentationStrategy', array('segmentationStrategy' => $segmentation_name));
+ }
+ }
+
+ $test_range_id = NULL;
+ if ($test_range_name) {
+ $test_range_id = $db->select_or_insert_row('analysis_strategies', 'strategy', array('name' => $test_range_name));
+ if (!$test_range_id) {
+ $db->rollback_transaction();
+ exit_with_error('CannotFindOrInsertTestRangeStrategy', array('testRangeStrategy' => $test_range_name));
+ }
+ }
+
</ins><span class="cx"> $duplicate = $db->select_first_row('analysis_tasks', 'task', array('start_run' => $start_run_id, 'end_run' => $end_run_id));
</span><span class="cx"> if ($duplicate) {
</span><span class="cx"> $db->rollback_transaction();
</span><span class="lines">@@ -35,7 +57,9 @@
</span><span class="cx"> 'platform' => $config['config_platform'],
</span><span class="cx"> 'metric' => $config['config_metric'],
</span><span class="cx"> 'start_run' => $start_run_id,
</span><del>- 'end_run' => $end_run_id));
</del><ins>+ 'end_run' => $end_run_id,
+ 'segmentation' => $segmentation_id,
+ 'test_range' => $test_range_id));
</ins><span class="cx"> $db->commit_transaction();
</span><span class="cx">
</span><span class="cx"> exit_with_success(array('taskId' => $task_id));
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicprivilegedapicreatetestgroupphp"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/privileged-api/create-test-group.php (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/privileged-api/create-test-group.php 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/privileged-api/create-test-group.php 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -3,10 +3,10 @@
</span><span class="cx"> require_once('../include/json-header.php');
</span><span class="cx">
</span><span class="cx"> function main() {
</span><del>- $data = ensure_privileged_api_data_and_token();
</del><ins>+ $db = connect();
+ $data = ensure_privileged_api_data_and_token_or_slave($db);
+ $author = remote_user_name($data);
</ins><span class="cx">
</span><del>- $author = remote_user_name();
-
</del><span class="cx"> $task_id = array_get($data, 'task');
</span><span class="cx"> $name = array_get($data, 'name');
</span><span class="cx"> $root_sets = array_get($data, 'rootSets');
</span><span class="lines">@@ -19,7 +19,6 @@
</span><span class="cx"> if ($repetition_count < 1)
</span><span class="cx"> exit_with_error('InvalidRepetitionCount', array('repetitionCount' => $repetition_count));
</span><span class="cx">
</span><del>- $db = connect();
</del><span class="cx"> $task = $db->select_first_row('analysis_tasks', 'task', array('id' => $task_id));
</span><span class="cx"> if (!$task)
</span><span class="cx"> exit_with_error('InvalidTask', array('task' => $task_id));
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicprivilegedapigeneratecsrftokenphp"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/privileged-api/generate-csrf-token.php (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/privileged-api/generate-csrf-token.php 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/privileged-api/generate-csrf-token.php 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -2,7 +2,7 @@
</span><span class="cx">
</span><span class="cx"> require_once('../include/json-header.php');
</span><span class="cx">
</span><del>-ensure_privileged_api_data();
</del><ins>+$data = ensure_privileged_api_data();
</ins><span class="cx">
</span><span class="cx"> $expiritaion = time() + 3600; // Valid for one hour.
</span><span class="cx"> $_COOKIE['CSRFSalt'] = rand();
</span><span class="lines">@@ -11,6 +11,6 @@
</span><span class="cx"> setcookie('CSRFSalt', $_COOKIE['CSRFSalt']);
</span><span class="cx"> setcookie('CSRFExpiration', $expiritaion);
</span><span class="cx">
</span><del>-exit_with_success(array('user' => remote_user_name(), 'token' => compute_token(), 'expiration' => $expiritaion * 1000));
</del><ins>+exit_with_success(array('user' => remote_user_name($data), 'token' => compute_token(), 'expiration' => $expiritaion * 1000));
</ins><span class="cx">
</span><span class="cx"> ?>
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicv2appjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/v2/app.js (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/v2/app.js 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/v2/app.js 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -583,19 +583,21 @@
</span><span class="cx"> _computeMovingAverageAndOutliers: function (chartData, movingAverageStrategy, envelopingStrategy, testRangeSelectionStrategy, anomalyDetectionStrategies)
</span><span class="cx"> {
</span><span class="cx"> var currentTimeSeriesData = chartData.current.series();
</span><ins>+
+ var rawValues = chartData.current.rawValues();
</ins><span class="cx"> var movingAverageIsSetByUser = movingAverageStrategy && movingAverageStrategy.execute;
</span><del>- var movingAverageValues = this._executeStrategy(
- movingAverageIsSetByUser ? movingAverageStrategy : Statistics.MovingAverageStrategies[0], currentTimeSeriesData);
</del><ins>+ var movingAverageValues = Statistics.executeStrategy(
+ movingAverageIsSetByUser ? movingAverageStrategy : Statistics.MovingAverageStrategies[0], rawValues);
</ins><span class="cx"> if (!movingAverageValues)
</span><span class="cx"> return null;
</span><span class="cx">
</span><span class="cx"> var testRangeCandidates = [];
</span><span class="cx"> if (movingAverageStrategy && movingAverageStrategy.isSegmentation && testRangeSelectionStrategy && testRangeSelectionStrategy.execute)
</span><del>- testRangeCandidates = this._executeStrategy(testRangeSelectionStrategy, currentTimeSeriesData, [movingAverageValues]);
</del><ins>+ testRangeCandidates = Statistics.executeStrategy(testRangeSelectionStrategy, rawValues, [movingAverageValues]);
</ins><span class="cx">
</span><span class="cx"> var envelopeIsSetByUser = envelopingStrategy && envelopingStrategy.execute;
</span><del>- var envelopeDelta = this._executeStrategy(envelopeIsSetByUser ? envelopingStrategy : Statistics.EnvelopingStrategies[0],
- currentTimeSeriesData, [movingAverageValues]);
</del><ins>+ var envelopeDelta = Statistics.executeStrategy(envelopeIsSetByUser ? envelopingStrategy : Statistics.EnvelopingStrategies[0],
+ rawValues, [movingAverageValues]);
</ins><span class="cx">
</span><span class="cx"> for (var i = 0; i < currentTimeSeriesData.length; i++) {
</span><span class="cx"> var currentValue = currentTimeSeriesData[i].value;
</span><span class="lines">@@ -610,7 +612,7 @@
</span><span class="cx"> if (anomalyDetectionStrategies.length) {
</span><span class="cx"> var isAnomalyArray = new Array(currentTimeSeriesData.length);
</span><span class="cx"> for (var strategy of anomalyDetectionStrategies) {
</span><del>- var anomalyLengths = this._executeStrategy(strategy, currentTimeSeriesData, [movingAverageValues, envelopeDelta]);
</del><ins>+ var anomalyLengths = Statistics.executeStrategy(strategy, rawValues, [movingAverageValues, envelopeDelta]);
</ins><span class="cx"> for (var i = 0; i < currentTimeSeriesData.length; i++)
</span><span class="cx"> isAnomalyArray[i] = isAnomalyArray[i] || anomalyLengths[i];
</span><span class="cx"> }
</span><span class="lines">@@ -642,15 +644,6 @@
</span><span class="cx"> testRangeCandidates: testRangeCandidates,
</span><span class="cx"> };
</span><span class="cx"> },
</span><del>- _executeStrategy: function (strategy, currentTimeSeriesData, additionalArguments)
- {
- var parameters = (strategy.parameterList || []).map(function (param) {
- var parsed = parseFloat(param.value);
- return Math.min(param.max || Infinity, Math.max(param.min || -Infinity, isNaN(parsed) ? 0 : parsed));
- });
- parameters.push(currentTimeSeriesData.map(function (point) { return point.value }));
- return strategy.execute.apply(window, parameters.concat(additionalArguments));
- },
</del><span class="cx"> _updateStrategyConfigIfNeeded: function (strategy, configName)
</span><span class="cx"> {
</span><span class="cx"> var config = null;
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicv2datajs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/v2/data.js (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/v2/data.js 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/v2/data.js 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -1,5 +1,15 @@
</span><span class="cx"> // We don't use DS.Model for these object types because we can't afford to process millions of them.
</span><span class="cx">
</span><ins>+if (!Array.prototype.find) {
+ Array.prototype.find = function (callback) {
+ for (var item of this) {
+ if (callback(item))
+ return item;
+ }
+ return undefined;
+ }
+}
+
</ins><span class="cx"> var PrivilegedAPI = {
</span><span class="cx"> _token: null,
</span><span class="cx"> _expiration: null,
</span><span class="lines">@@ -333,6 +343,7 @@
</span><span class="cx"> series.push({
</span><span class="cx"> measurement: measurement,
</span><span class="cx"> time: useCommitType ? measurement.latestCommitTime() : measurement.buildTime(),
</span><ins>+ secondaryTime: measurement.buildTime(),
</ins><span class="cx"> value: measurement.mean(),
</span><span class="cx"> interval: measurement.confidenceInterval(),
</span><span class="cx"> markedOutlier: measurement.markedOutlier(),
</span><span class="lines">@@ -345,28 +356,14 @@
</span><span class="cx"> // we don't have to fetch the entire time series to just show the last 3 days.
</span><span class="cx"> RunsData.fetchRuns = function (platformId, metricId, testGroupId, useCache)
</span><span class="cx"> {
</span><del>- var url = useCache ? '../data/' : '../api/runs/';
-
- url += platformId + '-' + metricId + '.json';
- if (testGroupId)
- url += '?testGroup=' + testGroupId;
-
</del><ins>+ var url = this.pathForFetchingRuns(platformId, metricId, testGroupId, useCache);
</ins><span class="cx"> return new Ember.RSVP.Promise(function (resolve, reject) {
</span><span class="cx"> $.getJSON(url, function (response) {
</span><span class="cx"> if (response.status != 'OK') {
</span><span class="cx"> reject(response.status);
</span><span class="cx"> return;
</span><span class="cx"> }
</span><del>- delete response.status;
-
- var data = response.configurations;
- for (var config in data)
- data[config] = new RunsData(data[config]);
-
- if (response.lastModified)
- response.lastModified = new Date(response.lastModified);
-
- resolve(response);
</del><ins>+ resolve(RunsData.createRunsDataInResponse(response));
</ins><span class="cx"> }).fail(function (xhr, status, error) {
</span><span class="cx"> if (xhr.status == 404 && useCache)
</span><span class="cx"> resolve(null);
</span><span class="lines">@@ -376,9 +373,58 @@
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+RunsData.pathForFetchingRuns = function (platformId, metricId, testGroupId, useCache)
+{
+ var path = useCache ? '/data/' : '/api/runs/';
+
+ path += platformId + '-' + metricId + '.json';
+ if (testGroupId)
+ path += '?testGroup=' + testGroupId;
+
+ return path;
+}
+
+RunsData.createRunsDataInResponse = function (response)
+{
+ delete response.status;
+
+ var data = response.configurations;
+ for (var config in data)
+ data[config] = new RunsData(data[config]);
+
+ if (response.lastModified)
+ response.lastModified = new Date(response.lastModified);
+
+ return response;
+}
+
+// FIXME: It was a mistake to put this in the client side. We should put this back in the JSON.
+RunsData.unitFromMetricName = function (metricName)
+{
+ var suffix = metricName.match('([A-z][a-z]+|FrameRate)$')[0];
+ var unit = {
+ 'FrameRate': 'fps',
+ 'Runs': '/s',
+ 'Time': 'ms',
+ 'Malloc': 'bytes',
+ 'Heap': 'bytes',
+ 'Allocations': 'bytes'
+ }[suffix];
+ return unit;
+}
+
+RunsData.isSmallerBetter = function (unit)
+{
+ return unit != 'fps' && unit != '/s';
+}
+
</ins><span class="cx"> function TimeSeries(series)
</span><span class="cx"> {
</span><del>- this._series = series.sort(function (a, b) { return a.time - b.time; });
</del><ins>+ this._series = series.sort(function (a, b) {
+ var diff = a.time - b.time;
+ return diff ? diff : a.secondaryTime - b.secondaryTime;
+ });
+
</ins><span class="cx"> var self = this;
</span><span class="cx"> var min = undefined;
</span><span class="cx"> var max = undefined;
</span><span class="lines">@@ -394,6 +440,13 @@
</span><span class="cx"> this._max = max;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+TimeSeries.prototype.findPointByIndex = function (index)
+{
+ if (!this._series || index < 0 || index >= this._series.length)
+ return null;
+ return this._series[index];
+}
+
</ins><span class="cx"> TimeSeries.prototype.findPointByBuild = function (buildId)
</span><span class="cx"> {
</span><span class="cx"> return this._series.find(function (point) { return point.measurement.buildId() == buildId; })
</span><span class="lines">@@ -462,6 +515,11 @@
</span><span class="cx">
</span><span class="cx"> TimeSeries.prototype.series = function () { return this._series; }
</span><span class="cx">
</span><ins>+TimeSeries.prototype.rawValues = function ()
+{
+ return this._series.map(function (point) { return point.value });
+}
+
</ins><span class="cx"> TimeSeries.prototype.lastPoint = function ()
</span><span class="cx"> {
</span><span class="cx"> if (!this._series || !this._series.length)
</span><span class="lines">@@ -482,3 +540,10 @@
</span><span class="cx"> return null;
</span><span class="cx"> return this._series[point.seriesIndex + 1];
</span><span class="cx"> }
</span><ins>+
+if (typeof module != 'undefined') {
+ Statistics = require('./js/statistics.js');
+ module.exports.Measurement = Measurement;
+ module.exports.RunsData = RunsData;
+ module.exports.TimeSeries = TimeSeries;
+}
</ins></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicv2jsstatisticsjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/v2/js/statistics.js (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/v2/js/statistics.js 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/v2/js/statistics.js 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -487,7 +487,7 @@
</span><span class="cx"> for (var leftEdge = i - 2, rightEdge = i + 2; leftEdge >= 0 && rightEdge <= values.length; leftEdge--, rightEdge++) {
</span><span class="cx"> if (segmentedValues[leftEdge] != previousMean || segmentedValues[rightEdge - 1] != currentMean)
</span><span class="cx"> break;
</span><del>- var result = Statistics.computeWelchsT(values, leftEdge, i - leftEdge, values, i, rightEdge - i);
</del><ins>+ var result = Statistics.computeWelchsT(values, leftEdge, i - leftEdge, values, i, rightEdge - i, 0.98);
</ins><span class="cx"> if (result.significantlyDifferent) {
</span><span class="cx"> selectedRanges.push([leftEdge, rightEdge - 1]);
</span><span class="cx"> found = true;
</span><span class="lines">@@ -495,7 +495,7 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx"> if (!found && Statistics.debuggingTestingRangeNomination)
</span><del>- console.log('Failed to find a testing range at', i, 'changing from', previousValue, 'to', currentValue);
</del><ins>+ console.log('Failed to find a testing range at', i, 'changing from', previousMean, 'to', currentMean);
</ins><span class="cx"> previousMean = currentMean;
</span><span class="cx"> }
</span><span class="cx"> return selectedRanges;
</span><span class="lines">@@ -565,6 +565,16 @@
</span><span class="cx"> },
</span><span class="cx"> ]
</span><span class="cx">
</span><ins>+ this.executeStrategy = function (strategy, rawValues, additionalArguments)
+ {
+ var parameters = (strategy.parameterList || []).map(function (param) {
+ var parsed = parseFloat(param.value);
+ return Math.min(param.max || Infinity, Math.max(param.min || -Infinity, isNaN(parsed) ? 0 : parsed));
+ });
+ parameters.push(rawValues);
+ return strategy.execute.apply(strategy, parameters.concat(additionalArguments));
+ };
+
</ins><span class="cx"> })();
</span><span class="cx">
</span><span class="cx"> if (typeof module != 'undefined') {
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicv2manifestjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/v2/manifest.js (183231 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/v2/manifest.js 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/v2/manifest.js 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -317,18 +317,9 @@
</span><span class="cx"> },
</span><span class="cx"> _formatFetchedData: function (metricName, configurations)
</span><span class="cx"> {
</span><del>- var suffix = metricName.match('([A-z][a-z]+|FrameRate)$')[0];
- var unit = {
- 'FrameRate': 'fps',
- 'Runs': '/s',
- 'Time': 'ms',
- 'Malloc': 'bytes',
- 'Heap': 'bytes',
- 'Allocations': 'bytes'
- }[suffix];
</del><ins>+ var unit = RunsData.unitFromMetricName(metricName);
+ var smallerIsBetter = RunsData.isSmallerBetter(unit);
</ins><span class="cx">
</span><del>- var smallerIsBetter = unit != 'fps' && unit != '/s'; // Assume smaller is better for unit-less metrics.
-
</del><span class="cx"> var useSI = unit == 'bytes';
</span><span class="cx"> var unitSuffix = unit ? ' ' + unit : '';
</span><span class="cx"> var deltaFormatterWithoutSign = useSI ? d3.format('.2s') : d3.format('.2g');
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtoolsdetectchangesjs"></a>
<div class="addfile"><h4>Added: trunk/Websites/perf.webkit.org/tools/detect-changes.js (0 => 183232)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tools/detect-changes.js (rev 0)
+++ trunk/Websites/perf.webkit.org/tools/detect-changes.js 2015-04-24 01:16:37 UTC (rev 183232)
</span><span class="lines">@@ -0,0 +1,374 @@
</span><ins>+#!/usr/local/bin/node
+
+var fs = require('fs');
+var http = require('http');
+var https = require('https');
+var data = require('../public/v2/data.js');
+var RunsData = data.RunsData;
+var Statistics = require('../public/v2/js/statistics.js');
+
+var settings;
+function main(argv)
+{
+ if (argv.length < 3) {
+ console.error('Please specify the settings JSON path');
+ return 1;
+ }
+
+ settings = JSON.parse(fs.readFileSync(argv[2], 'utf8'));
+
+ fetchManifestAndAnalyzeData();
+}
+
+function fetchManifestAndAnalyzeData()
+{
+ getJSON(settings.perfserver, '/data/manifest.json').then(function (manifest) {
+ return mapInOrder(configurationsForTesting(manifest), analyzeConfiguration);
+ }).catch(function (reason) {
+ console.error('Failed to obtain the manifest file');
+ }).then(function () {
+ console.log('');
+ console.log('Sleeing for', settings.secondsToSleep, 'seconds');
+ setTimeout(fetchManifestAndAnalyzeData, settings.secondsToSleep * 1000);
+ });
+}
+
+function mapInOrder(array, callback, startIndex)
+{
+ if (startIndex === undefined)
+ startIndex = 0;
+ if (startIndex >= array.length)
+ return;
+
+ var next = function () { return mapInOrder(array, callback, startIndex + 1); };
+ var returnValue = callback(array[startIndex]);
+ if (typeof(returnValue) === 'object' && returnValue instanceof Promise)
+ return returnValue.then(next).catch(next);
+ return next();
+}
+
+function configurationsForTesting(manifest)
+{
+ var configurations = [];
+ for (var name in manifest.dashboards) {
+ var dashboard = manifest.dashboards[name];
+ for (var row of dashboard) {
+ for (var cell of row) {
+ if (cell instanceof Array)
+ configurations.push({platformId: parseInt(cell[0]), metricId: parseInt(cell[1])});
+ }
+ }
+ }
+
+ var platforms = manifest.all;
+ for (var config of configurations) {
+ var metric = manifest.metrics[config.metricId];
+
+ var testPath = [];
+ var id = metric.test;
+ while (id) {
+ var test = manifest.tests[id];
+ testPath.push(test.name);
+ id = test.parentId;
+ }
+
+ config.unit = RunsData.unitFromMetricName(metric.name);
+ config.smallerIsBetter = RunsData.isSmallerBetter(config.unit);
+ config.platformName = platforms[config.platformId].name;
+ config.testName = testPath.reverse().join(' > ');
+ config.fullTestName = config.testName + ':' + metric.name;
+ config.repositories = manifest.repositories;
+ if (metric.aggregator)
+ config.fullTestName += ':' + metric.aggregator;
+ }
+
+ return configurations;
+}
+
+function analyzeConfiguration(config)
+{
+ var minTime = Date.now() - settings.maxDays * 24 * 3600 * 1000;
+
+ console.log('');
+ console.log('== Analyzing the last', settings.maxDays, 'days:', config.fullTestName, 'on', config.platformName, '==');
+
+ return computeRangesForTesting(settings.perfserver, settings.strategies, config.platformId, config.metricId).then(function (ranges) {
+ var filteredRanges = ranges.filter(function (range) { return range.endTime >= minTime && !range.overlappingAnalysisTasks.length; })
+ .sort(function (a, b) { return a.endTime - b.endTime });
+
+ var summary;
+ var range;
+ for (range of filteredRanges) {
+ var summary = summarizeRange(config, range);
+ console.log('Detected:', summary);
+ }
+
+ if (!range) {
+ console.log('Nothing to analyze');
+ return;
+ }
+
+ return createAnalysisTaskAndNotify(config, range, summary);
+ });
+}
+
+function computeRangesForTesting(server, strategies, platformId, metricId)
+{
+ // FIXME: Store the segmentation strategy on the server side.
+ // FIXME: Configure each strategy.
+ var segmentationStrategy = findStrategyByLabel(Statistics.MovingAverageStrategies, strategies.segmentation.label);
+ if (!segmentationStrategy) {
+ console.error('Failed to find the segmentation strategy: ' + strategies.segmentation.label);
+ return;
+ }
+
+ var testRangeStrategy = findStrategyByLabel(Statistics.TestRangeSelectionStrategies, strategies.testRange.label);
+ if (!testRangeStrategy) {
+ console.error('Failed to find the test range selection strategy: ' + strategies.testRange.label);
+ return;
+ }
+
+ var currentPromise = getJSON(server, RunsData.pathForFetchingRuns(platformId, metricId)).then(function (response) {
+ if (response.status != 'OK')
+ throw response;
+ return RunsData.createRunsDataInResponse(response).configurations.current;
+ }, function (reason) {
+ console.error('Failed to fetch the measurements:', reason);
+ });
+
+ var analysisTasksPromise = getJSON(server, '/api/analysis-tasks?platform=' + platformId + '&metric=' + metricId).then(function (response) {
+ if (response.status != 'OK')
+ throw response;
+ return response.analysisTasks.filter(function (task) { return task.startRun && task.endRun; });
+ }, function (reason) {
+ console.error('Failed to fetch the analysis tasks:', reason);
+ });
+
+ return Promise.all([currentPromise, analysisTasksPromise]).then(function (results) {
+ var currentTimeSeries = results[0].timeSeriesByCommitTime();
+ var analysisTasks = results[1];
+ var rawValues = currentTimeSeries.rawValues();
+ var segmentedValues = Statistics.executeStrategy(segmentationStrategy, rawValues);
+
+ var ranges = Statistics.executeStrategy(testRangeStrategy, rawValues, [segmentedValues]).map(function (range) {
+ var startPoint = currentTimeSeries.findPointByIndex(range[0]);
+ var endPoint = currentTimeSeries.findPointByIndex(range[1]);
+ return {
+ startIndex: range[0],
+ endIndex: range[1],
+ overlappingAnalysisTasks: [],
+ startTime: startPoint.time,
+ endTime: endPoint.time,
+ relativeChangeInSegmentedValues: (segmentedValues[range[1]] - segmentedValues[range[0]]) / segmentedValues[range[0]],
+ startMeasurement: startPoint.measurement,
+ endMeasurement: endPoint.measurement,
+ };
+ });
+
+ for (var task of analysisTasks) {
+ var taskStartPoint = currentTimeSeries.findPointByMeasurementId(task.startRun);
+ var taskEndPoint = currentTimeSeries.findPointByMeasurementId(task.endRun);
+ for (var range of ranges) {
+ var disjoint = range.endIndex < taskStartPoint.seriesIndex
+ || taskEndPoint.seriesIndex < range.startIndex;
+ if (!disjoint)
+ range.overlappingAnalysisTasks.push(task);
+ }
+ }
+
+ return ranges;
+ });
+}
+
+function createAnalysisTaskAndNotify(config, range, summary)
+{
+ var segmentationStrategy = settings.strategies.segmentation.label;
+ var testRangeStrategy = settings.strategies.testRange.label;
+
+ var analysisTaskData = {
+ name: summary,
+ startRun: range.startMeasurement.id(),
+ endRun: range.endMeasurement.id(),
+ segmentationStrategy: segmentationStrategy,
+ testRangeStrategy: testRangeStrategy,
+
+ slaveName: settings.slave.name,
+ slavePassword: settings.slave.password,
+ };
+
+ return postJSON(settings.perfserver, '/privileged-api/create-analysis-task', analysisTaskData).then(function (response) {
+ if (response['status'] != 'OK')
+ throw response;
+
+ var analysisTaskId = response['taskId'];
+
+ var title = '[' + config.testName + '][' + config.platformName + '] ' + summary;
+ var analysisTaskURL = settings.perfserver.scheme + '://' + settings.perfserver.host + '/v2/#/analysis/task/' + analysisTaskId;
+ var changeType = changeTypeForRange(config, range);
+ // FIXME: Templatize this.
+ var message = '<b>' + settings.notification.serviceName + '</b> detected a potential ' + changeType + ':<br><br>'
+ + '<table border=1><caption>' + summary + '</caption><tbody>'
+ + '<tr><th>Test</th><td>' + config.fullTestName + '</td></tr>'
+ + '<tr><th>Platform</th><td>' + config.platformName + '</td></tr>'
+ + '<tr><th>Algorithm</th><td>' + segmentationStrategy + '<br>' + testRangeStrategy + '</td></tr>'
+ + '</table><br>'
+ + '<a href="' + analysisTaskURL + '">Open the analysis task</a>';
+
+ return getJSON(settings.perfserver, '/api/triggerables?task=' + analysisTaskId).then(function (response) {
+ var status = response['status'];
+ var triggerables = response['triggerables'] || [];
+ if (status == 'TriggerableNotFoundForTask' || triggerables.length != 1) {
+ message += ' (A/B testing was not available)';
+ return;
+ }
+ if (status != 'OK')
+ throw response;
+
+ var triggerable = response['triggerables'][0];
+ var rootSets = {};
+ for (var repositoryId of triggerable['acceptedRepositories']) {
+ var startRevision = range.startMeasurement.revisionForRepository(repositoryId);
+ var endRevision = range.endMeasurement.revisionForRepository(repositoryId);
+ if (startRevision == null || endRevision == null)
+ continue;
+ rootSets[config.repositories[repositoryId].name] = [startRevision, endRevision];
+ }
+
+ var testData = {
+ task: analysisTaskId,
+ name: 'Confirming the ' + changeType,
+ rootSets: rootSets,
+ repetitionCount: Math.max(2, Math.min(8, Math.floor((range.endIndex - range.startIndex) / 4))),
+
+ slaveName: settings.slave.name,
+ slavePassword: settings.slave.password,
+ };
+
+ return postJSON(settings.perfserver, '/privileged-api/create-test-group', testData).then(function (response) {
+ if (response['status'] != 'OK')
+ throw response;
+ message += ' (triggered an A/B testing)';
+ });
+ }).catch(function (reason) {
+ console.error(reason);
+ message += ' (failed to create a new A/B testing)';
+ }).then(function () {
+ return postNotification(settings.notification.server, settings.notification.template, title, message).then(function () {
+ console.log(' Sent a notification');
+ }, function (reason) {
+ console.error(' Failed to send a notification', reason);
+ });
+ });
+ }).catch(function (reason) {
+ console.error(' Failed to create an analysis task', reason);
+ });
+}
+
+function findStrategyByLabel(list, label)
+{
+ for (var strategy of list) {
+ if (strategy.label == label)
+ return strategy;
+ }
+ return null;
+}
+
+function changeTypeForRange(config, range)
+{
+ var endValueIsLarger = range.relativeChangeInSegmentedValues > 0;
+ return endValueIsLarger == config.smallerIsBetter ? 'regression' : 'progression';
+}
+
+function summarizeRange(config, range)
+{
+ return 'Potential ' + Math.abs(range.relativeChangeInSegmentedValues * 100).toPrecision(2) + '% '
+ + changeTypeForRange(config, range) + ' between ' + formatTimeRange(range.startTime, range.endTime);
+}
+
+function formatTimeRange(start, end)
+{
+ var formatter = function (date) { return date.toISOString().replace('T', ' ').replace(/:\d{2}\.\d+Z$/, ''); }
+ var formattedStart = formatter(start);
+ var formattedEnd = formatter(end);
+ if (start.toDateString() == end.toDateString())
+ return formattedStart + ' and ' + formattedEnd.substring(formattedEnd.indexOf(' ') + 1);
+ if (start.getFullYear() == end.getFullYear())
+ return formattedStart + ' and ' + formattedEnd.substring(5);
+ return formattedStart + ' and ' + formattedEnd;
+}
+
+function getJSON(server, path, data)
+{
+ return fetchJSON(server.scheme, {
+ 'hostname': server.host,
+ 'port': server.port,
+ 'auth': server.auth,
+ 'path': path,
+ 'method': 'GET',
+ }, 'application/json');
+}
+
+function postJSON(server, path, data)
+{
+ return fetchJSON(server.scheme, {
+ 'hostname': server.host,
+ 'port': server.port,
+ 'auth': server.auth,
+ 'path': path,
+ 'method': 'POST',
+ }, 'application/json', JSON.stringify(data));
+}
+
+function postNotification(server, template, title, message)
+{
+ var notification = instantiateNotificationTemplate(template, title, message);
+ return fetchJSON(server.scheme, {
+ 'hostname': server.host,
+ 'port': server.port,
+ 'auth': server.auth,
+ 'path': server.path,
+ 'method': server.method,
+ }, 'application/json', JSON.stringify(notification));
+}
+
+function instantiateNotificationTemplate(template, title, message)
+{
+ var instance = {};
+ for (var name in template) {
+ var value = template[name];
+ if (typeof(value) === 'string')
+ instance[name] = value.replace(/\$title/g, title).replace(/\$message/g, message);
+ else if (typeof(template[name]) === 'object')
+ instance[name] = instantiateNotificationTemplate(value, title, message);
+ else
+ instance[name] = value;
+ }
+ return instance;
+}
+
+function fetchJSON(schemeName, options, contentType, content) {
+ var requester = schemeName == 'https' ? https : http;
+ return new Promise(function (resolve, reject) {
+ var request = requester.request(options, function (response) {
+ var responseText = '';
+ response.setEncoding('utf8');
+ response.on('data', function (chunk) { responseText += chunk; });
+ response.on('end', function () {
+ try {
+ var json = JSON.parse(responseText);
+ } catch (error) {
+ reject({error: error, responseText: responseText});
+ }
+ resolve(json);
+ });
+ });
+ request.on('error', function (error) { reject(error); });
+ if (contentType)
+ request.setHeader('Content-Type', contentType);
+ if (content)
+ request.write(content);
+ request.end();
+ });
+}
+
+main(process.argv);
</ins></span></pre>
</div>
</div>
</body>
</html>