<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[214193] trunk/Websites/perf.webkit.org</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/214193">214193</a></dd>
<dt>Author</dt> <dd>rniwa@webkit.org</dd>
<dt>Date</dt> <dd>2017-03-20 15:36:40 -0700 (Mon, 20 Mar 2017)</dd>
</dl>
<h3>Log Message</h3>
<pre>Fix os-build-fetcher.js and subprocess.js to make them work
https://bugs.webkit.org/show_bug.cgi?id=169844
Reviewed by Antti Koivisto.
The script added in <a href="http://trac.webkit.org/projects/webkit/changeset/213976">r213976</a> has a bug that it can execute commands to fetch subcommits in parallel.
Some commands to poll the lsit of system components is not desirable to be ran in parallel.
* server-tests/resources/mock-subprocess.js:
(MockSubprocess): Use const declaration.
(MockSubprocess.resetAndWaitForInvocation): Added.
(MockSubprocess.waitForInvocation): Renamed from waitingForInvocation. A function name must be a verb.
See https://webkit.org/code-style-guidelines/#names-verb
(MockSubprocess.reset): Set invocations.length to 0 so that tests can store a reference to the array
regardless of whether reset is called or when it's called.
* server-tests/tools-os-build-fetcher-tests.js: Updated tests per the code change. Most of codes now
expect each command to be ran seprately. e.g. if there were two commands to run, instead of expecting
them to be both ran, and resolving invocation promises, we'd wait for one command to run, resolve,
its subcommand to run, and then move onto the second top-level command. Also use a local reference
to MockSubprocess.invocations instead of using the fully qualified name.
* tools/js/os-build-fetcher.js:
(mapInSerialPromiseChain): Added. Calling a closure that returns a promise on each item in an array
in serial (not asynchronous) is a very common pattern in this class.
(OSBuildFetcher.fetchAndReportAllInOrder): Added.
(OSBuildFetcher.prototype.fetchAndReportNewBuilds): Log what the number of builds being submitted.
(OSBuildFetcher.prototype._fetchAvailableBuilds): Fixed the main bug. Using Promise.all would result
in each top-level command to be execued in parallel. Since each subcommand is executed as soon as
its parent command is executed, this results in commands to be executed in parallel.
Added a whole bunch of logging so that we can at least detect a bug like this in the future.
(OSBuildFetcher.prototype._commitsForAvailableBuilds): Cleanup the coding style.
(OSBuildFetcher.prototype._addSubCommitsForBuild): Use mapInSerialPromiseChain. Tightened the assertion
about the content returned by a subcommand.
* tools/js/subprocess.js: Fixed the bug that we were importing require('child_process').ChildProcess.
execFile is defined on require('child_process') itself.
(Subprocess.prototype.execute): Fixed a typo. this._childProcess doesn't exist.
(Subprocess):
* tools/sync-os-versions.js: Renamed from tools/pull-os-versions.js.
(syncLoop): Cleaned up the coding style a little. Also added logging about how long we're about to sleep.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkWebsitesperfwebkitorgChangeLog">trunk/Websites/perf.webkit.org/ChangeLog</a></li>
<li><a href="#trunkWebsitesperfwebkitorgservertestsresourcesmocksubprocessjs">trunk/Websites/perf.webkit.org/server-tests/resources/mock-subprocess.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgserverteststoolsosbuildfetchertestsjs">trunk/Websites/perf.webkit.org/server-tests/tools-os-build-fetcher-tests.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgtoolsjsosbuildfetcherjs">trunk/Websites/perf.webkit.org/tools/js/os-build-fetcher.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgtoolsjssubprocessjs">trunk/Websites/perf.webkit.org/tools/js/subprocess.js</a></li>
</ul>
<h3>Added Paths</h3>
<ul>
<li><a href="#trunkWebsitesperfwebkitorgtoolssyncosversionsjs">trunk/Websites/perf.webkit.org/tools/sync-os-versions.js</a></li>
</ul>
<h3>Removed Paths</h3>
<ul>
<li><a href="#trunkWebsitesperfwebkitorgtoolspullosversionsjs">trunk/Websites/perf.webkit.org/tools/pull-os-versions.js</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkWebsitesperfwebkitorgChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/ChangeLog (214192 => 214193)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/ChangeLog 2017-03-20 21:33:58 UTC (rev 214192)
+++ trunk/Websites/perf.webkit.org/ChangeLog 2017-03-20 22:36:40 UTC (rev 214193)
</span><span class="lines">@@ -1,3 +1,48 @@
</span><ins>+2017-03-20 Ryosuke Niwa <rniwa@webkit.org>
+
+ Fix os-build-fetcher.js and subprocess.js to make them work
+ https://bugs.webkit.org/show_bug.cgi?id=169844
+
+ Reviewed by Antti Koivisto.
+
+ The script added in r213976 has a bug that it can execute commands to fetch subcommits in parallel.
+ Some commands to poll the lsit of system components is not desirable to be ran in parallel.
+
+ * server-tests/resources/mock-subprocess.js:
+ (MockSubprocess): Use const declaration.
+ (MockSubprocess.resetAndWaitForInvocation): Added.
+ (MockSubprocess.waitForInvocation): Renamed from waitingForInvocation. A function name must be a verb.
+ See https://webkit.org/code-style-guidelines/#names-verb
+ (MockSubprocess.reset): Set invocations.length to 0 so that tests can store a reference to the array
+ regardless of whether reset is called or when it's called.
+
+ * server-tests/tools-os-build-fetcher-tests.js: Updated tests per the code change. Most of codes now
+ expect each command to be ran seprately. e.g. if there were two commands to run, instead of expecting
+ them to be both ran, and resolving invocation promises, we'd wait for one command to run, resolve,
+ its subcommand to run, and then move onto the second top-level command. Also use a local reference
+ to MockSubprocess.invocations instead of using the fully qualified name.
+
+ * tools/js/os-build-fetcher.js:
+ (mapInSerialPromiseChain): Added. Calling a closure that returns a promise on each item in an array
+ in serial (not asynchronous) is a very common pattern in this class.
+ (OSBuildFetcher.fetchAndReportAllInOrder): Added.
+ (OSBuildFetcher.prototype.fetchAndReportNewBuilds): Log what the number of builds being submitted.
+ (OSBuildFetcher.prototype._fetchAvailableBuilds): Fixed the main bug. Using Promise.all would result
+ in each top-level command to be execued in parallel. Since each subcommand is executed as soon as
+ its parent command is executed, this results in commands to be executed in parallel.
+ Added a whole bunch of logging so that we can at least detect a bug like this in the future.
+ (OSBuildFetcher.prototype._commitsForAvailableBuilds): Cleanup the coding style.
+ (OSBuildFetcher.prototype._addSubCommitsForBuild): Use mapInSerialPromiseChain. Tightened the assertion
+ about the content returned by a subcommand.
+
+ * tools/js/subprocess.js: Fixed the bug that we were importing require('child_process').ChildProcess.
+ execFile is defined on require('child_process') itself.
+ (Subprocess.prototype.execute): Fixed a typo. this._childProcess doesn't exist.
+ (Subprocess):
+
+ * tools/sync-os-versions.js: Renamed from tools/pull-os-versions.js.
+ (syncLoop): Cleaned up the coding style a little. Also added logging about how long we're about to sleep.
+
</ins><span class="cx"> 2017-03-16 Ryosuke Niwa <rniwa@webkit.org>
</span><span class="cx">
</span><span class="cx"> Add the file uploading capability to the perf dashboard.
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgservertestsresourcesmocksubprocessjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/server-tests/resources/mock-subprocess.js (214192 => 214193)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/server-tests/resources/mock-subprocess.js 2017-03-20 21:33:58 UTC (rev 214192)
+++ trunk/Websites/perf.webkit.org/server-tests/resources/mock-subprocess.js 2017-03-20 22:36:40 UTC (rev 214193)
</span><span class="lines">@@ -1,4 +1,4 @@
</span><del>-var MockSubprocess = {
</del><ins>+const MockSubprocess = {
</ins><span class="cx"> execute: function (command)
</span><span class="cx"> {
</span><span class="cx"> const invocation = {command};
</span><span class="lines">@@ -16,8 +16,13 @@
</span><span class="cx"> this.invocations.push(invocation);
</span><span class="cx"> return invocation.promise;
</span><span class="cx"> },
</span><del>- waitingForInvocation: function ()
</del><ins>+ resetAndWaitForInvocation: function ()
</ins><span class="cx"> {
</span><ins>+ this.reset();
+ return this.waitForInvocation();
+ },
+ waitForInvocation: function ()
+ {
</ins><span class="cx"> if (!this._waitingInvocation) {
</span><span class="cx"> this._waitingInvocation = new Promise(function (resolve, reject) {
</span><span class="cx"> MockSubprocess._waitingInvocationResolver = resolve;
</span><span class="lines">@@ -43,7 +48,7 @@
</span><span class="cx"> },
</span><span class="cx"> reset: function ()
</span><span class="cx"> {
</span><del>- MockSubprocess.invocations = [];
</del><ins>+ MockSubprocess.invocations.length = 0;
</ins><span class="cx"> MockSubprocess._waitingInvocation = null;
</span><span class="cx"> MockSubprocess._waitingInvocationResolver = null;
</span><span class="cx"> },
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgserverteststoolsosbuildfetchertestsjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/server-tests/tools-os-build-fetcher-tests.js (214192 => 214193)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/server-tests/tools-os-build-fetcher-tests.js 2017-03-20 21:33:58 UTC (rev 214192)
+++ trunk/Websites/perf.webkit.org/server-tests/tools-os-build-fetcher-tests.js 2017-03-20 22:36:40 UTC (rev 214193)
</span><span class="lines">@@ -125,10 +125,10 @@
</span><span class="cx"> it('should only return commits whose orders are higher than specified order', () => {
</span><span class="cx"> const logger = new MockLogger;
</span><span class="cx"> const fetchter = new OSBuildFetcher(null, null, null, MockSubprocess, logger);
</span><del>- const waitingForInvocationPromise = MockSubprocess.waitingForInvocation();
</del><ins>+ const waitForInvocationPromise = MockSubprocess.waitForInvocation();
</ins><span class="cx"> const fetchCommitsPromise = fetchter._commitsForAvailableBuilds('OSX', ['list', 'build1'], '^\\.*$', 1604000000);
</span><span class="cx">
</span><del>- return waitingForInvocationPromise.then(() => {
</del><ins>+ return waitForInvocationPromise.then(() => {
</ins><span class="cx"> assert.equal(MockSubprocess.invocations.length, 1);
</span><span class="cx"> assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'build1']);
</span><span class="cx"> MockSubprocess.invocations[0].resolve('16D321\n16E321z\n\n16F321');
</span><span class="lines">@@ -145,15 +145,15 @@
</span><span class="cx"> it('should add sub-commit info for commits', () => {
</span><span class="cx"> const logger = new MockLogger;
</span><span class="cx"> const fetchter = new OSBuildFetcher(null, null, null, MockSubprocess, logger);
</span><del>- const waitingForInvocationPromise = MockSubprocess.waitingForInvocation();
</del><ins>+ const waitForInvocationPromise = MockSubprocess.waitForInvocation();
</ins><span class="cx"> const addSubCommitPromise = fetchter._addSubCommitsForBuild([osxCommit, anotherOSXCommit], ['subCommit', 'for', 'revision']);
</span><span class="cx">
</span><del>- return waitingForInvocationPromise.then(() => {
</del><ins>+ return waitForInvocationPromise.then(() => {
</ins><span class="cx"> assert.equal(MockSubprocess.invocations.length, 1);
</span><span class="cx"> assert.deepEqual(MockSubprocess.invocations[0].command, ['subCommit', 'for', 'revision', 'Sierra16D32']);
</span><span class="cx"> MockSubprocess.invocations[0].resolve(JSON.stringify(subCommitWithWebKit));
</span><span class="cx"> MockSubprocess.reset();
</span><del>- return MockSubprocess.waitingForInvocation();
</del><ins>+ return MockSubprocess.waitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><span class="cx"> assert.equal(MockSubprocess.invocations.length, 1);
</span><span class="cx"> assert.deepEqual(MockSubprocess.invocations[0].command, ['subCommit', 'for', 'revision', 'Sierra16E32']);
</span><span class="lines">@@ -173,10 +173,10 @@
</span><span class="cx"> it('should fail if the command to get sub-commit info fails', () => {
</span><span class="cx"> const logger = new MockLogger;
</span><span class="cx"> const fetchter = new OSBuildFetcher(null, null, null, MockSubprocess, logger);
</span><del>- const waitingForInvocationPromise = MockSubprocess.waitingForInvocation();
</del><ins>+ const waitForInvocationPromise = MockSubprocess.waitForInvocation();
</ins><span class="cx"> const addSubCommitPromise = fetchter._addSubCommitsForBuild([osxCommit], ['subCommit', 'for', 'revision'])
</span><span class="cx">
</span><del>- return waitingForInvocationPromise.then(() => {
</del><ins>+ return waitForInvocationPromise.then(() => {
</ins><span class="cx"> assert.equal(MockSubprocess.invocations.length, 1);
</span><span class="cx"> assert.deepEqual(MockSubprocess.invocations[0].command, ['subCommit', 'for', 'revision', 'Sierra16D32']);
</span><span class="cx"> MockSubprocess.invocations[0].reject('Failed getting sub-commit');
</span><span class="lines">@@ -194,10 +194,10 @@
</span><span class="cx"> it('should fail if entries in sub-commits does not contain revision', () => {
</span><span class="cx"> const logger = new MockLogger;
</span><span class="cx"> const fetchter = new OSBuildFetcher(null, null, null, MockSubprocess, logger);
</span><del>- const waitingForInvocationPromise = MockSubprocess.waitingForInvocation();
</del><ins>+ const waitForInvocationPromise = MockSubprocess.waitForInvocation();
</ins><span class="cx"> const addSubCommitPromise = fetchter._addSubCommitsForBuild([osxCommit], ['subCommit', 'for', 'revision'])
</span><span class="cx">
</span><del>- return waitingForInvocationPromise.then(() => {
</del><ins>+ return waitForInvocationPromise.then(() => {
</ins><span class="cx"> assert.equal(MockSubprocess.invocations.length, 1);
</span><span class="cx"> assert.deepEqual(MockSubprocess.invocations[0].command, ['subCommit', 'for', 'revision', 'Sierra16D32']);
</span><span class="cx"> MockSubprocess.invocations[0].resolve('{"WebKit":{"RandomKey": "RandomValue"}}');
</span><span class="lines">@@ -213,6 +213,7 @@
</span><span class="cx"> })
</span><span class="cx">
</span><span class="cx"> describe('OSBuildFetcher.fetchAndReportNewBuilds', () => {
</span><ins>+ const invocations = MockSubprocess.invocations;
</ins><span class="cx">
</span><span class="cx"> beforeEach(function () {
</span><span class="cx"> TestServer.database().connect({keepAlive: true});
</span><span class="lines">@@ -222,7 +223,6 @@
</span><span class="cx"> TestServer.database().disconnect();
</span><span class="cx"> });
</span><span class="cx">
</span><del>-
</del><span class="cx"> it('should report all build commits with sub-commits', () => {
</span><span class="cx"> const logger = new MockLogger;
</span><span class="cx"> const fetchter = new OSBuildFetcher(config, TestServer.remoteAPI(), slaveAuth, MockSubprocess, logger);
</span><span class="lines">@@ -250,66 +250,63 @@
</span><span class="cx"> assert.equal(result['commits'].length, 1);
</span><span class="cx"> assert.equal(result['commits'][0]['revision'], 'Sierra16E33g');
</span><span class="cx"> assert.equal(result['commits'][0]['order'], 1604003307);
</span><del>- const waitingForInvocationPromise = MockSubprocess.waitingForInvocation();
</del><ins>+ const waitForInvocationPromise = MockSubprocess.waitForInvocation();
</ins><span class="cx"> fetchAvailableBuildsPromise = fetchter._fetchAvailableBuilds();
</span><del>- return waitingForInvocationPromise;
</del><ins>+ return waitForInvocationPromise;
</ins><span class="cx"> }).then(() => {
</span><del>- return MockSubprocess.waitingForInvocation();
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'all osx 16Dxx builds']);
+ invocations[0].resolve('\n\nSierra16D68\nSierra16D69\n');
+ return MockSubprocess.resetAndWaitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><del>- MockSubprocess.invocations.sort((invocation, antoherInvocation) => invocation['command'] > antoherInvocation['command']);
- assert.equal(MockSubprocess.invocations.length, 2);
- assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'all osx 16Dxx builds']);
- assert.deepEqual(MockSubprocess.invocations[1].command, ['list', 'all osx 16Exx builds']);
- MockSubprocess.invocations[0].resolve('\n\nSierra16D68\nSierra16D69\n');
- MockSubprocess.invocations[1].resolve('\n\nSierra16E32\nSierra16E33\nSierra16E33h\nSierra16E34');
- MockSubprocess.reset();
- return MockSubprocess.waitingForInvocation();
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16D69']);
+ invocations[0].resolve(JSON.stringify(subCommitWithWebKit));
+ return MockSubprocess.resetAndWaitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><del>- MockSubprocess.invocations.sort((invocation, antoherInvocation) => invocation['command'] > antoherInvocation['command']);
- assert.equal(MockSubprocess.invocations.length, 2);
- assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16D69']);
- assert.deepEqual(MockSubprocess.invocations[1].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E33h']);
-
- MockSubprocess.invocations[0].resolve(JSON.stringify(subCommitWithWebKit));
- MockSubprocess.invocations[1].resolve(JSON.stringify(anotherSubCommitWithWebKit));
- MockSubprocess.reset();
- return MockSubprocess.waitingForInvocation();
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'all osx 16Exx builds']);
+ invocations[0].resolve('\n\nSierra16E32\nSierra16E33\nSierra16E33h\nSierra16E34');
+ return MockSubprocess.resetAndWaitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><del>- assert.equal(MockSubprocess.invocations.length, 1);
- assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E34']);
- MockSubprocess.invocations[0].resolve(JSON.stringify(anotherSubCommitWithWebKitAndJavaScriptCore));
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E33h']);
+ invocations[0].resolve(JSON.stringify(anotherSubCommitWithWebKit));
+ return MockSubprocess.resetAndWaitForInvocation();
+ }).then(() => {
+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E34']);
+ invocations[0].resolve(JSON.stringify(anotherSubCommitWithWebKitAndJavaScriptCore));
</ins><span class="cx"> return fetchAvailableBuildsPromise;
</span><span class="cx"> }).then((results) => {
</span><span class="cx"> assert.equal(results.length, 3);
</span><span class="cx"> MockSubprocess.reset();
</span><span class="cx"> fetchAndReportPromise = fetchter.fetchAndReportNewBuilds();
</span><del>- return MockSubprocess.waitingForInvocation();
</del><ins>+ return MockSubprocess.waitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><del>- return MockSubprocess.waitingForInvocation();
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'all osx 16Dxx builds']);
+ invocations[0].resolve('\n\nSierra16D68\nSierra16D69\n');
+ return MockSubprocess.resetAndWaitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><del>- MockSubprocess.invocations.sort((invocation, antoherInvocation) => invocation['command'] > antoherInvocation['command']);
- assert.equal(MockSubprocess.invocations.length, 2);
- assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'all osx 16Dxx builds']);
- assert.deepEqual(MockSubprocess.invocations[1].command, ['list', 'all osx 16Exx builds']);
- MockSubprocess.invocations[0].resolve('\n\nSierra16D68\nSierra16D69\n');
- MockSubprocess.invocations[1].resolve('\n\nSierra16E32\nSierra16E33\nSierra16E33h\nSierra16E34');
- MockSubprocess.reset();
- return MockSubprocess.waitingForInvocation();
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16D69']);
+ invocations[0].resolve(JSON.stringify(subCommitWithWebKit));
+ return MockSubprocess.resetAndWaitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><del>- MockSubprocess.invocations.sort((invocation, antoherInvocation) => invocation['command'] > antoherInvocation['command']);
- assert.equal(MockSubprocess.invocations.length, 2);
- assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16D69']);
- assert.deepEqual(MockSubprocess.invocations[1].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E33h']);
-
- MockSubprocess.invocations[0].resolve(JSON.stringify(subCommitWithWebKit));
- MockSubprocess.invocations[1].resolve(JSON.stringify(anotherSubCommitWithWebKit));
-
- MockSubprocess.reset();
- return MockSubprocess.waitingForInvocation();
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'all osx 16Exx builds']);
+ invocations[0].resolve('\n\nSierra16E32\nSierra16E33\nSierra16E33h\nSierra16E34');
+ return MockSubprocess.resetAndWaitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><del>- assert.equal(MockSubprocess.invocations.length, 1);
- MockSubprocess.invocations[0].resolve(JSON.stringify(anotherSubCommitWithWebKitAndJavaScriptCore));
- assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E34']);
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E33h']);
+ invocations[0].resolve(JSON.stringify(anotherSubCommitWithWebKit));
+ return MockSubprocess.resetAndWaitForInvocation();
+ }).then(() => {
+ assert.equal(invocations.length, 1);
+ invocations[0].resolve(JSON.stringify(anotherSubCommitWithWebKitAndJavaScriptCore));
+ assert.deepEqual(invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E34']);
</ins><span class="cx">
</span><span class="cx"> return fetchAndReportPromise;
</span><span class="cx"> }).then((result) => {
</span><span class="lines">@@ -386,18 +383,18 @@
</span><span class="cx"> assert.equal(result['commits'].length, 1);
</span><span class="cx"> assert.equal(result['commits'][0]['revision'], 'Sierra16E33g');
</span><span class="cx"> assert.equal(result['commits'][0]['order'], 1604003307);
</span><del>- const waitingForInvocationPromise = MockSubprocess.waitingForInvocation();
</del><ins>+ const waitForInvocationPromise = MockSubprocess.waitForInvocation();
</ins><span class="cx"> fetchAndReportPromise = fetchter.fetchAndReportNewBuilds();
</span><del>- return waitingForInvocationPromise;
</del><ins>+ return waitForInvocationPromise;
</ins><span class="cx"> }).then(() => {
</span><del>- return MockSubprocess.waitingForInvocation();
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'all osx 16Dxx builds']);
+ invocations[0].resolve('\n\nSierra16D68\nSierra16D69\n');
+ return MockSubprocess.resetAndWaitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><del>- MockSubprocess.invocations.sort((invocation, antoherInvocation) => invocation['command'] > antoherInvocation['command']);
- assert.equal(MockSubprocess.invocations.length, 2);
- assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'all osx 16Dxx builds']);
- assert.deepEqual(MockSubprocess.invocations[1].command, ['list', 'all osx 16Exx builds']);
- MockSubprocess.invocations[0].resolve('\n\nSierra16D68\nSierra16D69\n');
- MockSubprocess.invocations[1].resolve('\n\nSierra16E32\nSierra16E33\nSierra16E33h\nSierra16E34');
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'all osx 16Exx builds']);
+ invocations[0].resolve('\n\nSierra16E32\nSierra16E33\nSierra16E33h\nSierra16E34');
</ins><span class="cx"> return fetchAndReportPromise;
</span><span class="cx"> }).then((result) => {
</span><span class="cx"> assert.equal(result['status'], 'OK');
</span><span class="lines">@@ -471,35 +468,32 @@
</span><span class="cx"> assert.equal(result['commits'][0]['revision'], 'Sierra16E33g');
</span><span class="cx"> assert.equal(result['commits'][0]['order'], 1604003307);
</span><span class="cx">
</span><del>- const waitingForInvocationPromise = MockSubprocess.waitingForInvocation();
</del><ins>+ const waitForInvocationPromise = MockSubprocess.waitForInvocation();
</ins><span class="cx"> fetchAndReportPromise = fetchter.fetchAndReportNewBuilds();
</span><del>- return waitingForInvocationPromise;
</del><ins>+ return waitForInvocationPromise;
</ins><span class="cx"> }).then(() => {
</span><del>- return MockSubprocess.waitingForInvocation();
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'all osx 16Dxx builds']);
+ invocations[0].resolve('\n\nSierra16D68\nSierra16D69\n');
+ return MockSubprocess.resetAndWaitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><del>- MockSubprocess.invocations.sort((invocation, antoherInvocation) => invocation['command'] > antoherInvocation['command']);
- assert.equal(MockSubprocess.invocations.length, 2);
- assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'all osx 16Dxx builds']);
- assert.deepEqual(MockSubprocess.invocations[1].command, ['list', 'all osx 16Exx builds']);
- MockSubprocess.invocations[0].resolve('\n\nSierra16D68\nSierra16D69\n');
- MockSubprocess.invocations[1].resolve('\n\nSierra16E32\nSierra16E33\nSierra16E33h\nSierra16E34');
- MockSubprocess.reset();
- return MockSubprocess.waitingForInvocation();
- }).then(() => {
- MockSubprocess.invocations.sort((invocation, antoherInvocation) => invocation['command'] > antoherInvocation['command']);
- assert.equal(MockSubprocess.invocations.length, 2);
- assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16D69']);
- assert.deepEqual(MockSubprocess.invocations[1].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E33h']);
-
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16D69']);
</ins><span class="cx"> MockSubprocess.invocations[0].resolve(JSON.stringify(subCommitWithWebKit));
</span><del>- MockSubprocess.invocations[1].resolve(JSON.stringify(anotherSubCommitWithWebKit));
- MockSubprocess.reset();
- return MockSubprocess.waitingForInvocation();
</del><ins>+ return MockSubprocess.resetAndWaitForInvocation();
</ins><span class="cx"> }).then(() => {
</span><del>- assert.equal(MockSubprocess.invocations.length, 1);
- assert.deepEqual(MockSubprocess.invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E34']);
- MockSubprocess.invocations[0].reject('Command failed');
-
</del><ins>+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'all osx 16Exx builds']);
+ invocations[0].resolve('\n\nSierra16E32\nSierra16E33\nSierra16E33h\nSierra16E34');
+ return MockSubprocess.resetAndWaitForInvocation();
+ }).then(() => {
+ assert.deepEqual(invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E33h']);
+ invocations[0].resolve(JSON.stringify(anotherSubCommitWithWebKit));
+ return MockSubprocess.resetAndWaitForInvocation();
+ }).then(() => {
+ assert.equal(invocations.length, 1);
+ assert.deepEqual(invocations[0].command, ['list', 'subCommit', 'for', 'revision', 'Sierra16E34']);
+ invocations[0].reject('Command failed');
</ins><span class="cx"> return fetchAndReportPromise.then(() => {
</span><span class="cx"> assert(false, 'should never be reached');
</span><span class="cx"> }, (error_output) => {
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtoolsjsosbuildfetcherjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/tools/js/os-build-fetcher.js (214192 => 214193)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tools/js/os-build-fetcher.js 2017-03-20 21:33:58 UTC (rev 214192)
+++ trunk/Websites/perf.webkit.org/tools/js/os-build-fetcher.js 2017-03-20 22:36:40 UTC (rev 214193)
</span><span class="lines">@@ -2,6 +2,14 @@
</span><span class="cx">
</span><span class="cx"> let assert = require('assert');
</span><span class="cx">
</span><ins>+function mapInSerialPromiseChain(list, callback)
+{
+ const results = [];
+ return list.reduce((chainedPromise, item) => {
+ return chainedPromise.then(() => callback(item)).then((result) => results.push(result));
+ }, Promise.resolve()).then(() => results);
+}
+
</ins><span class="cx"> class OSBuildFetcher {
</span><span class="cx">
</span><span class="cx"> constructor(osConfig, remoteAPI, slaveAuth, subprocess, logger)
</span><span class="lines">@@ -14,9 +22,15 @@
</span><span class="cx"> this._subprocess = subprocess;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ static fetchAndReportAllInOrder(fetcherList)
+ {
+ return mapInSerialPromiseChain(fetcherList, (fetcher) => fetcher.fetchAndReportNewBuilds());
+ }
+
</ins><span class="cx"> fetchAndReportNewBuilds()
</span><span class="cx"> {
</span><del>- return this._fetchAvailableBuilds().then((results) =>{
</del><ins>+ return this._fetchAvailableBuilds().then((results) => {
+ this._logger.log(`Submitting ${results.length} builds for ${this._osConfig['name']}`);
</ins><span class="cx"> return this._submitCommits(results);
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="lines">@@ -27,24 +41,29 @@
</span><span class="cx"> const repositoryName = config['name'];
</span><span class="cx"> let customCommands = config['customCommands'];
</span><span class="cx">
</span><del>- return Promise.all(customCommands.map((command) => {
</del><ins>+ return mapInSerialPromiseChain(customCommands, (command) => {
</ins><span class="cx"> assert(command['minRevision']);
</span><span class="cx"> assert(command['maxRevision']);
</span><span class="cx"> const minRevisionOrder = this._computeOrder(command['minRevision']);
</span><span class="cx"> const maxRevisionOrder = this._computeOrder(command['maxRevision']);
</span><span class="cx">
</span><del>- let fetchCommitsPromise = this._remoteAPI.getJSONWithStatus(`/api/commits/${escape(repositoryName)}/last-reported?from=${minRevisionOrder}&to=${maxRevisionOrder}`).then((result) => {
</del><ins>+ const url = `/api/commits/${escape(repositoryName)}/last-reported?from=${minRevisionOrder}&to=${maxRevisionOrder}`;
+
+ return this._remoteAPI.getJSONWithStatus(url).then((result) => {
</ins><span class="cx"> const minOrder = result['commits'].length == 1 ? parseInt(result['commits'][0]['order']) : 0;
</span><span class="cx"> return this._commitsForAvailableBuilds(repositoryName, command['command'], command['linesToIgnore'], minOrder);
</span><del>- })
</del><ins>+ }).then((commits) => {
+ const label = 'name' in command ? `"${command['name']}"` : `"command['minRevision']" to "command['maxRevision']"`;
+ this._logger.log(`Found ${commits.length} builds for ${label}`);
</ins><span class="cx">
</span><del>- if ('subCommitCommand' in command)
- fetchCommitsPromise = fetchCommitsPromise.then((commits) => this._addSubCommitsForBuild(commits, command['subCommitCommand']));
</del><ins>+ if ('subCommitCommand' in command) {
+ this._logger.log(`Resolving subcommits for "${label}"`);
+ return this._addSubCommitsForBuild(commits, command['subCommitCommand']);
+ }
</ins><span class="cx">
</span><del>- return fetchCommitsPromise;
- })).then(results => {
- return Array.prototype.concat.apply([], results);
- });
</del><ins>+ return commits;
+ });
+ }).then((results) => [].concat(...results));
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> _computeOrder(revision)
</span><span class="lines">@@ -65,27 +84,27 @@
</span><span class="cx"> let lines = output.split('\n');
</span><span class="cx"> if (linesToIgnore){
</span><span class="cx"> const regex = new RegExp(linesToIgnore);
</span><del>- lines = lines.filter(function(line) {return !regex.exec(line);});
</del><ins>+ lines = lines.filter((line) => !regex.exec(line));
</ins><span class="cx"> }
</span><del>- return lines.map(revision => ({repository, revision, 'order': this._computeOrder(revision)}))
- .filter(commit => commit['order'] > minOrder);
</del><ins>+ return lines.map((revision) => ({repository, revision, 'order': this._computeOrder(revision)}))
+ .filter((commit) => commit['order'] > minOrder);
</ins><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> _addSubCommitsForBuild(commits, command)
</span><span class="cx"> {
</span><del>- return commits.reduce((promise, commit) => {
- return promise.then(() => {
- return this._subprocess.execute(command.concat(commit['revision']));
- }).then((subCommitOutput) => {
</del><ins>+ return mapInSerialPromiseChain(commits, (commit) => {
+ return this._subprocess.execute(command.concat(commit['revision'])).then((subCommitOutput) => {
</ins><span class="cx"> const subCommits = JSON.parse(subCommitOutput);
</span><span class="cx"> for (let repositoryName in subCommits) {
</span><span class="cx"> const subCommit = subCommits[repositoryName];
</span><del>- assert(subCommit['revision']);
</del><ins>+ assert.deepEqual(Object.keys(subCommit), ['revision']);
+ assert(typeof(subCommit['revision']) == 'string');
</ins><span class="cx"> }
</span><span class="cx"> commit['subCommits'] = subCommits;
</span><ins>+ return commit;
</ins><span class="cx"> });
</span><del>- }, Promise.resolve()).then(() => commits);
</del><ins>+ });
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> _submitCommits(commits)
</span><span class="lines">@@ -94,5 +113,6 @@
</span><span class="cx"> return this._remoteAPI.postJSONWithStatus('/api/report-commits/', commitsToReport);
</span><span class="cx"> }
</span><span class="cx"> }
</span><ins>+
</ins><span class="cx"> if (typeof module != 'undefined')
</span><span class="cx"> module.exports.OSBuildFetcher = OSBuildFetcher;
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtoolsjssubprocessjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/tools/js/subprocess.js (214192 => 214193)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tools/js/subprocess.js 2017-03-20 21:33:58 UTC (rev 214192)
+++ trunk/Websites/perf.webkit.org/tools/js/subprocess.js 2017-03-20 22:36:40 UTC (rev 214193)
</span><span class="lines">@@ -1,10 +1,10 @@
</span><span class="cx"> 'use strict';
</span><del>-const childProcess = require('child_process').ChildProcess;
</del><ins>+const childProcess = require('child_process');
</ins><span class="cx">
</span><span class="cx"> class Subprocess {
</span><span class="cx"> execute(command) {
</span><span class="cx"> return new Promise((resolve, reject) => {
</span><del>- this._childProcess.execFile(command[0], command.slice(1), (error, stdout, stderr) => {
</del><ins>+ childProcess.execFile(command[0], command.slice(1), (error, stdout, stderr) => {
</ins><span class="cx"> if (error)
</span><span class="cx"> reject(stderr);
</span><span class="cx"> else
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtoolspullosversionsjs"></a>
<div class="delfile"><h4>Deleted: trunk/Websites/perf.webkit.org/tools/pull-os-versions.js (214192 => 214193)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tools/pull-os-versions.js 2017-03-20 21:33:58 UTC (rev 214192)
+++ trunk/Websites/perf.webkit.org/tools/pull-os-versions.js 2017-03-20 22:36:40 UTC (rev 214193)
</span><span class="lines">@@ -1,46 +0,0 @@
</span><del>-#!/usr/local/bin/node
-'use strict';
-
-
-let OSBuildFetcher = require('./js/os-build-fetcher.js').OSBuildFetcher;
-let RemoteAPI = require('./js/remote.js').RemoteAPI;
-let Subprocess = require('./js/subprocess.js').Subprocess;
-let fs = require('fs');
-let parseArguments = require('./js/parse-arguments.js').parseArguments;
-
-function main(argv)
-{
- let options = parseArguments(argv, [
- {name: '--os-config-json', required: true},
- {name: '--server-config-json', required: true},
- {name: '--seconds-to-sleep', type: parseFloat, default: 43200},
- ]);
- if (!options)
- return;
-
- syncLoop(options);
-}
-
-function syncLoop(options)
-{
- let osConfigList = JSON.parse(fs.readFileSync(options['--os-config-json'], 'utf8'));
- let serverConfig = JSON.parse(fs.readFileSync(options['--server-config-json'], 'utf8'));
-
- const remoteAPI = new RemoteAPI(serverConfig.server);
-
- Promise.all(osConfigList.map(osConfig => new OSBuildFetcher(osConfig, remoteAPI, serverConfig.slave, new Subprocess, console))).then((fetchers) => {
- return fetchers.reduce((promise, fetcher) => {
- return promise.then(() => fetcher.fetchAndReportNewBuilds());
- }, Promise.resolve());
- }).catch((error) => {
- console.error(error);
- if (typeof(error.stack) == 'string') {
- for (let line of error.stack.split('\n'))
- console.error(line);
- }
- }).then(function () {
- setTimeout(syncLoop.bind(global, options), options['--seconds-to-sleep'] * 1000);
- });
-}
-
-main(process.argv);
</del></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtoolssyncosversionsjsfromrev214192trunkWebsitesperfwebkitorgtoolspullosversionsjs"></a>
<div class="copfile"><h4>Copied: trunk/Websites/perf.webkit.org/tools/sync-os-versions.js (from rev 214192, trunk/Websites/perf.webkit.org/tools/pull-os-versions.js) (0 => 214193)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tools/sync-os-versions.js (rev 0)
+++ trunk/Websites/perf.webkit.org/tools/sync-os-versions.js 2017-03-20 22:36:40 UTC (rev 214193)
</span><span class="lines">@@ -0,0 +1,45 @@
</span><ins>+#!/usr/local/bin/node
+'use strict';
+
+
+let OSBuildFetcher = require('./js/os-build-fetcher.js').OSBuildFetcher;
+let RemoteAPI = require('./js/remote.js').RemoteAPI;
+let Subprocess = require('./js/subprocess.js').Subprocess;
+let fs = require('fs');
+let parseArguments = require('./js/parse-arguments.js').parseArguments;
+
+function main(argv)
+{
+ let options = parseArguments(argv, [
+ {name: '--os-config-json', required: true},
+ {name: '--server-config-json', required: true},
+ {name: '--seconds-to-sleep', type: parseFloat, default: 43200},
+ ]);
+ if (!options)
+ return;
+
+ syncLoop(options);
+}
+
+function syncLoop(options)
+{
+ let osConfigList = JSON.parse(fs.readFileSync(options['--os-config-json'], 'utf8'));
+ let serverConfig = JSON.parse(fs.readFileSync(options['--server-config-json'], 'utf8'));
+
+ const remoteAPI = new RemoteAPI(serverConfig.server);
+
+ const fetchers = osConfigList.map((osConfig) => new OSBuildFetcher(osConfig, remoteAPI, serverConfig.slave, new Subprocess, console));
+ OSBuildFetcher.fetchAndReportAllInOrder(fetchers).catch((error) => {
+ console.error(error);
+ if (typeof(error.stack) == 'string') {
+ for (let line of error.stack.split('\n'))
+ console.error(line);
+ }
+ }).then(() => {
+ const secondsToSleep = options['--seconds-to-sleep'];
+ console.log(`Sleeping for ${Math.floor(secondsToSleep / 3600)}h ${Math.floor(secondsToSleep / 60) % 60}m ${secondsToSleep % 60}s`);
+ setTimeout(() => syncLoop(options), secondsToSleep * 1000);
+ });
+}
+
+main(process.argv);
</ins></span></pre>
</div>
</div>
</body>
</html>