<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[172891] trunk/Tools</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/172891">172891</a></dd>
<dt>Author</dt> <dd>ap@apple.com</dd>
<dt>Date</dt> <dd>2014-08-23 18:22:00 -0700 (Sat, 23 Aug 2014)</dd>
</dl>
<h3>Log Message</h3>
<pre>build.webkit.org/dashboard: Make it possible to pull historic data from Buildbot
https://bugs.webkit.org/show_bug.cgi?id=136182
Reviewed by Timothy Hatcher.
* BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotIteration.js:
Make it possible to create a complete BuildbotIteration object with JSON, not only by
loading data from the network.
An iteration has three states:
1. Just created, it only knows the revision, and whether it's already finished.
BuildbotIteration constructor used to create these.
2. A complete JSON report loaded from buildbot. The new constructor form creates these.
3. Data about tests is fetched.
* BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueue.js:
Make it possible to pull all data from a buildbot queue. This needs to be better
optimized eventually, as it's somewhat slow.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkToolsBuildSlaveSupportbuildwebkitorgconfigpublic_htmldashboardScriptsBuildbotIterationjs">trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotIteration.js</a></li>
<li><a href="#trunkToolsBuildSlaveSupportbuildwebkitorgconfigpublic_htmldashboardScriptsBuildbotQueuejs">trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueue.js</a></li>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkToolsBuildSlaveSupportbuildwebkitorgconfigpublic_htmldashboardScriptsBuildbotIterationjs"></a>
<div class="modfile"><h4>Modified: trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotIteration.js (172890 => 172891)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotIteration.js 2014-08-23 20:39:07 UTC (rev 172890)
+++ trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotIteration.js 2014-08-24 01:22:00 UTC (rev 172891)
</span><span class="lines">@@ -23,15 +23,22 @@
</span><span class="cx"> * THE POSSIBILITY OF SUCH DAMAGE.
</span><span class="cx"> */
</span><span class="cx">
</span><del>-BuildbotIteration = function(queue, id, finished)
</del><ins>+BuildbotIteration = function(queue, dataOrID, finished)
</ins><span class="cx"> {
</span><span class="cx"> BaseObject.call(this);
</span><span class="cx">
</span><span class="cx"> console.assert(queue);
</span><span class="cx">
</span><span class="cx"> this.queue = queue;
</span><del>- this.id = id;
</del><span class="cx">
</span><ins>+ if (typeof dataOrID === "object") {
+ this._parseData(dataOrID);
+ return;
+ }
+
+ console.assert(typeof dataOrID === "number");
+ this.id = dataOrID;
+
</ins><span class="cx"> this.loaded = false;
</span><span class="cx">
</span><span class="cx"> this.openSourceRevision = null;
</span><span class="lines">@@ -153,14 +160,8 @@
</span><span class="cx"> return null;
</span><span class="cx"> },
</span><span class="cx">
</span><del>- update: function()
</del><ins>+ _parseData: function(data)
</ins><span class="cx"> {
</span><del>- if (this.loaded && this._finished)
- return;
-
- if (this.queue.buildbot.needsAuthentication && this.queue.buildbot.authenticationStatus === Buildbot.AuthenticationStatus.InvalidCredentials)
- return;
-
</del><span class="cx"> function collectTestResults(data, stepName)
</span><span class="cx"> {
</span><span class="cx"> var testStep = data.steps.findFirst(function(step) { return step.name === stepName; });
</span><span class="lines">@@ -214,68 +215,99 @@
</span><span class="cx"> return testResults;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- JSON.load(this.queue.baseURL + "/builds/" + this.id, function(data) {
- this.queue.buildbot.isAuthenticated = true;
- if (!data || !data.properties)
- return;
</del><ins>+ console.assert(!this.id || this.id === data.number);
+ this.id = data.number;
</ins><span class="cx">
</span><del>- // The property got_revision may have the following forms:
- //
- // ["got_revision",{"Internal":"1357","WebKitOpenSource":"2468"},"Source"]
- // OR
- // ["got_revision","2468_1357","Source"]
- // OR
- // ["got_revision","2468","Source"]
- //
- // When extracting the OpenSource revision from property got_revision we don't need to check whether the
- // value of got_revision is a dictionary (represents multiple codebases) or a string literal because we
- // assume that got_revision contains the OpenSource revision. However, it may not have the Internal
- // revision. Therefore, we only look at got_revision to extract the Internal revision when it's
- // a dictionary.
</del><ins>+ // The property got_revision may have the following forms:
+ //
+ // ["got_revision",{"Internal":"1357","WebKitOpenSource":"2468"},"Source"]
+ // OR
+ // ["got_revision","2468_1357","Source"]
+ // OR
+ // ["got_revision","2468","Source"]
+ //
+ // When extracting the OpenSource revision from property got_revision we don't need to check whether the
+ // value of got_revision is a dictionary (represents multiple codebases) or a string literal because we
+ // assume that got_revision contains the OpenSource revision. However, it may not have the Internal
+ // revision. Therefore, we only look at got_revision to extract the Internal revision when it's
+ // a dictionary.
</ins><span class="cx">
</span><del>- var openSourceRevisionProperty = data.properties.findFirst(function(property) { return property[0] === "got_revision" || property[0] === "revision" || property[0] === "opensource_got_revision"; });
- this.openSourceRevision = parseRevisionProperty(openSourceRevisionProperty, "WebKit");
</del><ins>+ var openSourceRevisionProperty = data.properties.findFirst(function(property) { return property[0] === "got_revision" || property[0] === "revision" || property[0] === "opensource_got_revision"; });
+ this.openSourceRevision = parseRevisionProperty(openSourceRevisionProperty, "WebKit");
</ins><span class="cx">
</span><del>- var internalRevisionProperty = data.properties.findFirst(function(property) { return property[0] === "internal_got_revision" || isMultiCodebaseGotRevisionProperty(property); });
- this.internalRevision = parseRevisionProperty(internalRevisionProperty, "Internal");
</del><ins>+ var internalRevisionProperty = data.properties.findFirst(function(property) { return property[0] === "internal_got_revision" || isMultiCodebaseGotRevisionProperty(property); });
+ this.internalRevision = parseRevisionProperty(internalRevisionProperty, "Internal");
</ins><span class="cx">
</span><del>- var layoutTestResults = collectTestResults.call(this, data, "layout-test");
- this.layoutTestResults = layoutTestResults ? new BuildbotTestResults(this, layoutTestResults) : null;
</del><ins>+ this.branch = data.properties.findFirst(function(property) { return property[0] === "branch" })[1];
</ins><span class="cx">
</span><del>- var javascriptTestResults = collectTestResults.call(this, data, "jscore-test");
- this.javascriptTestResults = javascriptTestResults ? new BuildbotTestResults(this, javascriptTestResults) : null;
</del><ins>+ this.changes = [];
+ var changes = data.sourceStamp.changes;
+ for (var i = 0; i < changes.length; ++i)
+ this.changes[i] = { revisionNumber: parseInt(changes[i].revision, 10) }
</ins><span class="cx">
</span><del>- var apiTestResults = collectTestResults.call(this, data, "run-api-tests");
- this.apiTestResults = apiTestResults ? new BuildbotTestResults(this, apiTestResults) : null;
</del><ins>+ this.startTime = new Date(data.times[0] * 1000);
+ this.endTime = new Date(data.times[1] * 1000);
</ins><span class="cx">
</span><del>- var platformAPITestResults = collectTestResults.call(this, data, "API tests");
- this.platformAPITestResults = platformAPITestResults ? new BuildbotTestResults(this, platformAPITestResults) : null;
</del><ins>+ var layoutTestResults = collectTestResults.call(this, data, "layout-test");
+ this.layoutTestResults = layoutTestResults ? new BuildbotTestResults(this, layoutTestResults) : null;
</ins><span class="cx">
</span><del>- var pythonTestResults = collectTestResults.call(this, data, "webkitpy-test");
- this.pythonTestResults = pythonTestResults ? new BuildbotTestResults(this, pythonTestResults) : null;
</del><ins>+ var javascriptTestResults = collectTestResults.call(this, data, "jscore-test");
+ this.javascriptTestResults = javascriptTestResults ? new BuildbotTestResults(this, javascriptTestResults) : null;
</ins><span class="cx">
</span><del>- var perlTestResults = collectTestResults.call(this, data, "webkitperl-test");
- this.perlTestResults = perlTestResults ? new BuildbotTestResults(this, perlTestResults) : null;
</del><ins>+ var apiTestResults = collectTestResults.call(this, data, "run-api-tests");
+ this.apiTestResults = apiTestResults ? new BuildbotTestResults(this, apiTestResults) : null;
</ins><span class="cx">
</span><del>- var bindingTestResults = collectTestResults.call(this, data, "bindings-generation-tests");
- this.bindingTestResults = bindingTestResults ? new BuildbotTestResults(this, bindingTestResults) : null;
</del><ins>+ var platformAPITestResults = collectTestResults.call(this, data, "API tests");
+ this.platformAPITestResults = platformAPITestResults ? new BuildbotTestResults(this, platformAPITestResults) : null;
</ins><span class="cx">
</span><del>- this.loaded = true;
</del><ins>+ var pythonTestResults = collectTestResults.call(this, data, "webkitpy-test");
+ this.pythonTestResults = pythonTestResults ? new BuildbotTestResults(this, pythonTestResults) : null;
</ins><span class="cx">
</span><del>- this._firstFailedStep = data.steps.findFirst(function(step) { return step.results[0] === BuildbotIteration.FAILURE; });
</del><ins>+ var perlTestResults = collectTestResults.call(this, data, "webkitperl-test");
+ this.perlTestResults = perlTestResults ? new BuildbotTestResults(this, perlTestResults) : null;
</ins><span class="cx">
</span><del>- console.assert(data.results === null || typeof data.results === "number");
- this._result = data.results;
</del><ins>+ var bindingTestResults = collectTestResults.call(this, data, "bindings-generation-tests");
+ this.bindingTestResults = bindingTestResults ? new BuildbotTestResults(this, bindingTestResults) : null;
</ins><span class="cx">
</span><del>- this.text = data.text.join(" ");
</del><ins>+ this.loaded = true;
</ins><span class="cx">
</span><del>- if (!data.currentStep)
- this.finished = true;
</del><ins>+ this._firstFailedStep = data.steps.findFirst(function(step) { return step.results[0] === BuildbotIteration.FAILURE; });
</ins><span class="cx">
</span><del>- // Update the sorting since it is based on the revisions we just loaded.
- this.queue.sortIterations();
</del><ins>+ console.assert(data.results === null || typeof data.results === "number");
+ this._result = data.results;
</ins><span class="cx">
</span><del>- this.dispatchEventToListeners(BuildbotIteration.Event.Updated);
</del><ins>+ this.text = data.text.join(" ");
+
+ if (!data.currentStep)
+ this.finished = true;
+
+ // Update the sorting since it is based on the revisions we just loaded.
+ this.queue.sortIterations();
+ },
+
+ _updateWithData: function(data)
+ {
+ if (this.loaded && this._finished)
+ return;
+
+ this._parseData(data);
+ this.dispatchEventToListeners(BuildbotIteration.Event.Updated);
+ },
+
+ update: function()
+ {
+ if (this.loaded && this._finished)
+ return;
+
+ if (this.queue.buildbot.needsAuthentication && this.queue.buildbot.authenticationStatus === Buildbot.AuthenticationStatus.InvalidCredentials)
+ return;
+
+ JSON.load(this.queue.baseURL + "/builds/" + this.id, function(data) {
+ this.queue.buildbot.isAuthenticated = true;
+ if (!data || !data.properties)
+ return;
+
+ this._updateWithData(data);
</ins><span class="cx"> }.bind(this),
</span><span class="cx"> function(data) {
</span><span class="cx"> if (data.errorType === JSON.LoadError && data.errorHTTPCode === 401) {
</span></span></pre></div>
<a id="trunkToolsBuildSlaveSupportbuildwebkitorgconfigpublic_htmldashboardScriptsBuildbotQueuejs"></a>
<div class="modfile"><h4>Modified: trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueue.js (172890 => 172891)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueue.js 2014-08-23 20:39:07 UTC (rev 172890)
+++ trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueue.js 2014-08-24 01:22:00 UTC (rev 172891)
</span><span class="lines">@@ -65,6 +65,12 @@
</span><span class="cx"> return this.buildbot.baseURL + "json/builders/" + encodeURIComponent(this.id);
</span><span class="cx"> },
</span><span class="cx">
</span><ins>+ get allIterationsURL()
+ {
+ // Getting too many builds results in a timeout error, 10000 is OK.
+ return this.buildbot.baseURL + "json/builders/" + encodeURIComponent(this.id) + "/builds/_all/?max=10000";
+ },
+
</ins><span class="cx"> get overviewURL()
</span><span class="cx"> {
</span><span class="cx"> return this.buildbot.baseURL + "builders/" + encodeURIComponent(this.id) + "?numbuilds=50";
</span><span class="lines">@@ -118,13 +124,39 @@
</span><span class="cx"> return null;
</span><span class="cx"> },
</span><span class="cx">
</span><del>- update: function()
</del><ins>+ _load: function(url, callback)
</ins><span class="cx"> {
</span><span class="cx"> if (this.buildbot.needsAuthentication && this.buildbot.authenticationStatus === Buildbot.AuthenticationStatus.InvalidCredentials)
</span><span class="cx"> return;
</span><span class="cx">
</span><del>- JSON.load(this.baseURL, function(data) {
- this.buildbot.isAuthenticated = true;
</del><ins>+ JSON.load(
+ url,
+ function(data) {
+ this.buildbot.isAuthenticated = true;
+ callback(data);
+ }.bind(this),
+ function(data) {
+ if (this.buildbot.isAuthenticated) {
+ // FIXME (128006): Safari/WebKit should coalesce authentication requests with the same origin and authentication realm.
+ // In absence of the fix, Safari presents additional authentication dialogs regardless of whether an earlier authentication
+ // dialog was dismissed. As a way to ameliorate the user experience where a person authenticated successfully using an
+ // earlier authentication dialog and cancelled the authentication dialog associated with the load for this queue, we call
+ // ourself so that we can schedule another load, which should complete successfully now that we have credentials.
+ this._load(url, callback);
+ return;
+ }
+ if (data.errorType === JSON.LoadError && data.errorHTTPCode === 401) {
+ this.buildbot.isAuthenticated = false;
+ this.dispatchEventToListeners(BuildbotQueue.Event.UnauthorizedAccess, { });
+ }
+ }.bind(this),
+ {withCredentials: this.buildbot.needsAuthentication}
+ );
+ },
+
+ update: function()
+ {
+ this._load(this.baseURL, function(data) {
</ins><span class="cx"> if (!(data.cachedBuilds instanceof Array))
</span><span class="cx"> return;
</span><span class="cx">
</span><span class="lines">@@ -155,22 +187,22 @@
</span><span class="cx"> this.sortIterations();
</span><span class="cx">
</span><span class="cx"> this.dispatchEventToListeners(BuildbotQueue.Event.IterationsAdded, {addedIterations: newIterations});
</span><del>- }.bind(this),
- function(data) {
- if (this.buildbot.isAuthenticated) {
- // FIXME (128006): Safari/WebKit should coallesce authentication requests with the same origin and authentication realm.
- // In absence of the fix, Safari presents additional authentication dialogs regardless of whether an earlier authentication
- // dialog was dismissed. As a way to ameliorate the user experience where a person authenticated successfully using an
- // earlier authentication dialog and cancelled the authentication dialog associated with the load for this queue, we call
- // ourself so that we can schedule another load, which should complete successfully now that we have credentials.
- this.update();
- return;
</del><ins>+ }.bind(this));
+ },
+
+ loadAll: function(callback)
+ {
+ // FIXME: Don't load everything at once, do it incrementally as requested.
+ this._load(this.allIterationsURL, function(data) {
+ for (var idString in data) {
+ console.assert(typeof idString === "string");
+ iteration = new BuildbotIteration(this, data[idString]);
+ this.iterations.push(iteration);
+ this._knownIterations[iteration.id] = iteration;
</ins><span class="cx"> }
</span><del>- if (data.errorType === JSON.LoadError && data.errorHTTPCode === 401) {
- this.buildbot.isAuthenticated = false;
- this.dispatchEventToListeners(BuildbotQueue.Event.UnauthorizedAccess, { });
- }
- }.bind(this), {withCredentials: this.buildbot.needsAuthentication});
</del><ins>+
+ callback(this);
+ }.bind(this));
</ins><span class="cx"> },
</span><span class="cx">
</span><span class="cx"> sortIterations: function()
</span></span></pre></div>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (172890 => 172891)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog 2014-08-23 20:39:07 UTC (rev 172890)
+++ trunk/Tools/ChangeLog 2014-08-24 01:22:00 UTC (rev 172891)
</span><span class="lines">@@ -1,3 +1,24 @@
</span><ins>+2014-08-23 Alexey Proskuryakov <ap@apple.com>
+
+ build.webkit.org/dashboard: Make it possible to pull historic data from Buildbot
+ https://bugs.webkit.org/show_bug.cgi?id=136182
+
+ Reviewed by Timothy Hatcher.
+
+ * BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotIteration.js:
+ Make it possible to create a complete BuildbotIteration object with JSON, not only by
+ loading data from the network.
+
+ An iteration has three states:
+ 1. Just created, it only knows the revision, and whether it's already finished.
+ BuildbotIteration constructor used to create these.
+ 2. A complete JSON report loaded from buildbot. The new constructor form creates these.
+ 3. Data about tests is fetched.
+
+ * BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueue.js:
+ Make it possible to pull all data from a buildbot queue. This needs to be better
+ optimized eventually, as it's somewhat slow.
+
</ins><span class="cx"> 2014-08-22 Renato Nagy <nagy.renato@stud.u-szeged.hu>
</span><span class="cx">
</span><span class="cx"> Remove LevelDB.
</span></span></pre>
</div>
</div>
</body>
</html>