<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[172747] trunk/Tools</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/172747">172747</a></dd>
<dt>Author</dt> <dd>ap@apple.com</dd>
<dt>Date</dt> <dd>2014-08-19 00:37:45 -0700 (Tue, 19 Aug 2014)</dd>
</dl>
<h3>Log Message</h3>
<pre>build.webkit.org/dashboard should not request 50 revisions from trac each time
https://bugs.webkit.org/show_bug.cgi?id=127130
build.webkit.org/dashboard sometimes fetches a Trac revision in an intermediate state, and never updates later
https://bugs.webkit.org/show_bug.cgi?id=127131
Reviewed by Timothy Hatcher.
Turns out that requesting 50 builds is much slower than requesting by date - even
if the request ends up returning more than 50 results. There is no way to only
request updates, but this change brings request time from 6-8 seconds down to
less than a second.
This patch generalizes date handling for later use in metrics code. As part of the
rewrite, I made newly fetched data update author e-mail in previously fetched
revisions, as it changes after commit queue first lands.
* BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueueView.js:
Updated for event rename. I changed the trac event to not contain the list of new
commits, as we now sometimes update old commits, and that couldn't be expressed
in event data. We never used the list anywhere in the first place.
* BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/Trac.js:
(Trac.prototype._xmlTimelineURL): Made this function take arbitrary dates. When called
without arguments, return commits for today and yesterday.
(Trac.prototype._loaded):
(Trac.prototype.update): Moved the function for processing loaded results out of
here for clarity, and also because I'm going to have a separate loading code path
for metrics.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkToolsBuildSlaveSupportbuildwebkitorgconfigpublic_htmldashboardScriptsBuildbotQueueViewjs">trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueueView.js</a></li>
<li><a href="#trunkToolsBuildSlaveSupportbuildwebkitorgconfigpublic_htmldashboardScriptsTracjs">trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/Trac.js</a></li>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkToolsBuildSlaveSupportbuildwebkitorgconfigpublic_htmldashboardScriptsBuildbotQueueViewjs"></a>
<div class="modfile"><h4>Modified: trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueueView.js (172746 => 172747)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueueView.js 2014-08-19 07:14:30 UTC (rev 172746)
+++ trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueueView.js 2014-08-19 07:37:45 UTC (rev 172747)
</span><span class="lines">@@ -48,9 +48,9 @@
</span><span class="cx"> queue.addEventListener(BuildbotQueue.Event.UnauthorizedAccess, this._unauthorizedAccess, this);
</span><span class="cx"> }.bind(this));
</span><span class="cx">
</span><del>- webkitTrac.addEventListener(Trac.Event.NewCommitsRecorded, this._newCommitsRecorded, this);
</del><ins>+ webkitTrac.addEventListener(Trac.Event.CommitsUpdated, this._newCommitsRecorded, this);
</ins><span class="cx"> if (typeof internalTrac != "undefined")
</span><del>- internalTrac.addEventListener(Trac.Event.NewCommitsRecorded, this._newCommitsRecorded, this);
</del><ins>+ internalTrac.addEventListener(Trac.Event.CommitsUpdated, this._newCommitsRecorded, this);
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> BaseObject.addConstructorFunctions(BuildbotQueueView);
</span></span></pre></div>
<a id="trunkToolsBuildSlaveSupportbuildwebkitorgconfigpublic_htmldashboardScriptsTracjs"></a>
<div class="modfile"><h4>Modified: trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/Trac.js (172746 => 172747)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/Trac.js 2014-08-19 07:14:30 UTC (rev 172746)
+++ trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/Trac.js 2014-08-19 07:37:45 UTC (rev 172747)
</span><span class="lines">@@ -44,7 +44,7 @@
</span><span class="cx"> Trac.UpdateInterval = 45000; // 45 seconds
</span><span class="cx">
</span><span class="cx"> Trac.Event = {
</span><del>- NewCommitsRecorded: "new-commits-recorded"
</del><ins>+ CommitsUpdated: "commits-updated"
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> Trac.prototype = {
</span><span class="lines">@@ -63,9 +63,24 @@
</span><span class="cx"> return this.baseURL + "changeset/" + encodeURIComponent(revision);
</span><span class="cx"> },
</span><span class="cx">
</span><del>- _xmlTimelineURL: function()
</del><ins>+ _xmlTimelineURL: function(fromDate, toDate)
</ins><span class="cx"> {
</span><del>- return this.baseURL + "timeline?changeset=on&max=50&format=rss";
</del><ins>+ if (typeof fromDate === "undefined") {
+ fromDate = new Date();
+ toDate = new Date(fromDate);
+ // By default, get at least one full day of changesets, as the current day may have only begun.
+ fromDate.setDate(fromDate.getDate() - 1);
+ } else if (typeof toDate === "undefined")
+ toDate = fromDate;
+
+ console.assert(fromDate <= toDate);
+
+ var fromDay = new Date(fromDate.getFullYear(), fromDate.getMonth(), fromDate.getDate());
+ var toDay = new Date(toDate.getFullYear(), toDate.getMonth(), toDate.getDate());
+
+ return this.baseURL + "timeline?changeset=on&format=rss&max=-1" +
+ "&from=" + (toDay.getMonth() + 1) + "%2F" + toDay.getDate() + "%2F" + (toDay.getFullYear() % 100) +
+ "&daysback=" + ((toDay - fromDay) / 1000 / 60 / 60 / 24);
</ins><span class="cx"> },
</span><span class="cx">
</span><span class="cx"> _convertCommitInfoElementToObject: function(doc, commitElement)
</span><span class="lines">@@ -109,30 +124,63 @@
</span><span class="cx"> };
</span><span class="cx"> },
</span><span class="cx">
</span><del>- update: function()
</del><ins>+ _loaded: function(dataDocument)
</ins><span class="cx"> {
</span><del>- loadXML(this._xmlTimelineURL(), function(dataDocument) {
- var latestKnownRevision = 0;
- if (this.recordedCommits.length)
- latestKnownRevision = this.recordedCommits[this.recordedCommits.length - 1].revisionNumber;
</del><ins>+ var earliestKnownRevision = 0;
+ var latestKnownRevision = 0;
+ if (this.recordedCommits.length) {
+ earliestKnownRevision = this.recordedCommits[0].revisionNumber;
+ latestKnownRevision = this.recordedCommits[this.recordedCommits.length - 1].revisionNumber;
+ }
</ins><span class="cx">
</span><del>- var newCommits = [];
</del><ins>+ var knownCommitsWereUpdated = false;
+ var newCommits = [];
+ var newCommitsBeforeEarliestKnownRevision = [];
</ins><span class="cx">
</span><del>- var commitInfoElements = dataDocument.evaluate("/rss/channel/item", dataDocument, null, XPathResult.ORDERED_NODE_ITERATOR_TYPE);
- var commitInfoElement = undefined;
- while (commitInfoElement = commitInfoElements.iterateNext()) {
- var commit = this._convertCommitInfoElementToObject(dataDocument, commitInfoElement);
- if (commit.revisionNumber == latestKnownRevision)
- break;
</del><ins>+ var commitInfoElements = dataDocument.evaluate("/rss/channel/item", dataDocument, null, XPathResult.ORDERED_NODE_ITERATOR_TYPE);
+ var commitInfoElement = undefined;
+ var indexInRecordedCommits = undefined;
+ while (commitInfoElement = commitInfoElements.iterateNext()) {
+ var commit = this._convertCommitInfoElementToObject(dataDocument, commitInfoElement);
+ if (commit.revisionNumber > latestKnownRevision) {
+ console.assert(typeof indexInRecordedCommits === "undefined");
</ins><span class="cx"> newCommits.push(commit);
</span><ins>+ } else if (commit.revisionNumber < earliestKnownRevision) {
+ console.assert(typeof indexInRecordedCommits === "undefined" || indexInRecordedCommits === -1);
+ newCommitsBeforeEarliestKnownRevision.push(commit);
+ } else {
+ if (typeof indexInRecordedCommits === "undefined") {
+ // We could have started anywhere in the recorded commits array, let's find where.
+ // With periodic updates, this will be the latest recorded commit, so starting from the end.
+ for (var i = this.recordedCommits.length - 1; i >= 0; --i) {
+ if (this.recordedCommits[i].revisionNumber === commit.revisionNumber) {
+ indexInRecordedCommits = i;
+ break;
+ }
+ }
+ }
+
+ console.assert(indexInRecordedCommits >= 0);
+ console.assert(this.recordedCommits[indexInRecordedCommits].revisionNumber === commit.revisionNumber);
+
+ // Author could have changed, as commit queue replaces it after the fact.
+ if (this.recordedCommits[indexInRecordedCommits].author !== commit.author) {
+ this.recordedCommits[indexInRecordedCommits].author = commit.author;
+ knownCommitWasUpdated = true;
+ }
+ --indexInRecordedCommits;
</ins><span class="cx"> }
</span><del>-
- if (!newCommits.length)
- return;
</del><ins>+ }
</ins><span class="cx">
</span><del>- this.recordedCommits = this.recordedCommits.concat(newCommits.reverse());
</del><ins>+ if (newCommits.length || newCommitsBeforeEarliestKnownRevision.length)
+ this.recordedCommits = newCommitsBeforeEarliestKnownRevision.reverse().concat(this.recordedCommits, newCommits.reverse());
</ins><span class="cx">
</span><del>- this.dispatchEventToListeners(Trac.Event.NewCommitsRecorded, {newCommits: newCommits});
- }.bind(this), this._needsAuthentication ? { withCredentials: true } : {});
</del><ins>+ if (newCommits.length || newCommitsBeforeEarliestKnownRevision.length || knownCommitsWereUpdated)
+ this.dispatchEventToListeners(Trac.Event.CommitsUpdated, null);
+ },
+
+ update: function()
+ {
+ loadXML(this._xmlTimelineURL(), this._loaded.bind(this), this._needsAuthentication ? { withCredentials: true } : {});
</ins><span class="cx"> }
</span><span class="cx"> };
</span></span></pre></div>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (172746 => 172747)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog 2014-08-19 07:14:30 UTC (rev 172746)
+++ trunk/Tools/ChangeLog 2014-08-19 07:37:45 UTC (rev 172747)
</span><span class="lines">@@ -1,3 +1,35 @@
</span><ins>+2014-08-19 Alexey Proskuryakov <ap@apple.com>
+
+ build.webkit.org/dashboard should not request 50 revisions from trac each time
+ https://bugs.webkit.org/show_bug.cgi?id=127130
+
+ build.webkit.org/dashboard sometimes fetches a Trac revision in an intermediate state, and never updates later
+ https://bugs.webkit.org/show_bug.cgi?id=127131
+
+ Reviewed by Timothy Hatcher.
+
+ Turns out that requesting 50 builds is much slower than requesting by date - even
+ if the request ends up returning more than 50 results. There is no way to only
+ request updates, but this change brings request time from 6-8 seconds down to
+ less than a second.
+
+ This patch generalizes date handling for later use in metrics code. As part of the
+ rewrite, I made newly fetched data update author e-mail in previously fetched
+ revisions, as it changes after commit queue first lands.
+
+ * BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BuildbotQueueView.js:
+ Updated for event rename. I changed the trac event to not contain the list of new
+ commits, as we now sometimes update old commits, and that couldn't be expressed
+ in event data. We never used the list anywhere in the first place.
+
+ * BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/Trac.js:
+ (Trac.prototype._xmlTimelineURL): Made this function take arbitrary dates. When called
+ without arguments, return commits for today and yesterday.
+ (Trac.prototype._loaded):
+ (Trac.prototype.update): Moved the function for processing loaded results out of
+ here for clarity, and also because I'm going to have a separate loading code path
+ for metrics.
+
</ins><span class="cx"> 2014-08-18 Dan Bernstein <mitz@apple.com>
</span><span class="cx">
</span><span class="cx"> Added an update-webkit option to override the ../Internal check.
</span></span></pre>
</div>
</div>
</body>
</html>