<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[190764] trunk/Websites/perf.webkit.org</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/190764">190764</a></dd>
<dt>Author</dt> <dd>rniwa@webkit.org</dd>
<dt>Date</dt> <dd>2015-10-08 18:13:26 -0700 (Thu, 08 Oct 2015)</dd>
</dl>
<h3>Log Message</h3>
<pre>pull-svn.py fails to sync revisions when SVN credentials is not setup
https://bugs.webkit.org/show_bug.cgi?id=149941
Reviewed by Chris Dumez.
Added the support for specifying subversion credentials.
Also added the support for pulling from multiple subversion servers. Subversion servers are specified
in a JSON configuration file specified by --svn-config formatted as follows:
[
{
"name": "WebKit",
"url": "http://svn.webkit.org/repository/webkit",
"username": "webkitten",
"password": "webkitten's password",
"trustCertificate": true,
"accountNameFinderScript":
["python", "/Volumes/Data/WebKit/Tools/Scripts/webkit-patch", "find-users"]
},
...
]
In addition, refactored it to use the shared server config JSON for the dashboard access.
* tools/pull-svn.py:
(main): Now takes --svn-config-json, --server-config-json, --seconds-to-sleep and --max-fetch-count
as required options instead of seven unnamed arguments.
(fetch_commits_and_submit): Extracted from main. Fetches at most max_fetch_count new revisions from
the subversion server, and submits them in accordance with server_config.
(fetch_commit_and_resolve_author): Now takes a single repository dictionary instead of two separate
arguments for name and URL to pass down the repository's authentication info to fetch_commit.
(fetch_commit): Ditto. Add appropriate arguments when username and passwords are specified.
(resolve_author_name_from_account): Use a list argument instead of a single string argument now that
the argument comes from a JSON instead of sys.argv.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkWebsitesperfwebkitorgChangeLog">trunk/Websites/perf.webkit.org/ChangeLog</a></li>
<li><a href="#trunkWebsitesperfwebkitorgtoolspullsvnpy">trunk/Websites/perf.webkit.org/tools/pull-svn.py</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkWebsitesperfwebkitorgChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/ChangeLog (190763 => 190764)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/ChangeLog 2015-10-09 00:47:47 UTC (rev 190763)
+++ trunk/Websites/perf.webkit.org/ChangeLog 2015-10-09 01:13:26 UTC (rev 190764)
</span><span class="lines">@@ -1,3 +1,41 @@
</span><ins>+2015-10-08 Ryosuke Niwa <rniwa@webkit.org>
+
+ pull-svn.py fails to sync revisions when SVN credentials is not setup
+ https://bugs.webkit.org/show_bug.cgi?id=149941
+
+ Reviewed by Chris Dumez.
+
+ Added the support for specifying subversion credentials.
+
+ Also added the support for pulling from multiple subversion servers. Subversion servers are specified
+ in a JSON configuration file specified by --svn-config formatted as follows:
+
+ [
+ {
+ "name": "WebKit",
+ "url": "http://svn.webkit.org/repository/webkit",
+ "username": "webkitten",
+ "password": "webkitten's password",
+ "trustCertificate": true,
+ "accountNameFinderScript":
+ ["python", "/Volumes/Data/WebKit/Tools/Scripts/webkit-patch", "find-users"]
+ },
+ ...
+ ]
+
+ In addition, refactored it to use the shared server config JSON for the dashboard access.
+
+ * tools/pull-svn.py:
+ (main): Now takes --svn-config-json, --server-config-json, --seconds-to-sleep and --max-fetch-count
+ as required options instead of seven unnamed arguments.
+ (fetch_commits_and_submit): Extracted from main. Fetches at most max_fetch_count new revisions from
+ the subversion server, and submits them in accordance with server_config.
+ (fetch_commit_and_resolve_author): Now takes a single repository dictionary instead of two separate
+ arguments for name and URL to pass down the repository's authentication info to fetch_commit.
+ (fetch_commit): Ditto. Add appropriate arguments when username and passwords are specified.
+ (resolve_author_name_from_account): Use a list argument instead of a single string argument now that
+ the argument comes from a JSON instead of sys.argv.
+
</ins><span class="cx"> 2015-10-07 Ryosuke Niwa <rniwa@webkit.org>
</span><span class="cx">
</span><span class="cx"> Unreviewed race condition fix. Exit early when xScale or yScale is not defined.
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtoolspullsvnpy"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/tools/pull-svn.py (190763 => 190764)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tools/pull-svn.py 2015-10-09 00:47:47 UTC (rev 190763)
+++ trunk/Websites/perf.webkit.org/tools/pull-svn.py 2015-10-09 01:13:26 UTC (rev 190764)
</span><span class="lines">@@ -1,5 +1,6 @@
</span><span class="cx"> #!/usr/bin/python
</span><span class="cx">
</span><ins>+import argparse
</ins><span class="cx"> import json
</span><span class="cx"> import re
</span><span class="cx"> import subprocess
</span><span class="lines">@@ -8,48 +9,60 @@
</span><span class="cx"> import urllib2
</span><span class="cx">
</span><span class="cx"> from xml.dom.minidom import parseString as parseXmlString
</span><ins>+from util import setup_auth
</ins><span class="cx"> from util import submit_commits
</span><span class="cx"> from util import text_content
</span><span class="cx">
</span><span class="cx">
</span><span class="cx"> def main(argv):
</span><del>- if len(argv) < 7:
- sys.exit('Usage: pull-svn <repository-name> <repository-URL> <dashboard-URL> <slave-name> <slave-password> <seconds-to-sleep> [<account-to-name-helper>]')
</del><ins>+ parser = argparse.ArgumentParser()
+ parser.add_argument('--svn-config-json', required=True, help='The path to a JSON file that specifies subversion syncing options')
+ parser.add_argument('--server-config-json', required=True, help='The path to a JSON file that specifies the perf dashboard')
+ parser.add_argument('--seconds-to-sleep', type=float, default=900, help='The seconds to sleep between iterations')
+ parser.add_argument('--max-fetch-count', type=int, default=10, help='The number of commits to fetch at once')
+ args = parser.parse_args()
</ins><span class="cx">
</span><del>- repository_name = argv[1]
- repository_url = argv[2]
- dashboard_url = argv[3]
- slave_name = argv[4]
- slave_password = argv[5]
- seconds_to_sleep = float(argv[6])
- account_to_name_helper = argv[7] if len(argv) > 7 else None
</del><ins>+ with open(args.server_config_json) as server_config_json:
+ server_config = json.load(server_config_json)
+ setup_auth(server_config['server'])
</ins><span class="cx">
</span><del>- print "Submitting revision logs for %s at %s to %s" % (repository_name, repository_url, dashboard_url)
</del><ins>+ with open(args.svn_config_json) as svn_config_json:
+ svn_config = json.load(svn_config_json)
</ins><span class="cx">
</span><del>- revision_to_fetch = determine_first_revision_to_fetch(dashboard_url, repository_name)
- print "Start fetching commits at r%d" % revision_to_fetch
</del><ins>+ while True:
+ for repository_info in svn_config:
+ fetch_commits_and_submit(repository_info, server_config, args.max_fetch_count)
+ print "Sleeping for %d seconds..." % args.seconds_to_sleep
+ time.sleep(args.seconds_to_sleep)
</ins><span class="cx">
</span><del>- pending_commits_to_send = []
</del><span class="cx">
</span><del>- while True:
- commit = fetch_commit_and_resolve_author(repository_name, repository_url, account_to_name_helper, revision_to_fetch)
</del><ins>+def fetch_commits_and_submit(repository, server_config, max_fetch_count):
+ assert 'name' in repository, 'The repository name should be specified'
+ assert 'url' in repository, 'The SVN repository URL should be specified'
</ins><span class="cx">
</span><del>- if commit:
- print "Fetched r%d." % revision_to_fetch
- pending_commits_to_send += [commit]
- revision_to_fetch += 1
- else:
- print "Revision %d not found" % revision_to_fetch
</del><ins>+ if 'revisionToFetch' not in repository:
+ print "Determining the stating revision for %s" % repository['name']
+ repository['revisionToFecth'] = determine_first_revision_to_fetch(server_config['server']['url'], repository['name'])
</ins><span class="cx">
</span><del>- if not commit or len(pending_commits_to_send) >= 10:
- if pending_commits_to_send:
- print "Submitting the above commits to %s..." % dashboard_url
- submit_commits(pending_commits_to_send, dashboard_url, slave_name, slave_password)
- print "Successfully submitted."
- pending_commits_to_send = []
- time.sleep(seconds_to_sleep)
</del><ins>+ pending_commits = []
+ for unused in range(max_fetch_count):
+ commit = fetch_commit_and_resolve_author(repository, repository.get('accountNameFinderScript', None), repository['revisionToFecth'])
+ if not commit:
+ break
+ pending_commits += [commit]
+ repository['revisionToFecth'] += 1
</ins><span class="cx">
</span><ins>+ if not pending_commits:
+ print "No new revision found for %s (waiting for r%d)" % (repository['name'], repository['revisionToFecth'])
+ return
</ins><span class="cx">
</span><ins>+ revision_list = 'r' + ', r'.join(map(lambda commit: str(commit['revision']), pending_commits))
+ print "Submitting revisions %s for %s to %s" % (revision_list, repository['name'], server_config['server']['url'])
+ submit_commits(pending_commits, server_config['server']['url'], server_config['slave']['name'], server_config['slave']['password'])
+ print "Successfully submitted."
+ print
+
+
</ins><span class="cx"> def determine_first_revision_to_fetch(dashboard_url, repository_name):
</span><span class="cx"> try:
</span><span class="cx"> last_reported_revision = fetch_revision_from_dasbhoard(dashboard_url, repository_name, 'last-reported')
</span><span class="lines">@@ -76,9 +89,9 @@
</span><span class="cx"> return int(commits[0]['revision']) if commits else None
</span><span class="cx">
</span><span class="cx">
</span><del>-def fetch_commit_and_resolve_author(repository_name, repository_url, account_to_name_helper, revision_to_fetch):
</del><ins>+def fetch_commit_and_resolve_author(repository, account_to_name_helper, revision_to_fetch):
</ins><span class="cx"> try:
</span><del>- commit = fetch_commit(repository_name, repository_url, revision_to_fetch)
</del><ins>+ commit = fetch_commit(repository, revision_to_fetch)
</ins><span class="cx"> except Exception as error:
</span><span class="cx"> sys.exit('Failed to fetch the commit %d: %s' % (revision_to_fetch, str(error)))
</span><span class="cx">
</span><span class="lines">@@ -96,8 +109,13 @@
</span><span class="cx"> return commit
</span><span class="cx">
</span><span class="cx">
</span><del>-def fetch_commit(repository_name, repository_url, revision):
- args = ['svn', 'log', '--revision', str(revision), '--xml', repository_url]
</del><ins>+def fetch_commit(repository, revision):
+ args = ['svn', 'log', '--revision', str(revision), '--xml', repository['url'], '--non-interactive']
+ if 'username' in repository and 'password' in repository:
+ args += ['--no-auth-cache', '--username', repository['username'], '--password', repository['password']]
+ if repository.get('trustCertificate', False):
+ args += ['--trust-server-cert']
+
</ins><span class="cx"> try:
</span><span class="cx"> output = subprocess.check_output(args, stderr=subprocess.STDOUT)
</span><span class="cx"> except subprocess.CalledProcessError as error:
</span><span class="lines">@@ -109,7 +127,7 @@
</span><span class="cx"> author_account = text_content(xml.getElementsByTagName("author")[0])
</span><span class="cx"> message = text_content(xml.getElementsByTagName("msg")[0])
</span><span class="cx"> return {
</span><del>- 'repository': repository_name,
</del><ins>+ 'repository': repository['name'],
</ins><span class="cx"> 'revision': revision,
</span><span class="cx"> 'time': time,
</span><span class="cx"> 'author': {'account': author_account},
</span><span class="lines">@@ -121,7 +139,7 @@
</span><span class="cx">
</span><span class="cx">
</span><span class="cx"> def resolve_author_name_from_account(helper, account):
</span><del>- output = subprocess.check_output(helper + ' ' + account, shell=True)
</del><ins>+ output = subprocess.check_output(helper + [account])
</ins><span class="cx"> match = name_account_compound_regex.match(output)
</span><span class="cx"> if match:
</span><span class="cx"> return match.group('name').strip('"')
</span></span></pre>
</div>
</div>
</body>
</html>