<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[163570] trunk/Tools</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/163570">163570</a></dd>
<dt>Author</dt> <dd>commit-queue@webkit.org</dd>
<dt>Date</dt> <dd>2014-02-06 16:03:15 -0800 (Thu, 06 Feb 2014)</dd>
</dl>
<h3>Log Message</h3>
<pre>Add support for multiple sources for AutoInstaller
https://bugs.webkit.org/show_bug.cgi?id=124848
Patch by Jozsef Berta <jberta.u-szeged@partner.samsung.com> on 2014-02-06
Reviewed by Ryosuke Niwa.
The autoinstaller in the webkitpy currently fails if the download source of a package is unavailable.
This patch adds support for multiple sources to the script. The sources are provided in three environment variables.
If it exists, the script will look at a local cache. If not, it will try to download the package from the original url.
If it fails, it gets a mirror from the corresponding environment variable.(One for sourceforge.org and one for pypi.python.org)
* Scripts/webkitpy/common/system/autoinstall.py:
(AutoInstaller._copy_unpackaged_files_from_local_cache): If the package is not packaged in its original form,
this method will copy it to the scratch directory. Otherwise it would be deleted from the cache, which we do not want.
(AutoInstaller._prepare_package): If the package is not zipped or tarred, and the file is in the cache, the
_copy_unpackaged_files_from_local_cache function is called.
(AutoInstaller._parse_colon_separated_mirrors_from_env): This will read the mirrors from the environment variables if possible,
and prepares them for further use.
(AutoInstaller):
(AutoInstaller._replace_domain_with_next_mirror): If the original download url fails, it is replaced by a mirror provided
in the environment variables. The function identifies the original source, and replaces it with a mirror. If it can't be done,
the return url will be None, indicating that no mirrors are provided, or none of them could be reached.
(AutoInstaller._download_to_stream): The timeout for one try is now limited to 30 seconds. Without this, the script waited for
roughly 4 minutes before retrying. After three failiures the script will try to switch to a mirror.
(AutoInstaller._check_package_in_local_autoinstall_cache): This method searches the cache for the currently downloaded module.
If it's found there, its path is returned.
(AutoInstaller._download): Before downloading the module, it is looked up in the cache. If it's not found there,
the script will continue with the download, and cache the module.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
<li><a href="#trunkToolsScriptswebkitpycommonsystemautoinstallpy">trunk/Tools/Scripts/webkitpy/common/system/autoinstall.py</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (163569 => 163570)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog 2014-02-06 23:45:57 UTC (rev 163569)
+++ trunk/Tools/ChangeLog 2014-02-07 00:03:15 UTC (rev 163570)
</span><span class="lines">@@ -1,3 +1,33 @@
</span><ins>+2014-02-06 Jozsef Berta <jberta.u-szeged@partner.samsung.com>
+
+ Add support for multiple sources for AutoInstaller
+ https://bugs.webkit.org/show_bug.cgi?id=124848
+
+ Reviewed by Ryosuke Niwa.
+
+ The autoinstaller in the webkitpy currently fails if the download source of a package is unavailable.
+ This patch adds support for multiple sources to the script. The sources are provided in three environment variables.
+ If it exists, the script will look at a local cache. If not, it will try to download the package from the original url.
+ If it fails, it gets a mirror from the corresponding environment variable.(One for sourceforge.org and one for pypi.python.org)
+
+ * Scripts/webkitpy/common/system/autoinstall.py:
+ (AutoInstaller._copy_unpackaged_files_from_local_cache): If the package is not packaged in its original form,
+ this method will copy it to the scratch directory. Otherwise it would be deleted from the cache, which we do not want.
+ (AutoInstaller._prepare_package): If the package is not zipped or tarred, and the file is in the cache, the
+ _copy_unpackaged_files_from_local_cache function is called.
+ (AutoInstaller._parse_colon_separated_mirrors_from_env): This will read the mirrors from the environment variables if possible,
+ and prepares them for further use.
+ (AutoInstaller):
+ (AutoInstaller._replace_domain_with_next_mirror): If the original download url fails, it is replaced by a mirror provided
+ in the environment variables. The function identifies the original source, and replaces it with a mirror. If it can't be done,
+ the return url will be None, indicating that no mirrors are provided, or none of them could be reached.
+ (AutoInstaller._download_to_stream): The timeout for one try is now limited to 30 seconds. Without this, the script waited for
+ roughly 4 minutes before retrying. After three failiures the script will try to switch to a mirror.
+ (AutoInstaller._check_package_in_local_autoinstall_cache): This method searches the cache for the currently downloaded module.
+ If it's found there, its path is returned.
+ (AutoInstaller._download): Before downloading the module, it is looked up in the cache. If it's not found there,
+ the script will continue with the download, and cache the module.
+
</ins><span class="cx"> 2014-02-06 Benjamin Poulain <benjamin@webkit.org>
</span><span class="cx">
</span><span class="cx"> Remove run-test-webkit-api
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpycommonsystemautoinstallpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/common/system/autoinstall.py (163569 => 163570)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/common/system/autoinstall.py 2014-02-06 23:45:57 UTC (rev 163569)
+++ trunk/Tools/Scripts/webkitpy/common/system/autoinstall.py 2014-02-07 00:03:15 UTC (rev 163570)
</span><span class="lines">@@ -42,10 +42,17 @@
</span><span class="cx"> import urllib2
</span><span class="cx"> import urlparse
</span><span class="cx"> import zipfile
</span><ins>+import re
+from distutils import dir_util
+from glob import glob
+import urlparse
</ins><span class="cx">
</span><span class="cx"> _log = logging.getLogger(__name__)
</span><ins>+_MIRROR_REGEXS = re.compile('.*sourceforge.*'), re.compile('.*pypi.*')
+_PYPI_ENV_VAR = 'PYPI_MIRRORS'
+_SOURCEFORGE_ENV_VAR = 'SOURCEFORGE_MIRRORS'
+_CACHE_ENV_VAR = 'LOCAL_AUTOINSTALL_CACHE'
</ins><span class="cx">
</span><del>-
</del><span class="cx"> class AutoInstaller(object):
</span><span class="cx">
</span><span class="cx"> """Supports automatically installing Python packages from an URL.
</span><span class="lines">@@ -259,6 +266,14 @@
</span><span class="cx">
</span><span class="cx"> return target_path
</span><span class="cx">
</span><ins>+ def _copy_unpackaged_files_from_local_cache(self, path, scratch_dir):
+
+ target_basename = os.path.basename(path)
+ target_path = os.path.join(scratch_dir, target_basename)
+
+ shutil.copy(path, target_path)
+ return target_path
+
</ins><span class="cx"> def _prepare_package(self, path, scratch_dir):
</span><span class="cx"> """Prepare a package for use, if necessary, and return the new path.
</span><span class="cx">
</span><span class="lines">@@ -277,26 +292,68 @@
</span><span class="cx"> new_path = self._unzip(path, scratch_dir)
</span><span class="cx"> elif path.endswith(".tar.gz"):
</span><span class="cx"> new_path = self._extract_targz(path, scratch_dir)
</span><ins>+ elif _CACHE_ENV_VAR in os.environ:
+ new_path = path
+ if os.path.dirname(path) == os.path.normpath(os.environ[_CACHE_ENV_VAR]):
+ new_path = self._copy_unpackaged_files_from_local_cache(path, scratch_dir)
</ins><span class="cx"> else:
</span><span class="cx"> # No preparation is needed.
</span><span class="cx"> new_path = path
</span><span class="cx">
</span><span class="cx"> return new_path
</span><span class="cx">
</span><ins>+ def _parse_colon_separated_mirrors_from_env(self):
+ """
+ Pypi mirror examle: PYPI_MIRRORS=pypi.hustunique.com...
+ Sourceforge mirror example: SOURCEFORGE_MIRRORS=aarnet.dl.sourceforge.net:citylan.dl.sourceforge.net...
+ Mirror sources: http://www.pypi-mirrors.org/, http://sourceforge.net/apps/trac/sourceforge/wiki/Mirrors
+ """
+ try:
+ pypi_mirrors_list = os.environ[_PYPI_ENV_VAR].split(':')
+ except(KeyError):
+ pypi_mirrors_list = ()
+
+ try:
+ sourceforge_mirrors_list = os.environ[_SOURCEFORGE_ENV_VAR].split(':')
+ except(KeyError):
+ sourceforge_mirrors_list = ()
+
+ mirroriterators = iter(sourceforge_mirrors_list), iter(pypi_mirrors_list)
+ return zip(_MIRROR_REGEXS, mirroriterators)
+
+ def _replace_domain_with_next_mirror(self, url, mirrors):
+ parsed_url = list(urlparse.urlparse(url))
+ new_url = None
+ try:
+ for regex, addresses in mirrors:
+ if regex.match(parsed_url[1]):
+ parsed_url[1] = addresses.next()
+ new_url = urlparse.urlunparse(parsed_url)
+ except StopIteration, e:
+ _log.info('Ran out of mirrors.')
+
+ return new_url
+
</ins><span class="cx"> def _download_to_stream(self, url, stream):
</span><ins>+ mirrors = self._parse_colon_separated_mirrors_from_env()
</ins><span class="cx"> failures = 0
</span><span class="cx"> while True:
</span><span class="cx"> try:
</span><del>- netstream = urllib2.urlopen(url)
</del><ins>+ netstream = urllib2.urlopen(url, timeout=30)
</ins><span class="cx"> break
</span><span class="cx"> except IOError, err:
</span><span class="cx"> # Try multiple times
</span><del>- if failures < 5:
</del><ins>+ if failures < 2:
</ins><span class="cx"> _log.warning("Failed to download %s, %s retrying" % (
</span><span class="cx"> url, err))
</span><span class="cx"> failures += 1
</span><span class="cx"> continue
</span><span class="cx">
</span><ins>+ url = self._replace_domain_with_next_mirror(url, mirrors)
+ if url:
+ failures = 0
+ continue
+
</ins><span class="cx"> # Append existing Error message to new Error.
</span><span class="cx"> message = ('Could not download Python modules from URL "%s".\n'
</span><span class="cx"> " Make sure you are connected to the internet.\n"
</span><span class="lines">@@ -319,15 +376,31 @@
</span><span class="cx"> stream.write(data)
</span><span class="cx"> netstream.close()
</span><span class="cx">
</span><ins>+ def _check_package_in_local_autoinstall_cache(self, filename):
+ if _CACHE_ENV_VAR not in os.environ:
+ return False
+ path = glob(os.path.join(os.environ[_CACHE_ENV_VAR], filename) + '*')
+ if not path:
+ return False
+
+ return path[0]
+
</ins><span class="cx"> def _download(self, url, scratch_dir):
</span><span class="cx"> url_path = urlparse.urlsplit(url)[2]
</span><span class="cx"> url_path = os.path.normpath(url_path) # Removes trailing slash.
</span><span class="cx"> target_filename = os.path.basename(url_path)
</span><ins>+
+ cache = self._check_package_in_local_autoinstall_cache(target_filename)
+ if cache:
+ return cache
+
</ins><span class="cx"> target_path = os.path.join(scratch_dir, target_filename)
</span><del>-
</del><span class="cx"> with open(target_path, "wb") as stream:
</span><span class="cx"> self._download_to_stream(url, stream)
</span><span class="cx">
</span><ins>+ if _CACHE_ENV_VAR in os.environ:
+ dir_util.copy_tree(scratch_dir, os.environ[_CACHE_ENV_VAR])
+
</ins><span class="cx"> return target_path
</span><span class="cx">
</span><span class="cx"> def _install(self, scratch_dir, package_name, target_path, url, url_subpath, files_to_remove):
</span></span></pre>
</div>
</div>
</body>
</html>