<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[265382] trunk/Tools</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/265382">265382</a></dd>
<dt>Author</dt> <dd>jbedard@apple.com</dd>
<dt>Date</dt> <dd>2020-08-07 11:06:26 -0700 (Fri, 07 Aug 2020)</dd>
</dl>

<h3>Log Message</h3>
<pre>[webkitcorepy] Support wheels in the autoinstaller
https://bugs.webkit.org/show_bug.cgi?id=215230
<rdar://problem/66636527>

Reviewed by Stephanie Lewis.

* Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py: Bump version, include packaging and it's dependencies.
* Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py:
(Package.Archive.unpack): .whl files are zip files.
(Package.__init__): Allow programs to request packages as wheels.
(Package.archives): Search for compatible .whl files.
(Package.install): Ensure that .egg-info directories are appropriately chowned, unpack wheels, format manifest.json.
(AutoInstall.tags): List the packaging tags associated with this distribution, but with a work-around for Big Sur.</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
<li><a href="#trunkToolsScriptslibrarieswebkitcorepywebkitcorepy__init__py">trunk/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py</a></li>
<li><a href="#trunkToolsScriptslibrarieswebkitcorepywebkitcorepyautoinstallpy">trunk/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (265381 => 265382)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog    2020-08-07 17:52:56 UTC (rev 265381)
+++ trunk/Tools/ChangeLog       2020-08-07 18:06:26 UTC (rev 265382)
</span><span class="lines">@@ -1,3 +1,19 @@
</span><ins>+2020-08-07  Jonathan Bedard  <jbedard@apple.com>
+
+        [webkitcorepy] Support wheels in the autoinstaller
+        https://bugs.webkit.org/show_bug.cgi?id=215230
+        <rdar://problem/66636527>
+
+        Reviewed by Stephanie Lewis.
+
+        * Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py: Bump version, include packaging and it's dependencies.
+        * Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py:
+        (Package.Archive.unpack): .whl files are zip files.
+        (Package.__init__): Allow programs to request packages as wheels.
+        (Package.archives): Search for compatible .whl files.
+        (Package.install): Ensure that .egg-info directories are appropriately chowned, unpack wheels, format manifest.json.
+        (AutoInstall.tags): List the packaging tags associated with this distribution, but with a work-around for Big Sur.
+
</ins><span class="cx"> 2020-08-07  Aakash Jain  <aakash_jain@apple.com>
</span><span class="cx"> 
</span><span class="cx">         Do not email bot watcher when first run of layout tests exceeds failure limit
</span></span></pre></div>
<a id="trunkToolsScriptslibrarieswebkitcorepywebkitcorepy__init__py"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py (265381 => 265382)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py      2020-08-07 17:52:56 UTC (rev 265381)
+++ trunk/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py 2020-08-07 18:06:26 UTC (rev 265382)
</span><span class="lines">@@ -29,7 +29,7 @@
</span><span class="cx"> from webkitcorepy.version import Version
</span><span class="cx"> from webkitcorepy.string_utils import BytesIO, StringIO, UnicodeIO, unicode
</span><span class="cx"> 
</span><del>-version = Version(0, 1, 2)
</del><ins>+version = Version(0, 2, 0)
</ins><span class="cx"> 
</span><span class="cx"> from webkitcorepy.autoinstall import Package, AutoInstall
</span><span class="cx"> if sys.version_info > (3, 0):
</span><span class="lines">@@ -43,7 +43,10 @@
</span><span class="cx"> AutoInstall.register(Package('chardet', Version(3, 0, 4)))
</span><span class="cx"> AutoInstall.register(Package('funcsigs', Version(1, 0, 2)))
</span><span class="cx"> AutoInstall.register(Package('idna', Version(2, 10)))
</span><ins>+AutoInstall.register(Package('packaging', Version(20, 4)))
+AutoInstall.register(Package('pyparsing', Version(2, 4, 7)))
</ins><span class="cx"> AutoInstall.register(Package('requests', Version(2, 24)))
</span><span class="cx"> AutoInstall.register(Package('setuptools', Version(44, 1,  1)))
</span><span class="cx"> AutoInstall.register(Package('socks', Version(1, 7, 1), pypi_name='PySocks'))
</span><ins>+AutoInstall.register(Package('six', Version(1, 15, 0)))
</ins><span class="cx"> AutoInstall.register(Package('urllib3', Version(1, 25, 10)))
</span></span></pre></div>
<a id="trunkToolsScriptslibrarieswebkitcorepywebkitcorepyautoinstallpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py (265381 => 265382)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py   2020-08-07 17:52:56 UTC (rev 265381)
+++ trunk/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py      2020-08-07 18:06:26 UTC (rev 265382)
</span><span class="lines">@@ -23,6 +23,7 @@
</span><span class="cx"> import json
</span><span class="cx"> import math
</span><span class="cx"> import os
</span><ins>+import platform
</ins><span class="cx"> import re
</span><span class="cx"> import subprocess
</span><span class="cx"> import shutil
</span><span class="lines">@@ -82,18 +83,19 @@
</span><span class="cx">                     file.extractall(target)
</span><span class="cx">                 finally:
</span><span class="cx">                     file.close()
</span><del>-            elif self.extension == 'zip':
</del><ins>+            elif self.extension in ['whl', 'zip']:
</ins><span class="cx">                 with zipfile.ZipFile(self.path, 'r') as file:
</span><span class="cx">                     file.extractall(target)
</span><span class="cx">             else:
</span><span class="cx">                 raise OSError('{} has an  unrecognized package format'.format(self.path))
</span><span class="cx"> 
</span><del>-    def __init__(self, name, version=None, pypi_name=None, slow_install=False):
</del><ins>+    def __init__(self, name, version=None, pypi_name=None, slow_install=False, wheel=False):
</ins><span class="cx">         self.name = name
</span><span class="cx">         self.version = version
</span><span class="cx">         self._archives = []
</span><span class="cx">         self.pypi_name = pypi_name or self.name
</span><span class="cx">         self.slow_install = slow_install
</span><ins>+        self.wheel = wheel
</ins><span class="cx"> 
</span><span class="cx">     @property
</span><span class="cx">     def location(self):
</span><span class="lines">@@ -115,7 +117,9 @@
</span><span class="cx">                 raise ValueError('The package {} was not found on {}'.format(self.pypi_name, AutoInstall.index))
</span><span class="cx"> 
</span><span class="cx">             page = minidom.parseString(response.read())
</span><del>-            for element in page.getElementsByTagName("a"):
</del><ins>+            cached_tags = None
+
+            for element in reversed(page.getElementsByTagName("a")):
</ins><span class="cx">                 if not len(element.childNodes):
</span><span class="cx">                     continue
</span><span class="cx">                 if element.childNodes[0].nodeType != minidom.Node.TEXT_NODE:
</span><span class="lines">@@ -127,12 +131,28 @@
</span><span class="cx">                 if not attributes.get('href', None):
</span><span class="cx">                     continue
</span><span class="cx"> 
</span><del>-                if element.childNodes[0].data.endswith('tar.gz'):
-                    extension = 'tar.gz'
-                elif element.childNodes[0].data.endswith('.zip'):
-                    extension = 'zip'
</del><ins>+                if self.wheel:
+                    match = re.search(r'.+-([^-]+-[^-]+-[^-]+).whl', element.childNodes[0].data)
+                    if not match:
+                        continue
+
+                    from packaging import tags
+
+                    if not cached_tags:
+                        cached_tags = set(AutoInstall.tags())
+
+                    if all([tag not in cached_tags for tag in tags.parse_tag(match.group(1))]):
+                        continue
+
+                    extension = 'whl'
+
</ins><span class="cx">                 else:
</span><del>-                    continue
</del><ins>+                    if element.childNodes[0].data.endswith('.tar.gz'):
+                        extension = 'tar.gz'
+                    elif element.childNodes[0].data.endswith('.zip'):
+                        extension = 'zip'
+                    else:
+                        continue
</ins><span class="cx"> 
</span><span class="cx">                 requires = attributes.get('data-requires-python')
</span><span class="cx">                 if requires and not AutoInstall.version.matches(requires):
</span><span class="lines">@@ -191,7 +211,6 @@
</span><span class="cx">         try:
</span><span class="cx">             install_location = os.path.dirname(self.location)
</span><span class="cx">             shutil.rmtree(self.location, ignore_errors=True)
</span><del>-            already_owned = set(os.listdir(install_location))
</del><span class="cx"> 
</span><span class="cx">             log.warning('Installing {}...'.format(archive))
</span><span class="cx">             archive.download()
</span><span class="lines">@@ -203,6 +222,7 @@
</span><span class="cx">                 os.path.join(temp_location, str(archive)),
</span><span class="cx">                 os.path.join(temp_location, '{}-{}.{}'.format(archive.name, archive.version.major, archive.version.minor)),
</span><span class="cx">                 os.path.join(temp_location, '{}-{}.{}.{}'.format(archive.name.replace('-', '_'), archive.version.major, archive.version.minor, archive.version.tiny)),
</span><ins>+                os.path.join(temp_location, '{}-{}'.format(archive.name.capitalize(), archive.version)),
</ins><span class="cx">             ]:
</span><span class="cx">                 if not os.path.exists(os.path.join(candidate, 'setup.py')):
</span><span class="cx">                     continue
</span><span class="lines">@@ -222,7 +242,7 @@
</span><span class="cx">                             '--root=/',
</span><span class="cx">                             '--single-version-externally-managed',
</span><span class="cx">                             '--install-lib={}'.format(install_location),
</span><del>-                            '--install-scripts={}'.format(install_location),
</del><ins>+                            '--install-scripts={}'.format(os.path.join(install_location, 'bin')),
</ins><span class="cx">                             '--install-data={}'.format(os.path.join(install_location, 'data')),
</span><span class="cx">                             '--install-headers={}'.format(os.path.join(install_location, 'headers')),
</span><span class="cx">                             # Do not automatically install package dependencies, force scripts to be explicit
</span><span class="lines">@@ -242,7 +262,13 @@
</span><span class="cx"> 
</span><span class="cx">                 break
</span><span class="cx">             else:
</span><del>-                raise OSError('Cannot install {}, could not find setup.py'.format(self.name))
</del><ins>+                # We might not need setup.py at all, check if we have dist-info and the library in the temporary location
+                to_be_moved = os.listdir(temp_location)
+                if self.name not in to_be_moved and any(element.endswith('.dist-info') for element in to_be_moved):
+                    raise OSError('Cannot install {}, could not find setup.py'.format(self.name))
+                for directory in to_be_moved:
+                    shutil.rmtree(os.path.join(install_location, directory), ignore_errors=True)
+                    shutil.move(os.path.join(temp_location, directory), install_location)
</ins><span class="cx"> 
</span><span class="cx">             self.do_post_install(temp_location)
</span><span class="cx"> 
</span><span class="lines">@@ -249,8 +275,7 @@
</span><span class="cx">             os.remove(archive.path)
</span><span class="cx">             shutil.rmtree(temp_location, ignore_errors=True)
</span><span class="cx"> 
</span><del>-            for installed in set(os.listdir(install_location)) - already_owned:
-                AutoInstall.userspace_should_own(os.path.join(install_location, installed))
</del><ins>+            AutoInstall.userspace_should_own(install_location)
</ins><span class="cx"> 
</span><span class="cx">             AutoInstall.manifest[self.name] = {
</span><span class="cx">                 'index': AutoInstall.index,
</span><span class="lines">@@ -259,7 +284,7 @@
</span><span class="cx"> 
</span><span class="cx">             manifest = os.path.join(AutoInstall.directory, 'manifest.json')
</span><span class="cx">             with open(manifest, 'w') as file:
</span><del>-                json.dump(AutoInstall.manifest, file)
</del><ins>+                json.dump(AutoInstall.manifest, file, indent=4)
</ins><span class="cx">             AutoInstall.userspace_should_own(manifest)
</span><span class="cx"> 
</span><span class="cx">             log.warning('Installed {}!'.format(archive))
</span><span class="lines">@@ -406,5 +431,20 @@
</span><span class="cx">         if cls.packages.get(name):
</span><span class="cx">             cls.install(name)
</span><span class="cx"> 
</span><ins>+    @classmethod
+    def tags(cls):
+        from packaging import tags
</ins><span class="cx"> 
</span><ins>+        for tag in tags.sys_tags():
+            yield tag
+
+        # FIXME: Work around for https://github.com/pypa/packaging/pull/319 and Big Sur
+        if sys.platform == 'darwin' and Version.from_string(platform.mac_ver()[0]) > Version(10):
+            for override in tags.mac_platforms(version=(10, 16)):
+                for tag in tags.sys_tags():
+                    if not tag.platform:
+                        pass
+                    yield tags.Tag(tag.interpreter, tag.abi, override)
+
+
</ins><span class="cx"> sys.meta_path.insert(0, AutoInstall)
</span></span></pre>
</div>
</div>

</body>
</html>