<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[173498] trunk/Tools</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/173498">173498</a></dd>
<dt>Author</dt> <dd>commit-queue@webkit.org</dd>
<dt>Date</dt> <dd>2014-09-10 17:01:47 -0700 (Wed, 10 Sep 2014)</dd>
</dl>
<h3>Log Message</h3>
<pre>import-w3c-tests doesn't handle relative paths to support files in ref files correctly
https://bugs.webkit.org/show_bug.cgi?id=135929
Patch by Rebecca Hauck <rhauck@adobe.com> on 2014-09-10
Reviewed by Bem Jones-Bey.
The recent refactor of the W3C test repo falsified a bunch of assmumptions that
were made when this script was originally written with respect to relative paths
in ref files. This patch updates import-w3c-tests to update paths in ref files if
they move relative to the test file.
* Scripts/webkitpy/w3c/test_converter.py:
(convert_for_webkit):
(_W3CTestConverter.__init__):
(_W3CTestConverter.convert_reference_relpaths):
(_W3CTestConverter.convert_style_data):
(_W3CTestConverter.convert_attributes_if_needed):
* Scripts/webkitpy/w3c/test_importer.py:
(TestImporter.find_importable_tests):
(TestImporter.import_tests):
* Scripts/webkitpy/w3c/test_parser.py:
(TestParser.load_file):
(TestParser.analyze_test):
(TestParser.support_files):</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
<li><a href="#trunkToolsScriptswebkitpyw3ctest_converterpy">trunk/Tools/Scripts/webkitpy/w3c/test_converter.py</a></li>
<li><a href="#trunkToolsScriptswebkitpyw3ctest_importerpy">trunk/Tools/Scripts/webkitpy/w3c/test_importer.py</a></li>
<li><a href="#trunkToolsScriptswebkitpyw3ctest_parserpy">trunk/Tools/Scripts/webkitpy/w3c/test_parser.py</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (173497 => 173498)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog 2014-09-10 23:57:34 UTC (rev 173497)
+++ trunk/Tools/ChangeLog 2014-09-11 00:01:47 UTC (rev 173498)
</span><span class="lines">@@ -1,3 +1,30 @@
</span><ins>+2014-09-10 Rebecca Hauck <rhauck@adobe.com>
+
+ import-w3c-tests doesn't handle relative paths to support files in ref files correctly
+ https://bugs.webkit.org/show_bug.cgi?id=135929
+
+ Reviewed by Bem Jones-Bey.
+
+ The recent refactor of the W3C test repo falsified a bunch of assmumptions that
+ were made when this script was originally written with respect to relative paths
+ in ref files. This patch updates import-w3c-tests to update paths in ref files if
+ they move relative to the test file.
+
+
+ * Scripts/webkitpy/w3c/test_converter.py:
+ (convert_for_webkit):
+ (_W3CTestConverter.__init__):
+ (_W3CTestConverter.convert_reference_relpaths):
+ (_W3CTestConverter.convert_style_data):
+ (_W3CTestConverter.convert_attributes_if_needed):
+ * Scripts/webkitpy/w3c/test_importer.py:
+ (TestImporter.find_importable_tests):
+ (TestImporter.import_tests):
+ * Scripts/webkitpy/w3c/test_parser.py:
+ (TestParser.load_file):
+ (TestParser.analyze_test):
+ (TestParser.support_files):
+
</ins><span class="cx"> 2014-09-10 Michael Catanzaro <mcatanzaro@igalia.com>
</span><span class="cx">
</span><span class="cx"> [GTK] allow overwriting destination of download
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpyw3ctest_converterpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/w3c/test_converter.py (173497 => 173498)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/w3c/test_converter.py 2014-09-10 23:57:34 UTC (rev 173497)
+++ trunk/Tools/Scripts/webkitpy/w3c/test_converter.py 2014-09-11 00:01:47 UTC (rev 173498)
</span><span class="lines">@@ -37,12 +37,12 @@
</span><span class="cx"> _log = logging.getLogger(__name__)
</span><span class="cx">
</span><span class="cx">
</span><del>-def convert_for_webkit(new_path, filename, host=Host()):
</del><ins>+def convert_for_webkit(new_path, filename, reference_support_info, host=Host()):
</ins><span class="cx"> """ Converts a file's |contents| so it will function correctly in its |new_path| in Webkit.
</span><span class="cx">
</span><span class="cx"> Returns the list of modified properties and the modified text if the file was modifed, None otherwise."""
</span><span class="cx"> contents = host.filesystem.read_binary_file(filename)
</span><del>- converter = _W3CTestConverter(new_path, filename, host)
</del><ins>+ converter = _W3CTestConverter(new_path, filename, reference_support_info, host)
</ins><span class="cx"> if filename.endswith('.css'):
</span><span class="cx"> return converter.add_webkit_prefix_to_unprefixed_properties(contents)
</span><span class="cx"> else:
</span><span class="lines">@@ -52,7 +52,7 @@
</span><span class="cx">
</span><span class="cx">
</span><span class="cx"> class _W3CTestConverter(HTMLParser):
</span><del>- def __init__(self, new_path, filename, host=Host()):
</del><ins>+ def __init__(self, new_path, filename, reference_support_info, host=Host()):
</ins><span class="cx"> HTMLParser.__init__(self)
</span><span class="cx">
</span><span class="cx"> self._host = host
</span><span class="lines">@@ -64,6 +64,7 @@
</span><span class="cx"> self.in_style_tag = False
</span><span class="cx"> self.style_data = []
</span><span class="cx"> self.filename = filename
</span><ins>+ self.reference_support_info = reference_support_info
</ins><span class="cx">
</span><span class="cx"> resources_path = self.path_from_webkit_root('LayoutTests', 'resources')
</span><span class="cx"> resources_relpath = self._filesystem.relpath(resources_path, new_path)
</span><span class="lines">@@ -126,12 +127,28 @@
</span><span class="cx"> # FIXME: Handle the JS versions of these properties and GetComputedStyle, too.
</span><span class="cx"> return (converted_properties, ''.join(text_chunks))
</span><span class="cx">
</span><ins>+ def convert_reference_relpaths(self, text):
+ """ Searches |text| for instances of files in reference_support_info and updates the relative path to be correct for the new ref file location"""
+ converted = text
+ for path in self.reference_support_info['files']:
+ if text.find(path) != -1:
+ # FIXME: This doesn't handle an edge case where simply removing the relative path doesn't work.
+ # See http://webkit.org/b/135677 for details.
+ new_path = re.sub(self.reference_support_info['reference_relpath'], '', path, 1)
+ converted = re.sub(path, new_path, text)
+
+ return converted
+
</ins><span class="cx"> def convert_style_data(self, data):
</span><span class="cx"> converted = self.add_webkit_prefix_to_unprefixed_properties(data)
</span><span class="cx"> if converted[0]:
</span><span class="cx"> self.converted_properties.extend(list(converted[0]))
</span><del>- return converted[1]
</del><span class="cx">
</span><ins>+ if self.reference_support_info is None or self.reference_support_info == {}:
+ return converted[1]
+
+ return self.convert_reference_relpaths(converted[1])
+
</ins><span class="cx"> def convert_attributes_if_needed(self, tag, attrs):
</span><span class="cx"> converted = self.get_starttag_text()
</span><span class="cx"> if tag in ('script', 'link'):
</span><span class="lines">@@ -148,6 +165,12 @@
</span><span class="cx"> new_style = self.convert_style_data(attr[1])
</span><span class="cx"> converted = re.sub(attr[1], new_style, converted)
</span><span class="cx">
</span><ins>+ src_tags = ('script', 'img', 'frame', 'iframe', 'input', 'layer', 'textarea', 'video', 'audio')
+ if tag in src_tags and self.reference_support_info is not None and self.reference_support_info != {}:
+ for attr_name, attr_value in attrs:
+ if attr_name == 'src':
+ converted = self.convert_reference_relpaths(attr_value)
+
</ins><span class="cx"> self.converted_data.append(converted)
</span><span class="cx">
</span><span class="cx"> def handle_starttag(self, tag, attrs):
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpyw3ctest_importerpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/w3c/test_importer.py (173497 => 173498)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/w3c/test_importer.py 2014-09-10 23:57:34 UTC (rev 173497)
+++ trunk/Tools/Scripts/webkitpy/w3c/test_importer.py 2014-09-11 00:01:47 UTC (rev 173498)
</span><span class="lines">@@ -207,21 +207,9 @@
</span><span class="cx"> ref_file = os.path.splitext(test_basename)[0] + '-expected'
</span><span class="cx"> ref_file += os.path.splitext(test_basename)[1]
</span><span class="cx">
</span><del>- copy_list.append({'src': test_info['reference'], 'dest': ref_file})
</del><ins>+ copy_list.append({'src': test_info['reference'], 'dest': ref_file, 'reference_support_info': test_info['reference_support_info']})
</ins><span class="cx"> copy_list.append({'src': test_info['test'], 'dest': filename})
</span><span class="cx">
</span><del>- # Update any support files that need to move as well to remain relative to the -expected file.
- if 'refsupport' in test_info.keys():
- for support_file in test_info['refsupport']:
- source_file = os.path.join(os.path.dirname(test_info['reference']), support_file)
- source_file = os.path.normpath(source_file)
-
- # Keep the dest as it was
- to_copy = {'src': source_file, 'dest': support_file}
-
- # Only add it once
- if not(to_copy in copy_list):
- copy_list.append(to_copy)
</del><span class="cx"> elif 'jstest' in test_info.keys():
</span><span class="cx"> jstests += 1
</span><span class="cx"> total_tests += 1
</span><span class="lines">@@ -230,11 +218,6 @@
</span><span class="cx"> total_tests += 1
</span><span class="cx"> copy_list.append({'src': fullpath, 'dest': filename})
</span><span class="cx">
</span><del>- if not total_tests:
- # We can skip the support directory if no tests were found.
- if 'support' in dirs:
- dirs.remove('support')
-
</del><span class="cx"> if copy_list:
</span><span class="cx"> # Only add this directory to the list if there's something to import
</span><span class="cx"> self.import_list.append({'dirname': root, 'copy_list': copy_list,
</span><span class="lines">@@ -282,6 +265,10 @@
</span><span class="cx"> continue
</span><span class="cx">
</span><span class="cx"> new_filepath = os.path.join(new_path, file_to_copy['dest'])
</span><ins>+ if 'reference_support_info' in file_to_copy.keys() and file_to_copy['reference_support_info'] != {}:
+ reference_support_info = file_to_copy['reference_support_info']
+ else:
+ reference_support_info = None
</ins><span class="cx">
</span><span class="cx"> if not(os.path.exists(os.path.dirname(new_filepath))):
</span><span class="cx"> os.makedirs(os.path.dirname(new_filepath))
</span><span class="lines">@@ -300,10 +287,11 @@
</span><span class="cx"> mimetype = mimetypes.guess_type(orig_filepath)
</span><span class="cx"> if 'html' in str(mimetype[0]) or 'xml' in str(mimetype[0]) or 'css' in str(mimetype[0]):
</span><span class="cx"> try:
</span><del>- converted_file = convert_for_webkit(new_path, filename=orig_filepath)
</del><ins>+ converted_file = convert_for_webkit(new_path, filename=orig_filepath, reference_support_info=reference_support_info)
</ins><span class="cx"> except:
</span><span class="cx"> _log.warn('Failed converting %s', orig_filepath)
</span><span class="cx"> failed_conversion_files.append(orig_filepath)
</span><ins>+ converted_file = None
</ins><span class="cx">
</span><span class="cx"> if not converted_file:
</span><span class="cx"> shutil.copyfile(orig_filepath, new_filepath) # The file was unmodified.
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpyw3ctest_parserpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/w3c/test_parser.py (173497 => 173498)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/w3c/test_parser.py 2014-09-10 23:57:34 UTC (rev 173497)
+++ trunk/Tools/Scripts/webkitpy/w3c/test_parser.py 2014-09-11 00:01:47 UTC (rev 173498)
</span><span class="lines">@@ -49,37 +49,31 @@
</span><span class="cx"> self.ref_doc = None
</span><span class="cx"> self.load_file(filename)
</span><span class="cx">
</span><del>- def load_file(self, filename):
</del><ins>+ def load_file(self, filename, is_ref=False):
</ins><span class="cx"> if self.filesystem.isfile(filename):
</span><span class="cx"> try:
</span><del>- self.test_doc = Parser(self.filesystem.read_binary_file(filename))
</del><ins>+ doc = Parser(self.filesystem.read_binary_file(filename))
</ins><span class="cx"> except:
</span><span class="cx"> # FIXME: Figure out what to do if we can't parse the file.
</span><span class="cx"> _log.error("Failed to parse %s", filename)
</span><del>- self.test_doc is None
</del><ins>+ doc = None
</ins><span class="cx"> else:
</span><span class="cx"> if self.filesystem.isdir(filename):
</span><span class="cx"> # FIXME: Figure out what is triggering this and what to do about it.
</span><span class="cx"> _log.error("Trying to load %s, which is a directory", filename)
</span><del>- self.test_doc = None
- self.ref_doc = None
</del><ins>+ doc = None
</ins><span class="cx">
</span><ins>+ if is_ref:
+ self.ref_doc = doc
+ else:
+ self.test_doc = doc
+
</ins><span class="cx"> def analyze_test(self, test_contents=None, ref_contents=None):
</span><span class="cx"> """ Analyzes a file to determine if it's a test, what type of test, and what reference or support files it requires. Returns all of the test info """
</span><span class="cx">
</span><span class="cx"> test_info = None
</span><span class="cx">
</span><del>- if test_contents is None and self.test_doc is None:
- return test_info
-
- if test_contents is not None:
- self.test_doc = Parser(test_contents)
-
- if ref_contents is not None:
- self.ref_doc = Parser(ref_contents)
-
</del><span class="cx"> # First check if it's a reftest
</span><del>-
</del><span class="cx"> matches = self.reference_links_of_type('match') + self.reference_links_of_type('mismatch')
</span><span class="cx"> if matches:
</span><span class="cx"> if len(matches) > 1:
</span><span class="lines">@@ -95,29 +89,17 @@
</span><span class="cx"> return None
</span><span class="cx">
</span><span class="cx"> if self.ref_doc is None:
</span><del>- self.ref_doc = self.load_file(ref_file)
</del><ins>+ self.load_file(ref_file, True)
</ins><span class="cx">
</span><span class="cx"> test_info = {'test': self.filename, 'reference': ref_file}
</span><span class="cx">
</span><del>- # If the ref file path is relative, we need to check it for
- # relative paths also because when it lands in WebKit, it will be
- # moved down into the test dir.
- #
- # Note: The test files themselves are not checked for support files
- # outside their directories as the convention in the CSSWG is to
- # put all support files in the same dir or subdir as the test.
- #
- # All non-test files in the test's directory tree are normally
- # copied as part of the import as they are assumed to be required
- # support files.
- #
- # *But*, there is exactly one case in the entire css2.1 suite where
- # a test depends on a file that lives in a different directory,
- # which depends on another file that lives outside of its
- # directory. This code covers that case :)
- if matches[0]['href'].startswith('..'):
- support_files = self.support_files(self.ref_doc)
- test_info['refsupport'] = support_files
</del><ins>+ # If the ref file does not live in the same directory as the test file, check it for support files
+ test_info['reference_support_info'] = {}
+ if self.filesystem.dirname(ref_file) != self.filesystem.dirname(self.filename):
+ reference_support_files = self.support_files(self.ref_doc)
+ if len(reference_support_files) > 0:
+ reference_relpath = self.filesystem.relpath(self.filesystem.dirname(self.filename), self.filesystem.dirname(ref_file)) + self.filesystem.sep
+ test_info['reference_support_info'] = {'reference_relpath': reference_relpath, 'files': reference_support_files}
</ins><span class="cx">
</span><span class="cx"> elif self.is_jstest():
</span><span class="cx"> test_info = {'test': self.filename, 'jstest': True}
</span><span class="lines">@@ -156,7 +138,8 @@
</span><span class="cx">
</span><span class="cx"> paths = src_paths + href_paths + urls
</span><span class="cx"> for path in paths:
</span><del>- if not(path.startswith('http:')) and not(path.startswith('mailto:')):
</del><ins>+ uri_scheme_pattern = re.compile(r"[A-Za-z][A-Za-z+.-]*:")
+ if uri_scheme_pattern.match(path):
</ins><span class="cx"> support_files.append(path)
</span><span class="cx">
</span><span class="cx"> return support_files
</span></span></pre>
</div>
</div>
</body>
</html>