<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[211954] trunk</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/211954">211954</a></dd>
<dt>Author</dt> <dd>commit-queue@webkit.org</dd>
<dt>Date</dt> <dd>2017-02-09 08:03:05 -0800 (Thu, 09 Feb 2017)</dd>
</dl>
<h3>Log Message</h3>
<pre>Improve W3C importer command line options
https://bugs.webkit.org/show_bug.cgi?id=167448
Patch by Youenn Fablet <youenn@apple.com> on 2017-02-09
Reviewed by Ryosuke Niwa.
LayoutTests/imported/w3c:
* resources/TestRepositories: Skipped module generation and gitignore generation for the moment.
Tools:
Making the list of test paths to import the main arguments of the script.
Allow passing the source directory to import as a -s option.
Adding a warning when given test paths are skipped by default in ImportExpectations.
Making link conversion off by default when importing from a specific directory since this script is mostly used with web-platform-tests.
Link conversion rules do not change when imported tests are downloaded by the script.
* Scripts/webkitpy/w3c/test_importer.py:
(main):
(parse_args):
(TestImporter.__init__):
(TestImporter.do_import):
(TestImporter):
(TestImporter.check_imported_expectations):</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkLayoutTestsimportedw3cChangeLog">trunk/LayoutTests/imported/w3c/ChangeLog</a></li>
<li><a href="#trunkLayoutTestsimportedw3cresourcesTestRepositories">trunk/LayoutTests/imported/w3c/resources/TestRepositories</a></li>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
<li><a href="#trunkToolsScriptswebkitpyw3ctest_importerpy">trunk/Tools/Scripts/webkitpy/w3c/test_importer.py</a></li>
<li><a href="#trunkToolsScriptswebkitpyw3ctest_importer_unittestpy">trunk/Tools/Scripts/webkitpy/w3c/test_importer_unittest.py</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkLayoutTestsimportedw3cChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/imported/w3c/ChangeLog (211953 => 211954)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/imported/w3c/ChangeLog 2017-02-09 15:59:22 UTC (rev 211953)
+++ trunk/LayoutTests/imported/w3c/ChangeLog 2017-02-09 16:03:05 UTC (rev 211954)
</span><span class="lines">@@ -1,3 +1,12 @@
</span><ins>+2017-02-09 Youenn Fablet <youenn@apple.com>
+
+ Improve W3C importer command line options
+ https://bugs.webkit.org/show_bug.cgi?id=167448
+
+ Reviewed by Ryosuke Niwa.
+
+ * resources/TestRepositories: Skipped module generation and gitignore generation for the moment.
+
</ins><span class="cx"> 2017-02-08 Chris Dumez <cdumez@apple.com>
</span><span class="cx">
</span><span class="cx"> Update web-platform-tests resources
</span></span></pre></div>
<a id="trunkLayoutTestsimportedw3cresourcesTestRepositories"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/imported/w3c/resources/TestRepositories (211953 => 211954)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/imported/w3c/resources/TestRepositories 2017-02-09 15:59:22 UTC (rev 211953)
+++ trunk/LayoutTests/imported/w3c/resources/TestRepositories 2017-02-09 16:03:05 UTC (rev 211954)
</span><span class="lines">@@ -1,7 +1,7 @@
</span><span class="cx"> [
</span><span class="cx"> {
</span><span class="cx"> "name": "csswg-test",
</span><del>- "url": "https://github.com/w3c/csswg-test.git",
</del><ins>+ "url": "https://github.com/w3c/csswg-test.git",
</ins><span class="cx"> "revision": "b2daa426addd5ccb8e9ce1c5d800f9d82603f1ec",
</span><span class="cx"> "paths_to_skip": [
</span><span class="cx"> "support",
</span><span class="lines">@@ -30,6 +30,6 @@
</span><span class="cx"> "config.default.json",
</span><span class="cx"> "serve.py"
</span><span class="cx"> ],
</span><del>- "import_options": ["generate_git_submodules_description", "generate_gitignore", "generate_init_py"]
</del><ins>+ "import_options": ["generate_init_py"]
</ins><span class="cx"> }
</span><span class="cx"> ]
</span></span></pre></div>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (211953 => 211954)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog 2017-02-09 15:59:22 UTC (rev 211953)
+++ trunk/Tools/ChangeLog 2017-02-09 16:03:05 UTC (rev 211954)
</span><span class="lines">@@ -1,3 +1,25 @@
</span><ins>+2017-02-09 Youenn Fablet <youenn@apple.com>
+
+ Improve W3C importer command line options
+ https://bugs.webkit.org/show_bug.cgi?id=167448
+
+ Reviewed by Ryosuke Niwa.
+
+ Making the list of test paths to import the main arguments of the script.
+ Allow passing the source directory to import as a -s option.
+ Adding a warning when given test paths are skipped by default in ImportExpectations.
+
+ Making link conversion off by default when importing from a specific directory since this script is mostly used with web-platform-tests.
+ Link conversion rules do not change when imported tests are downloaded by the script.
+
+ * Scripts/webkitpy/w3c/test_importer.py:
+ (main):
+ (parse_args):
+ (TestImporter.__init__):
+ (TestImporter.do_import):
+ (TestImporter):
+ (TestImporter.check_imported_expectations):
+
</ins><span class="cx"> 2017-02-08 Chris Dumez <cdumez@apple.com>
</span><span class="cx">
</span><span class="cx"> Update web-platform-tests resources
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpyw3ctest_importerpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/w3c/test_importer.py (211953 => 211954)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/w3c/test_importer.py 2017-02-09 15:59:22 UTC (rev 211953)
+++ trunk/Tools/Scripts/webkitpy/w3c/test_importer.py 2017-02-09 16:03:05 UTC (rev 211954)
</span><span class="lines">@@ -77,7 +77,6 @@
</span><span class="cx"> import json
</span><span class="cx"> import logging
</span><span class="cx"> import mimetypes
</span><del>-import sys
</del><span class="cx">
</span><span class="cx"> from webkitpy.common.host import Host
</span><span class="cx"> from webkitpy.common.system.filesystem import FileSystem
</span><span class="lines">@@ -92,15 +91,11 @@
</span><span class="cx">
</span><span class="cx">
</span><span class="cx"> def main(_argv, _stdout, _stderr):
</span><del>- options, args = parse_args(_argv)
- import_dir = args[0] if args else None
- filesystem = FileSystem()
- if import_dir and not filesystem.exists(import_dir):
- sys.exit('Source directory %s not found!' % import_dir)
</del><ins>+ options, test_paths = parse_args(_argv)
</ins><span class="cx">
</span><span class="cx"> configure_logging()
</span><span class="cx">
</span><del>- test_importer = TestImporter(Host(), import_dir, options)
</del><ins>+ test_importer = TestImporter(Host(), test_paths, options)
</ins><span class="cx"> test_importer.do_import()
</span><span class="cx">
</span><span class="cx">
</span><span class="lines">@@ -121,7 +116,11 @@
</span><span class="cx">
</span><span class="cx">
</span><span class="cx"> def parse_args(args):
</span><del>- parser = argparse.ArgumentParser(prog='import-w3c-tests [w3c_test_source_directory]')
</del><ins>+ description = """
+To import a web-platform-tests test suite named xyz, use: 'import-w3c-tests web-platform-tests/xyz'.
+To import a csswg-test test suite named abc, use 'import-w3c-tests csswg-test/abc'.
+To import a web-platform-tests/csswg-test test suite from a specific folder, use 'import-w3c-tests web-platform-tests/xyz -s my-folder-containing-web-platform-tests-folder'"""
+ parser = argparse.ArgumentParser(prog='import-w3c-tests [web-platform-tests/test-suite-name...]', description=description, formatter_class=argparse.RawDescriptionHelpFormatter)
</ins><span class="cx">
</span><span class="cx"> parser.add_argument('-n', '--no-overwrite', dest='overwrite', action='store_false', default=True,
</span><span class="cx"> help='Flag to prevent duplicate test files from overwriting existing tests. By default, they will be overwritten')
</span><span class="lines">@@ -134,9 +133,8 @@
</span><span class="cx"> parser.add_argument('-d', '--dest-dir', dest='destination', default=fs.join('imported', 'w3c'),
</span><span class="cx"> help='Import into a specified directory relative to the LayoutTests root. By default, imports into imported/w3c')
</span><span class="cx">
</span><del>- list_of_repositories = ' or '.join([test_repository['name'] for test_repository in TestDownloader.load_test_repositories()])
- parser.add_argument('-t', '--test-path', action='append', dest='test_paths', default=[],
- help='Import only tests in the supplied subdirectory of the source directory. Can be supplied multiple times to give multiple paths. For tests directly cloned from W3C repositories, use ' + list_of_repositories + ' prefixes to filter specific tests')
</del><ins>+ parser.add_argument('-s', '--src-dir', dest='source', default=None,
+ help='Import from a specific folder which contains web-platform-tests and/or csswg-test folders. If not provided, the script will clone the necessary repositories.')
</ins><span class="cx">
</span><span class="cx"> parser.add_argument('-v', '--verbose', action='store_true', default=False,
</span><span class="cx"> help='Print maximal log')
</span><span class="lines">@@ -149,17 +147,16 @@
</span><span class="cx"> help='Clean destination directory. All files in the destination directory will be deleted except for WebKit specific files (test expectations, .gitignore...) before new tests import. Dangling test expectations (expectation file that is no longer related to a test) are removed after tests import.')
</span><span class="cx">
</span><span class="cx"> options, args = parser.parse_known_args(args)
</span><del>- if len(args) > 1:
- parser.error('Incorrect number of arguments')
</del><span class="cx"> return options, args
</span><span class="cx">
</span><span class="cx">
</span><span class="cx"> class TestImporter(object):
</span><span class="cx">
</span><del>- def __init__(self, host, source_directory, options):
</del><ins>+ def __init__(self, host, test_paths, options):
</ins><span class="cx"> self.host = host
</span><del>- self.source_directory = source_directory
</del><ins>+ self.source_directory = options.source
</ins><span class="cx"> self.options = options
</span><ins>+ self.test_paths = test_paths if test_paths else []
</ins><span class="cx">
</span><span class="cx"> self.filesystem = self.host.filesystem
</span><span class="cx">
</span><span class="lines">@@ -177,7 +174,7 @@
</span><span class="cx"> self._potential_test_resource_files = []
</span><span class="cx">
</span><span class="cx"> self.import_list = []
</span><del>- self._importing_downloaded_tests = source_directory is None
</del><ins>+ self._importing_downloaded_tests = self.source_directory is None
</ins><span class="cx">
</span><span class="cx"> self._test_resource_files_json_path = self.filesystem.join(self.layout_tests_w3c_path, "resources", "resource-files.json")
</span><span class="cx"> self._test_resource_files = json.loads(self.filesystem.read_text_file(self._test_resource_files_json_path)) if self.filesystem.exists(self._test_resource_files_json_path) else None
</span><span class="lines">@@ -197,9 +194,9 @@
</span><span class="cx"> self.source_directory = self.filesystem.join(self.tests_download_path, 'to-be-imported')
</span><span class="cx"> self.filesystem.maybe_make_directory(self.tests_download_path)
</span><span class="cx"> self.filesystem.maybe_make_directory(self.source_directory)
</span><del>- self.test_downloader().download_tests(self.source_directory, self.options.test_paths)
</del><ins>+ self.test_downloader().download_tests(self.source_directory, self.test_paths)
</ins><span class="cx">
</span><del>- test_paths = self.options.test_paths if self.options.test_paths else [test_repository['name'] for test_repository in self.test_downloader().test_repositories]
</del><ins>+ test_paths = self.test_paths if self.test_paths else [test_repository['name'] for test_repository in self.test_downloader().test_repositories]
</ins><span class="cx"> for test_path in test_paths:
</span><span class="cx"> self.find_importable_tests(self.filesystem.join(self.source_directory, test_path))
</span><span class="cx">
</span><span class="lines">@@ -216,6 +213,12 @@
</span><span class="cx"> if self._importing_downloaded_tests:
</span><span class="cx"> self.generate_git_submodules_description_for_all_repositories()
</span><span class="cx">
</span><ins>+ self.check_imported_expectations()
+
+ def check_imported_expectations(self):
+ for path in [path for path in self.test_paths if path in self.test_downloader().paths_to_skip]:
+ _log.warn('Please update LayoutTests/imported/w3c/resources/ImportExpectations to automate importing of ' + path)
+
</ins><span class="cx"> def generate_git_submodules_description_for_all_repositories(self):
</span><span class="cx"> for test_repository in self._test_downloader.test_repositories:
</span><span class="cx"> if 'generate_git_submodules_description' in test_repository['import_options']:
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpyw3ctest_importer_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/w3c/test_importer_unittest.py (211953 => 211954)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/w3c/test_importer_unittest.py 2017-02-09 15:59:22 UTC (rev 211953)
+++ trunk/Tools/Scripts/webkitpy/w3c/test_importer_unittest.py 2017-02-09 16:03:05 UTC (rev 211954)
</span><span class="lines">@@ -72,11 +72,13 @@
</span><span class="cx"> return options
</span><span class="cx">
</span><span class="cx"> def test_import_dir_with_no_tests_and_no_hg(self):
</span><ins>+ FAKE_FILES.update(FAKE_REPOSITORY)
+
</ins><span class="cx"> host = MockHost()
</span><span class="cx"> host.executive = MockExecutive2(exception=OSError())
</span><span class="cx"> host.filesystem = MockFileSystem(files=FAKE_FILES)
</span><span class="cx">
</span><del>- importer = TestImporter(host, FAKE_SOURCE_DIR, self._parse_options(['-n', '-d', 'w3c', '-t', FAKE_TEST_PATH]))
</del><ins>+ importer = TestImporter(host, FAKE_TEST_PATH, self._parse_options(['-n', '-d', 'w3c', '-s', FAKE_SOURCE_DIR]))
</ins><span class="cx">
</span><span class="cx"> oc = OutputCapture()
</span><span class="cx"> oc.capture_output()
</span><span class="lines">@@ -86,11 +88,13 @@
</span><span class="cx"> oc.restore_output()
</span><span class="cx">
</span><span class="cx"> def test_import_dir_with_no_tests(self):
</span><ins>+ FAKE_FILES.update(FAKE_REPOSITORY)
+
</ins><span class="cx"> host = MockHost()
</span><span class="cx"> host.executive = MockExecutive2(exception=ScriptError("abort: no repository found in '/Volumes/Source/src/wk/Tools/Scripts/webkitpy/w3c' (.hg not found)!"))
</span><span class="cx"> host.filesystem = MockFileSystem(files=FAKE_FILES)
</span><span class="cx">
</span><del>- importer = TestImporter(host, FAKE_SOURCE_DIR, self._parse_options(['-n', '-d', 'w3c', '-t', FAKE_TEST_PATH]))
</del><ins>+ importer = TestImporter(host, FAKE_TEST_PATH, self._parse_options(['-n', '-d', 'w3c', '-s', FAKE_SOURCE_DIR]))
</ins><span class="cx"> oc = OutputCapture()
</span><span class="cx"> oc.capture_output()
</span><span class="cx"> try:
</span><span class="lines">@@ -103,11 +107,12 @@
</span><span class="cx"> '/tests/csswg/test1/__init__.py': '',
</span><span class="cx"> '/tests/csswg/test2/__init__.py': 'NOTEMPTY',
</span><span class="cx"> }
</span><ins>+ FAKE_FILES.update(FAKE_REPOSITORY)
</ins><span class="cx">
</span><span class="cx"> host = MockHost()
</span><span class="cx"> host.filesystem = MockFileSystem(files=FAKE_FILES)
</span><span class="cx">
</span><del>- importer = TestImporter(host, FAKE_SOURCE_DIR, self._parse_options(['-n', '-d', 'w3c', '-t', '/tests/csswg']))
</del><ins>+ importer = TestImporter(host, ['test1', 'test2'], self._parse_options(['-n', '-d', 'w3c', '-s', FAKE_SOURCE_DIR]))
</ins><span class="cx"> importer.do_import()
</span><span class="cx">
</span><span class="cx"> self.assertTrue(host.filesystem.exists("/mock-checkout/LayoutTests/w3c/test1/__init__.py"))
</span></span></pre>
</div>
</div>
</body>
</html>