<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[197163] trunk/Tools</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/197163">197163</a></dd>
<dt>Author</dt> <dd>youenn.fablet@crf.canon.fr</dd>
<dt>Date</dt> <dd>2016-02-26 02:10:06 -0800 (Fri, 26 Feb 2016)</dd>
</dl>
<h3>Log Message</h3>
<pre>W3C test importer should have an option to clean the destination directory
https://bugs.webkit.org/show_bug.cgi?id=152685
Reviewed by Darin Adler.
Adding --clean-dest-dir option to W3C test importer.
When this option is set, all files in the destination directory will be deleted
except for WebKit specific files (test expectations, .gitignore...) before new tests import.
Dangling test expectations are removed after tests import.'
Adding unit test and minor refactoring for the other tests.
* Scripts/webkitpy/w3c/test_importer.py:
(parse_args): Add '--clean-dest-dir' option.
(TestImporter.do_import):
(TestImporter._is_baseline): helper routine to capture -expected.txt files.
(TestImporter):
(TestImporter._should_not_keep_when_importing): helper routine to filter files that should not be cleaned before importing.
(TestImporter.clean_destination_directory):
(TestImporter.remove_dangling_expectations):
* Scripts/webkitpy/w3c/test_importer_unittest.py:
(TestImporterTest._parse_options):
(TestImporterTest.test_import_dir_with_no_tests_and_no_hg):
(TestImporterTest.test_import_dir_with_no_tests):
(TestImporterTest.test_import_dir_with_empty_init_py):
(test_clean_directory_option):</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
<li><a href="#trunkToolsScriptswebkitpyw3ctest_importerpy">trunk/Tools/Scripts/webkitpy/w3c/test_importer.py</a></li>
<li><a href="#trunkToolsScriptswebkitpyw3ctest_importer_unittestpy">trunk/Tools/Scripts/webkitpy/w3c/test_importer_unittest.py</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (197162 => 197163)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog 2016-02-26 08:55:07 UTC (rev 197162)
+++ trunk/Tools/ChangeLog 2016-02-26 10:10:06 UTC (rev 197163)
</span><span class="lines">@@ -1,3 +1,32 @@
</span><ins>+2016-02-26 Youenn Fablet <youenn.fablet@crf.canon.fr>
+
+ W3C test importer should have an option to clean the destination directory
+ https://bugs.webkit.org/show_bug.cgi?id=152685
+
+ Reviewed by Darin Adler.
+
+ Adding --clean-dest-dir option to W3C test importer.
+ When this option is set, all files in the destination directory will be deleted
+ except for WebKit specific files (test expectations, .gitignore...) before new tests import.
+ Dangling test expectations are removed after tests import.'
+
+ Adding unit test and minor refactoring for the other tests.
+
+ * Scripts/webkitpy/w3c/test_importer.py:
+ (parse_args): Add '--clean-dest-dir' option.
+ (TestImporter.do_import):
+ (TestImporter._is_baseline): helper routine to capture -expected.txt files.
+ (TestImporter):
+ (TestImporter._should_not_keep_when_importing): helper routine to filter files that should not be cleaned before importing.
+ (TestImporter.clean_destination_directory):
+ (TestImporter.remove_dangling_expectations):
+ * Scripts/webkitpy/w3c/test_importer_unittest.py:
+ (TestImporterTest._parse_options):
+ (TestImporterTest.test_import_dir_with_no_tests_and_no_hg):
+ (TestImporterTest.test_import_dir_with_no_tests):
+ (TestImporterTest.test_import_dir_with_empty_init_py):
+ (test_clean_directory_option):
+
</ins><span class="cx"> 2016-02-25 Myles C. Maxfield <mmaxfield@apple.com>
</span><span class="cx">
</span><span class="cx"> REGRESSION(r195795): [WK2] fast/text/crash-complex-text-surrogate.html is flakey
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpyw3ctest_importerpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/w3c/test_importer.py (197162 => 197163)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/w3c/test_importer.py 2016-02-26 08:55:07 UTC (rev 197162)
+++ trunk/Tools/Scripts/webkitpy/w3c/test_importer.py 2016-02-26 10:10:06 UTC (rev 197163)
</span><span class="lines">@@ -147,6 +147,9 @@
</span><span class="cx"> parser.add_argument('--import-all', action='store_true', default=False,
</span><span class="cx"> help='Ignore the ImportExpectations file. All tests will be imported. This option only applies when tests are downloaded from W3C repository')
</span><span class="cx">
</span><ins>+ parser.add_argument('--clean-dest-dir', action='store_true', dest='clean_destination_directory', default=False,
+ help='Clean destination directory. All files in the destination directory will be deleted except for WebKit specific files (test expectations, .gitignore...) before new tests import. Dangling test expectations (expectation file that is no longer related to a test) are removed after tests import.')
+
</ins><span class="cx"> options, args = parser.parse_known_args(args)
</span><span class="cx"> if len(args) > 1:
</span><span class="cx"> parser.error('Incorrect number of arguments')
</span><span class="lines">@@ -185,14 +188,20 @@
</span><span class="cx"> self.filesystem.maybe_make_directory(self.source_directory)
</span><span class="cx"> self.test_downloader().download_tests(self.source_directory, self.options.test_paths)
</span><span class="cx">
</span><del>- if not self.options.test_paths or self._importing_downloaded_tests:
- self.find_importable_tests(self.source_directory)
- else:
- for test_path in self.options.test_paths:
- self.find_importable_tests(self.filesystem.join(self.source_directory, test_path))
</del><ins>+ test_paths = self.options.test_paths if self.options.test_paths else [test_repository['name'] for test_repository in self.test_downloader().test_repositories]
+ for test_path in test_paths:
+ self.find_importable_tests(self.filesystem.join(self.source_directory, test_path))
</ins><span class="cx">
</span><ins>+ if self.options.clean_destination_directory:
+ for test_path in test_paths:
+ self.clean_destination_directory(test_path)
+
</ins><span class="cx"> self.import_tests()
</span><span class="cx">
</span><ins>+ if self.options.clean_destination_directory:
+ for test_path in test_paths:
+ self.remove_dangling_expectations(test_path)
+
</ins><span class="cx"> if self._importing_downloaded_tests:
</span><span class="cx"> self.generate_git_submodules_description_for_all_repositories()
</span><span class="cx">
</span><span class="lines">@@ -220,6 +229,29 @@
</span><span class="cx"> return not filename == '.htaccess'
</span><span class="cx"> return False
</span><span class="cx">
</span><ins>+ def _is_baseline(self, filesystem, dirname, filename):
+ return filename.endswith('-expected.txt')
+
+ def _should_remove_before_importing(self, filesystem, dirname, filename):
+ if self._is_baseline(filesystem, dirname, filename):
+ return False
+ if filename.startswith("."):
+ return False
+ return True
+
+ def clean_destination_directory(self, filename):
+ directory = self.filesystem.join(self.destination_directory, filename)
+ for relative_path in self.filesystem.files_under(directory, file_filter=self._should_remove_before_importing):
+ self.filesystem.remove(self.filesystem.join(directory, relative_path))
+
+ def remove_dangling_expectations(self, filename):
+ #FIXME: Clean also the expected files stored in all platform specific folders.
+ directory = self.filesystem.join(self.destination_directory, filename)
+ for relative_path in self.filesystem.files_under(directory, file_filter=self._is_baseline):
+ path = self.filesystem.join(directory, relative_path)
+ if self.filesystem.glob(path.replace('-expected.txt', '*')) == [path]:
+ self.filesystem.remove(path)
+
</ins><span class="cx"> def find_importable_tests(self, directory):
</span><span class="cx"> def should_keep_subdir(filesystem, path):
</span><span class="cx"> if self._importing_downloaded_tests:
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpyw3ctest_importer_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/w3c/test_importer_unittest.py (197162 => 197163)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/w3c/test_importer_unittest.py 2016-02-26 08:55:07 UTC (rev 197162)
+++ trunk/Tools/Scripts/webkitpy/w3c/test_importer_unittest.py 2016-02-26 10:10:06 UTC (rev 197163)
</span><span class="lines">@@ -70,12 +70,16 @@
</span><span class="cx">
</span><span class="cx"> class TestImporterTest(unittest.TestCase):
</span><span class="cx">
</span><ins>+ def _parse_options(self, args):
+ options, args = parse_args(args)
+ return options
+
</ins><span class="cx"> def test_import_dir_with_no_tests_and_no_hg(self):
</span><span class="cx"> host = MockHost()
</span><span class="cx"> host.executive = MockExecutive2(exception=OSError())
</span><span class="cx"> host.filesystem = MockFileSystem(files=FAKE_FILES)
</span><span class="cx">
</span><del>- importer = TestImporter(host, FAKE_SOURCE_DIR, optparse.Values({"overwrite": False, 'destination': 'w3c', 'test_paths': [FAKE_TEST_PATH]}))
</del><ins>+ importer = TestImporter(host, FAKE_SOURCE_DIR, self._parse_options(['-n', '-d', 'w3c', '-t', FAKE_TEST_PATH]))
</ins><span class="cx">
</span><span class="cx"> oc = OutputCapture()
</span><span class="cx"> oc.capture_output()
</span><span class="lines">@@ -89,7 +93,7 @@
</span><span class="cx"> host.executive = MockExecutive2(exception=ScriptError("abort: no repository found in '/Volumes/Source/src/wk/Tools/Scripts/webkitpy/w3c' (.hg not found)!"))
</span><span class="cx"> host.filesystem = MockFileSystem(files=FAKE_FILES)
</span><span class="cx">
</span><del>- importer = TestImporter(host, FAKE_SOURCE_DIR, optparse.Values({"overwrite": False, 'destination': 'w3c', 'test_paths': [FAKE_TEST_PATH]}))
</del><ins>+ importer = TestImporter(host, FAKE_SOURCE_DIR, self._parse_options(['-n', '-d', 'w3c', '-t', FAKE_TEST_PATH]))
</ins><span class="cx"> oc = OutputCapture()
</span><span class="cx"> oc.capture_output()
</span><span class="cx"> try:
</span><span class="lines">@@ -106,7 +110,7 @@
</span><span class="cx"> host = MockHost()
</span><span class="cx"> host.filesystem = MockFileSystem(files=FAKE_FILES)
</span><span class="cx">
</span><del>- importer = TestImporter(host, FAKE_SOURCE_DIR, optparse.Values({"overwrite": False, 'destination': 'w3c', 'test_paths': ['/tests/csswg']}))
</del><ins>+ importer = TestImporter(host, FAKE_SOURCE_DIR, self._parse_options(['-n', '-d', 'w3c', '-t', '/tests/csswg']))
</ins><span class="cx"> importer.do_import()
</span><span class="cx">
</span><span class="cx"> self.assertTrue(host.filesystem.exists("/mock-checkout/LayoutTests/w3c/test1/__init__.py"))
</span><span class="lines">@@ -185,5 +189,29 @@
</span><span class="cx"> self.assertFalse(fs.exists('/mock-checkout/LayoutTests/w3c/streams-api/reference-implementation/web-platform-tests/test.html'))
</span><span class="cx"> self.assertTrue(fs.exists('/mock-checkout/LayoutTests/w3c/streams-api/test.html'))
</span><span class="cx">
</span><ins>+ def test_clean_directory_option(self):
+ FAKE_FILES = {
+ '/mock-checkout/LayoutTests/w3c/web-platform-tests/.gitattributes': '-1',
+ '/mock-checkout/LayoutTests/w3c/web-platform-tests/.gitignore': '-1',
+ '/mock-checkout/LayoutTests/w3c/web-platform-tests/.svn/wc.db': '0',
+ '/mock-checkout/LayoutTests/w3c/web-platform-tests/old-test.html': '1',
+ '/mock-checkout/LayoutTests/w3c/web-platform-tests/old-test-expected.txt': '2',
+ '/mock-checkout/LayoutTests/w3c/web-platform-tests/existing-test.html': '3',
+ '/mock-checkout/LayoutTests/w3c/web-platform-tests/existing-test-expected.txt': '4',
+ '/mock-checkout/WebKitBuild/w3c-tests/web-platform-tests/existing-test.html': '5',
+ '/mock-checkout/WebKitBuild/w3c-tests/csswg-tests/test.html': '1',
+ }
</ins><span class="cx">
</span><ins>+ FAKE_FILES.update(FAKE_REPOSITORY)
+
+ fs = self.import_downloaded_tests(['--no-fetch', '--import-all', '-d', 'w3c', '--clean-dest-dir'], FAKE_FILES)
+
+ self.assertFalse(fs.exists('/mock-checkout/LayoutTests/w3c/web-platform-tests/old-test.html'))
+ self.assertFalse(fs.exists('/mock-checkout/LayoutTests/w3c/web-platform-tests/old-test-expected.txt'))
+ self.assertTrue(fs.exists('/mock-checkout/LayoutTests/w3c/web-platform-tests/existing-test.html'))
+ self.assertTrue(fs.exists('/mock-checkout/LayoutTests/w3c/web-platform-tests/existing-test-expected.txt'))
+ self.assertTrue(fs.exists('/mock-checkout/LayoutTests/w3c/web-platform-tests/.gitattributes'))
+ self.assertTrue(fs.exists('/mock-checkout/LayoutTests/w3c/web-platform-tests/.gitignore'))
+ self.assertTrue(fs.exists('/mock-checkout/LayoutTests/w3c/web-platform-tests/.svn'))
+
</ins><span class="cx"> # FIXME: Needs more tests.
</span></span></pre>
</div>
</div>
</body>
</html>