<div class="gmail_quote">On Tue, Jun 12, 2012 at 9:50 AM, Darin Adler <span dir="ltr"><<a href="mailto:darin@apple.com" target="_blank">darin@apple.com</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- Each time we change the tests we need to make sure we haven’t broken new tests on various platforms and that we’ve updated the skipped or test expectations files properly.<br></blockquote><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
- When rearranging tests that share the test shell JavaScript files we’d likely need to update relative paths. Might also be nice to put that test shell in a more logical location.</blockquote><div><br></div><div>For these 2 points, we probably need some script to move a test or a set of tests.</div>
<div> </div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">- Calling the entire directory that hosts our regression test suite LayoutTests seems wrong; that name was apropos when most of the tests were for layout issues, very early in the life of the test machinery.</blockquote>
<div><br></div><div>The last time we had this discussion, the consensus appeared to be renaming it to RegressionTests. Do people have different opinions at this time?</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- Imported test suites ought to be gathered together in a single directory then grouped by where they were imported from by having a directory for each source.<br></blockquote><div><br></div><div>Are you suggesting that we have LayoutTests/W3C?</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- We want to continue the practice of keeping copies imported test suites as close to the original as possible. In some cases in the past that meant preserving incorrect imported tests with expected “failures” that weren’t really failures. Often we’d put corrected copies elsewhere in the test tree while waiting for or working to get the imported test to fixed at the source.<br>
<br>
- Keeping the tests that require the HTTP server in a separate directory still seems like it might serve a purpose, although there seems to be an extra directory level in the test hierarchy there that is not helpful.<br>
</blockquote><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- Making JavaScript-only tests runnable without WebKit’s involvement would be a welcome feature; great for anyone working on JavaScriptCore. That means we’d want to be sure to put those into their own directories. Ideally that would eventually be coupled by a project to move the Mozilla test suite currently in JavaScriptCore into the main WebKit regression tests.<br>
</blockquote><div><br></div><div>Another point. Related to Oliver's response: <a href="http://lists.webkit.org/pipermail/webkit-dev/2011-September/017872.html">http://lists.webkit.org/pipermail/webkit-dev/2011-September/017872.html</a> on September 8th, 2011, we have at least 15 test running scripts (the number has since increased :( ) at this point:</div>
<div><div><ol><li>run-api-tests</li><li>run-binding-tests</li><li>run-chromium-webkit-unit-tests</li><li>run-gtk-tests</li><li>run-iexploder-tests</li><li>run-javascriptcore-tests</li><li>run-mangleme-tests</li><li>run-perf-tests</li>
<li>run-qtwebkit-tests</li><li>run-regexp-tests</li><li>run-test-webkit-api # what's the difference between this and run-api-tests?</li><li>test-webkit-scripts</li><li>run-webkit-tests</li><li>test-webkitpy</li><li>test-webkitperl</li>
</ol></div></div><div>And I find it really annoying that it's non-trivial to figure out which script runs which tests and where the test code is located. Some of these test runners appear to be port specific while others seem to be port-agnostic.</div>
<div><br></div><div>Also, given that the vast majority of these test runners also run some sort of regression tests (e.g. run-api-tests; API tests certainly are regression tests for some APIs or so I hope), renaming LayoutTests to RegressionTests might be a little confusing because of that.</div>
<div><br></div><div>- Ryosuke</div><div><br></div></div>