[webkit-dev] Can we distinguish imported tests in LayoutTests/css3 ?

[Ryosuke Niwa]

On Tue, Jun 12, 2012 at 9:50 AM, Darin Adler <darin at apple.com> wrote:
>
> - Each time we change the tests we need to make sure we haven’t broken new
> tests on various platforms and that we’ve updated the skipped or test
> expectations files properly.
>

- When rearranging tests that share the test shell JavaScript files we’d
> likely need to update relative paths. Might also be nice to put that test
> shell in a more logical location.


For these 2 points, we probably need some script to move a test or a set of
tests.


> - Calling the entire directory that hosts our regression test suite
> LayoutTests seems wrong; that name was apropos when most of the tests were
> for layout issues, very early in the life of the test machinery.


The last time we had this discussion, the consensus appeared to be renaming
it to RegressionTests. Do people have different opinions at this time?

- Imported test suites ought to be gathered together in a single directory
> then grouped by where they were imported from by having a directory for
> each source.
>

Are you suggesting that we have LayoutTests/W3C?

- We want to continue the practice of keeping copies imported test suites
> as close to the original as possible. In some cases in the past that meant
> preserving incorrect imported tests with expected “failures” that weren’t
> really failures. Often we’d put corrected copies elsewhere in the test tree
> while waiting for or working to get the imported test to fixed at the
> source.
>
> - Keeping the tests that require the HTTP server in a separate directory
> still seems like it might serve a purpose, although there seems to be an
> extra directory level in the test hierarchy there that is not helpful.
>

- Making JavaScript-only tests runnable without WebKit’s involvement would
> be a welcome feature; great for anyone working on JavaScriptCore. That
> means we’d want to be sure to put those into their own directories. Ideally
> that would eventually be coupled by a project to move the Mozilla test
> suite currently in JavaScriptCore into the main WebKit regression tests.
>

Another point. Related to Oliver's response:
http://lists.webkit.org/pipermail/webkit-dev/2011-September/017872.html on
September 8th, 2011, we have at least 15 test running scripts (the number
has since increased :( ) at this point:

   1. run-api-tests
   2. run-binding-tests
   3. run-chromium-webkit-unit-tests
   4. run-gtk-tests
   5. run-iexploder-tests
   6. run-javascriptcore-tests
   7. run-mangleme-tests
   8. run-perf-tests
   9. run-qtwebkit-tests
   10. run-regexp-tests
   11. run-test-webkit-api # what's the difference between this and
   run-api-tests?
   12. test-webkit-scripts
   13. run-webkit-tests
   14. test-webkitpy
   15. test-webkitperl

And I find it really annoying that it's non-trivial to figure out which
script runs which tests and where the test code is located. Some of these
test runners appear to be port specific while others seem to be
port-agnostic.

Also, given that the vast majority of these test runners also run some sort
of regression tests (e.g. run-api-tests; API tests certainly are regression
tests for some APIs or so I hope), renaming LayoutTests to RegressionTests
might be a little confusing because of that.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120612/ee13dc8c/attachment.html>