[webkit-dev] Rename LayoutTests

[Ryosuke Niwa]

On Tue, May 9, 2017 at 10:23 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
>
>> On May 9, 2017, at 9:07 PM, Michael Catanzaro <mcatanzaro at igalia.com> wrote:
>>
>> On Tue, May 9, 2017 at 11:01 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>>> How about just Tests?
>>> Or alternately, RegressionTests. But I like just plain Tests.
>>
>> Then we should move ManualTests
>
> I'd be in favor of burying this somewhere deeper. As it is, people are still adding tests here, which is kind of a disaster. These tests are very rarely run, so a manual test is often barely better than no test at all. We should also put a file in this directory strongly discouraging the addition of new manual tests IMO.
>
>> , PerformanceTests,
>
> Could be renamed Benchmarks.

I'm not sure benchmarks would be a good description given that
directory also contains perf tests that are written to test specific
feature like line layout and DOM bindings.

>
>> JSTests, and
>
> Could go under JavaScriptCore, since these by design don't test anything outside of JavaScriptCore. But I also think having Tests, JSTests and Benchmarks at top level would be totally understandable.

This used to be the case but it got moved out of JavaScriptCore in
https://trac.webkit.org/changeset/203972 to be aligned with
LayoutTests and ManualTests and ease the pain of grep'ing
Source/JavaScriptCore, which used to find a whole bunch of things
inside Source/JavaScriptCote/Tests

>> TestWebKitAPI
>
> That might be reasonable for TestWebKitAPI/Tests but probably not TestWebKitAPI itself, since the part other than the tests is a harness like WebKitTestRunner. And I'm not sure it's practical to move just the tests. It's probably easier to have them contained inside the tool in the source tree.
>
> Also this is a weird mix of native code tests of public APIs for different frameworks, and native code unit tests for some internal classes (mostly low-level data structures).
>
> If these didn't need to be linked into a single tool, I might suggest that each framework should contain its own units and API tests.

I'm not sure if that would be a good idea for the same reason JSTests
got moved out of Source/JavaScriptCore. Chromium use a similar
directory structure where tests are located next to code, and I find
that to be extremely annoying when grep'ing the code because I end up
finding a whole bunch of code in tests while looking for some keyword.

I find it much nicer to have a separate test directory under which the
source code structure is mirrored such as UnitTests/WTFTests/,
UnitTests/WebCoreTests/, etc...

>> underneath it, because it would be weird to have tests outside of the Tests directory. Right?
>>
>> And then we would probably want to move all the layout tests to a new subdirectory, to separate them from the ManualTests, PerformanceTests, JSTests, and API tests. Then we have to find a name that subdirectory....
>
> If we did add any special directories to Tests with different semantics, they could just be special directories that are peers to the others, much like the http/ directory.
>
> What are now called LayoutTests have the distinction (along with PerformanceTests) of being tests that can cover things up and down the stack. Most other tests could be assigned to one of the subdirectories of Source.

This is why I think IntegrationTests or FunctionalTests most
accurately describe these tests.

> But I'd also be ok with having special subdirectories under Source.

I'd be strongly against that approach given my prior experience
working with (sub-)projects which used that structure including but
not limited to JavaScriptCore, webkitpy, and Chromium.

> BTW we also have:
>  bindings tests (under WebCore/bindings/scripts/tests)
>  perl tests (under Tools)
>  python tests (under Tools)
>  builtins-generator-tests (not sure what or where these are)
>  dashboard-tests (also not sure what or where these are)
>
> In general, I think it's good for more specific kinds of tests like this to be next to their relevant code.

I think these are small enough that they don't suffer from the same
issue of making grep'ing much harder although I've found binding tests
to be annoying for the same reason.

- R. Niwa