[Webkit-unassigned] [Bug 171578] New: run-javascriptcore-tests misses tests JSTests/asyncFunctionTests.yaml JSTests/heapProfiler.yaml and JSTests/test262.yaml

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=171578

            Bug ID: 171578
           Summary: run-javascriptcore-tests misses tests
                    JSTests/asyncFunctionTests.yaml
                    JSTests/heapProfiler.yaml and JSTests/test262.yaml
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: clopez at igalia.com
                CC: aakash_jain at apple.com, joepeck at webkit.org,
                    lforschler at apple.com

The script run-javascript-tests has a list of tests that it should run:
https://trac.webkit.org/browser/trunk/Tools/Scripts/run-javascriptcore-tests?rev=215108#L268

On this list, currently there are 3 tests missing:

JSTests/asyncFunctionTests.yaml
JSTests/heapProfiler.yaml
JSTests/test262.yaml

I have checked this tests, and all 3 work fine. So this seems an overlook.

In bug 170523, new bots were added to just run the JSTests/test262.yaml alone for the Mac port instead of adding JSTests/test262.yaml to the list of tests that the JSC bots run with the script run-javascript-tests.

The other two tests (JSTests/asyncFunctionTests.yaml and JSTests/heapProfiler.yaml) are currently not tested by any bot.


In order to fix this, my proposal is the following:
 1) Remove the list of tests to run from run-javascriptcore-tests
 2) Add instead a list of tests to be skipped
 3) Make the script auto-discover current tests by looking at *.yaml files on the test directories
    (shell example: find JSTests/ LayoutTests/ PerformanceTests/ -type f -name \*.yaml )

So the tool will be automatically running all the discovered tests except the ones skipped.

This will ensure we don't hit this situation in the future of JavaScriptCore tests that are not ran by any bot.
This is similar to how we run the layout tests (tests to run are auto-discovered from the filesystem).

The skipped list should contain JSTests/test262.yaml (this test is currently run in another step on the Mac bots).
 - The skipped list could be empty if the Mac port maintainers think that it would be a good idea to run JSTests/test262.yaml as part of run-javascript-tests, and migrate the curent test262 bots to general JSC testers (A build slave can have several workers). In any case this should be done in another bug.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20170502/85d9b157/attachment.html>