[webkit-dev] Importing Test262 into WebKit

[Keith Miller]

I already have a patch with the necessary changes to our test runner: https://bugs.webkit.org/show_bug.cgi?id=157797, which should answer most of your questions about my plans to import test262. 

I’ll try to answer your questions here, anyway, so people don’t have to read the bugzilla. 

On May 17, 2016, at 7:35 AM, Osztrogonác Csaba <oszi at inf.u-szeged.hu> wrote:
> 
> Hi,
> 
> I checked the current status on Linux (JSCOnly port), and got the following:
> - 38109 tests (19758 different tests, most of them are run in
>  strict and non-strict mode too, some of them only in one mode.)
> - 2742 failures
> - runtime: 12.5 minutes (on a Core i5 760 @ 2.80GHz CPU)
>  note: test262's (deprecated!) test runner can't run tests parallel
> 
> How do you plan to include test262?
> - check-in all tests (20Mb, ~20k files!) / update them periodically ?
> - clone a stable revision automatically by test-harness / manually by the user ?
> 

The plan is to check in the tests. This is the most consistent with the current way we do tests and makes it easier for developers to run the tests (and the best way to make sure we don’t have broken tests)

> How do you plan to handle [onlyStrict], [noStrict] flags and
> negative/includes directives in test cases?
> - I would prefer offline parsing of the test cases and check-in
> a generated yaml file describes how to run each tests properly.
> (With a checked in script to be able to regenerate it easily after an update.)
> 

That's exactly what I went with. :) The script to regenerate the yaml isn’t fully automated but I imagine with time it will be.

> Do you plan to run all tests with different JSC configurations?
> (noJIT,noLLINT, DFG/FTLEager, noCJIT, …)

Currently, I just do a default run, with nothing special. I would guess the non-optimizing tier configurations are something we should add in the future. In theory we could do all the modes but since 99+% of the tests I have seen would never tier up anyway its not clear how profitable those configurations would be.

> 
> It would be great to see preliminary results, such as runtime, conformance,
> stability before making JSC tester bots run these tests by default.
> 

On release builds (with x86-64, I haven’t tried 32-bit or ARM yet) , there is only one crashing test, which I omitted from the test suite and have an open bug to fix. On my system (Xeon E5 @ 2.7 GHz) it takes about 6min to run the test suite. It seems like a large amount of that time, probably ~2-3min, is setting up the testing environment, which we could probably speed up in the future. As far as conformance, we currently fail 2610 tests in the suite.

> I willingly help testing patches on X86_64/ARM Linux environments,
> reviewing run-jsc-stress-tests improvements, please cc me to the
> related bug reports and feel free to ping me on IRC.
> 
> br,
> Ossy
> 
> On 2016-05-13 01:02, Keith Miller wrote:
>> Hi everyone,
>> 
>> For those of you that have not already heard of Test262, it is a continually updated conformance test suite for upcoming ECMAScript standards (https://github.com/tc39/test262). I think it’s in our best interest to include the test suite in our normal testing infrastructure for JavaScriptCore. The current plan is to only run the Test262 tests when running run-javascript-core tests. The test suite is fairly large (~15000 tests) and the last time I ran it, it took about ~15-20 minutes to run. Are there any questions or concerns with this proposal?
>> 
>> Cheers,
>> Keith
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev