[webkit-dev] Importing Test262 into WebKit

[Osztrogonác Csaba]

Hi,

I checked the current status on Linux (JSCOnly port), and got the following:
- 38109 tests (19758 different tests, most of them are run in
   strict and non-strict mode too, some of them only in one mode.)
- 2742 failures
- runtime: 12.5 minutes (on a Core i5 760 @ 2.80GHz CPU)
   note: test262's (deprecated!) test runner can't run tests parallel

How do you plan to include test262?
- check-in all tests (20Mb, ~20k files!) / update them periodically ?
- clone a stable revision automatically by test-harness / manually by the user ?

How do you plan to handle [onlyStrict], [noStrict] flags and
negative/includes directives in test cases?
- I would prefer offline parsing of the test cases and check-in
a generated yaml file describes how to run each tests properly.
(With a checked in script to be able to regenerate it easily after an update.)

Do you plan to run all tests with different JSC configurations?
(noJIT,noLLINT, DFG/FTLEager, noCJIT, ...)

It would be great to see preliminary results, such as runtime, conformance,
stability before making JSC tester bots run these tests by default.

I willingly help testing patches on X86_64/ARM Linux environments,
reviewing run-jsc-stress-tests improvements, please cc me to the
related bug reports and feel free to ping me on IRC.

br,
Ossy

On 2016-05-13 01:02, Keith Miller wrote:
> Hi everyone,
>
> For those of you that have not already heard of Test262, it is a continually updated conformance test suite for upcoming ECMAScript standards (https://github.com/tc39/test262). I think it’s in our best interest to include the test suite in our normal testing infrastructure for JavaScriptCore. The current plan is to only run the Test262 tests when running run-javascript-core tests. The test suite is fairly large (~15000 tests) and the last time I ran it, it took about ~15-20 minutes to run. Are there any questions or concerns with this proposal?
>
> Cheers,
> Keith