[Webkit-unassigned] [Bug 223020] Slower tests than [ Slow ] cannot be tested in consistent way

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=223020

--- Comment #4 from Kimmo Kinnunen <kkinnunen at apple.com> ---
(In reply to Jonathan Bedard from comment #1)
> Is there any way to speed up these tests or maybe split them up? My concern
> here is that "Slow" already gives a test a timeout of minutes, which can
> have a meaningful impact on the time it takes CI to run things.
>(In reply to Alexey Proskuryakov from comment #2)
> I agree, even Slow tests are problematic, and overall test suite time has
> become excessive already.

I don't think these comments are productive.

I'm looking for a technical solution for an actual technical problem that exist today and is unaddressed.

I'm not looking to expand the test suite time. I'm looking to shrink the time.

What is in the test suite is a policy decision. I'm not looking to discuss this.

So let me rephrase:
Currently random amount of tests timeout because they're slower than the slow timeout.
This cause the tests to be re-run at the end, sequentially.
This causes longer test runs and flaky results.
This is an unilateral problem for development as well as bot infrastructure.

I'm looking for a solution for this.

One solution to this is that these could be tagged as VerySlow, increasing their timeout. Increasing these tests timeout would not cause any further increase of test suite time, because the timeouts are due to slowness, not hangs. Since the increased time would be executed in parallel instead of the extra sequential runs in the end, we can increase the timeout up to Slow*2 and probably not see any increase in runtime. If there's a problem wrt sequencing, the deqp bot and the developers can be instructed to use --order=random.

Other, only partial, solution is to allow [ Timeout Slow ] as the result. 

There may be other solutions, what do you think?

Note, for these test being discussed, deqp tests, the tests are not even run on main bots or anybody in particular. They're run by (probably mostly only) *me* and the deqp bot.

The tests are already split quite a lot. Spliting them more is quite an effort, as they're part of the standardisation effort. Splitting tests do not make the test run faster.

The tests are slow, because they (for the most part) simulate GPU pixel rasterisation in JavaScript. So they execute a lot of JavaScript in Debug process.

There is a related slowness problem for the main WebGL tests. However, in this bug I'd like to address the slowness and timeouts of deqp tests.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20210310/e2600335/attachment.htm>