[webkit-dev] increase the number of tests before bailing?

[Geoffrey Garen]

Hi Ojan.

I wonder if it would help to distinguish --exit-after-n-failures from --exit-after-n-crashes.

I think that crashing tests are the biggest problem, since they can cause a bot to lag behind quite a bit.

Geoff

On Jun 16, 2010, at 1:57 PM, Ojan Vafai wrote:

> Currently, --exit-after-n-failures on the bots is set to 20. I like the idea of exiting early, but I think 20 is too low. We should up it to 100. Is anyone opposed to that?
> 
> There are some straightforward, mechanical patches that cause more than 20 tests to fail where they just need new expected results (e.g. changing form control or scrollbar metrics). Right now, to make such a change you need access to every platform so you can create new results or you need to get someone who has access to that platform to pull in your change and create new results.
> 
> The problem that confounds this is that many people have trouble ever getting all the tests to pass on Windows. I've never succeeded. There are always ~50 tests that fail for me and it's not due to lack of trying. So, even though I have access to Windows, it's hard for me to get new expected results for Windows changes.
> 
> Long-term we really need a solution that lets you get expected results for a platform you don't have access to without committing code, e.g., the EWS archiving results for failing tests.
> 
> Ojan
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev