[webkit-dev] increase the number of tests before bailing?

[Ojan Vafai]

Currently, --exit-after-n-failures on the bots is set to 20. I like the idea
of exiting early, but I think 20 is too low. We should up it to 100. Is
anyone opposed to that?

There are some straightforward, mechanical patches that cause more than 20
tests to fail where they just need new expected results (e.g. changing form
control or scrollbar metrics). Right now, to make such a change you need
access to every platform so you can create new results or you need to get
someone who has access to that platform to pull in your change and create
new results.

The problem that confounds this is that many people have trouble ever
getting all the tests to pass on Windows. I've never succeeded. There are
always ~50 tests that fail for me and it's not due to lack of trying. So,
even though I have access to Windows, it's hard for me to get new expected
results for Windows changes.

Long-term we really need a solution that lets you get expected results for a
platform you don't have access to without committing code, e.g., the EWS
archiving results for failing tests.

Ojan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100616/805d6390/attachment.html>