[webkit-dev] increase the number of tests before bailing?

[Eric Seidel]

I think --exit-after-N-failures is actually very useful as-is.  I
think peopel just use it for different things.  For teh commit-queue
--exit-after-N-failures is great for keeping it quick.

Perhaps people want to use the bots more like try-bots and have
--exit-after-N-failures higher.

I think these are all symptoms of WebKit's total lack of real trybots. ;)

-eric

On Wed, Jun 16, 2010 at 3:27 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
> On Jun 16, 2010, at 2:04 PM, Eric Seidel wrote:
>
>> We could add a separate option to DumpRenderTree to disable
>> ReportCrash (sign up for all the crashing signals and simply exit(2)
>> or similar).  That would be useful in many instances besides the bots.
>>
>> Yes, --exit-after-N-failures was designed to prevent crashers from
>> eating the bots.
>
> I think --exit-after-n-crashes is probably the most expedient solution to the problem.
>
> I think when only a few tests crash, having the crash report is very useful, particularly if the developer of the patch cannot reproduce the crash off the bot. All we need to do is limit the number of crashes to prevent the bots falling way behind.
>
> Regards,
> Maciej
>
>>
>> On Wed, Jun 16, 2010 at 1:59 PM, Geoffrey Garen <ggaren at apple.com> wrote:
>>> Hi Ojan.
>>>
>>> I wonder if it would help to distinguish --exit-after-n-failures from --exit-after-n-crashes.
>>>
>>> I think that crashing tests are the biggest problem, since they can cause a bot to lag behind quite a bit.
>>>
>>> Geoff
>>>
>>> On Jun 16, 2010, at 1:57 PM, Ojan Vafai wrote:
>>>
>>>> Currently, --exit-after-n-failures on the bots is set to 20. I like the idea of exiting early, but I think 20 is too low. We should up it to 100. Is anyone opposed to that?
>>>>
>>>> There are some straightforward, mechanical patches that cause more than 20 tests to fail where they just need new expected results (e.g. changing form control or scrollbar metrics). Right now, to make such a change you need access to every platform so you can create new results or you need to get someone who has access to that platform to pull in your change and create new results.
>>>>
>>>> The problem that confounds this is that many people have trouble ever getting all the tests to pass on Windows. I've never succeeded. There are always ~50 tests that fail for me and it's not due to lack of trying. So, even though I have access to Windows, it's hard for me to get new expected results for Windows changes.
>>>>
>>>> Long-term we really need a solution that lets you get expected results for a platform you don't have access to without committing code, e.g., the EWS archiving results for failing tests.
>>>>
>>>> Ojan
>>>> _______________________________________________
>>>> webkit-dev mailing list
>>>> webkit-dev at lists.webkit.org
>>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>