[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Ami Fischman]

Should we add random sleeps to DRT?  It'll certainly help find some
regressions (and even security bugs).
Of course the down-side is that it makes tests non-repeatable and difficult
to reason about.

I'm baffled by your priorities and don't know how to continue this
conversation productively.  Sorry.

Cheers,
-a

On Fri, Oct 26, 2012 at 12:43 PM, Alexey Proskuryakov <ap at webkit.org> wrote:

>
> 26.10.2012, в 11:04, Antti Koivisto <koivisto at iki.fi> написал(а):
>
> The reality is that this "test coverage" today shows up as flakiness and
>> so is ignored anyway, meaning we don't actually have useful coverage here.
>>  Even when flakiness is investigated, the "fix" is to cache-bust using
>> unique URL params, which just means we "lose" the coverage you describe for
>> that test, anyway.
>>
>
> I think that this is the real issue here. Test flakiness is very important
> to investigate, this often leads to discovery of bad bugs, including
> security ones. The phrase "flaky test" often misplaces the blame.
>
> When making cache related changes I have frequently found bugs from my
> patches because some seemingly random test started failing and I
> investigated. Without the test coverage some of those bugs would probably
> now be in the tree.
>
>
> I agree with Antti. Finding regressions is what tests are for, and it
> would be difficult to make enough explicit tests to compensate for such
> loss of coverage. It would certainly be very unfortunate to lose test
> coverage without even an attempt to compensate for that.
>
>
> - WBR, Alexey Proskuryakov
>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121026/595b1f30/attachment.html>