[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Alexey Proskuryakov]

26.10.2012, в 11:04, Antti Koivisto <koivisto at iki.fi> написал(а):

> The reality is that this "test coverage" today shows up as flakiness and so is ignored anyway, meaning we don't actually have useful coverage here.  Even when flakiness is investigated, the "fix" is to cache-bust using unique URL params, which just means we "lose" the coverage you describe for that test, anyway.

I think that this is the real issue here. Test flakiness is very important to investigate, this often leads to discovery of bad bugs, including security ones. The phrase "flaky test" often misplaces the blame.

> When making cache related changes I have frequently found bugs from my patches because some seemingly random test started failing and I investigated. Without the test coverage some of those bugs would probably now be in the tree.

I agree with Antti. Finding regressions is what tests are for, and it would be difficult to make enough explicit tests to compensate for such loss of coverage. It would certainly be very unfortunate to lose test coverage without even an attempt to compensate for that.


- WBR, Alexey Proskuryakov


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121026/81e2fee9/attachment.html>