[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Alexey Proskuryakov]

26.10.2012, в 14:57, Dirk Pranke <dpranke at chromium.org> написал(а):

> Perhaps a slight variant of this is that we can agree to make the
> changes on the Chromium port to clear the cache (much like the Qt and
> EFL ports already do), and you can continue to not clear the cache on
> the Apple Mac port until you feel comfortable that you've added
> additional tests?

This means that when someone introduces flakiness into resource caching, it will be only seen on Apple Mac bots. How is this good for anyone? I personally find this unacceptable, as this will reduce usefulness of Apple Mac bots.

The whole idea to clear cache between tests seems very arbitrary to me. There are lot of things remaining in the process across tests runs, and I'm not sure why you are picking on the one with the least explicit test coverage.

Historically, test flakiness appears to increase whenever we do anything to address it without actual investigation of the root cause. Not long ago, we could run tests without re-running flaky tests, and get 100% pass. Now, we have many more flaky tests, re-run them, but flakiness remains even after second run. I don't think that this is a result of project scale change - I think that this is a result of the desire to get green bots without doing real WebCore work to fix underlying bugs.

- WBR, Alexey Proskuryakov