[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Alexey Proskuryakov]

27.10.2012, в 20:47, Ami Fischman <fischman at chromium.org> написал(а):

> There are lot of things remaining in the process across tests runs
> 
> What "things" remain in the process across test runs that are visible to DRT/JS?

Memory allocator state. Computer's real time clock. Hard drive's head position if you have a spinning hard drive, or SSD controller state if you have an SSD. HTTP cookies. Should I continue the list?

> As I've said before in this thread, it seems axiomatic to me that tests can only be reasoned about if they run in a pristine environment.

This is an empty statement. A computer always provides you with a "pristine environment" until its RAM or other storage starts randomly failing. I would agree that tests would become useless if ran on a machine with faulty RAM. But people working on the project have successfully reasoned about flaky test failures many times in the past.

> This is why we TestShell::resetTestController(); so that a given test passes or fails the same way regardless of what other tests have run in the same process earlier.  Given that we *do* reset the execution environment between tests, it seems arbitrary (and unworkable) to *not* reset the cache.


I don't think that pure logic can prove the need. As mentioned before, cache is just an entirely arbitrary target from this point of view.

We do reset preferences that are temporarily changed by tests. This is basically modeled on user expectations - changing a preference is expected to change how your browser behaves, so it's OK for tests to depend on that. But visiting site A is not expected to affect behavior on site B, even though cache state was affected by site A.

- WBR, Alexey Proskuryakov

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121027/abd3fec2/attachment.html>