[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Dirk Pranke]

On Mon, Oct 29, 2012 at 4:17 AM, Maciej Stachowiak <mjs at apple.com> wrote:
>
> On Oct 28, 2012, at 3:30 PM, Antti Koivisto <koivisto at iki.fi> wrote:
>
>> We could clear the cache between tests but run each test twice in a row. Second run will then happen with deterministically pre-populated cache. That would both make things more predictable and improve our test coverage for cached cases. Unfortunately it would also slow down testing significantly, though less than 2x.
>
> I actually really like this idea. Doing it this way would effectively run each test both completely uncached, and fully cached, which would be better test coverage than our current approach. Can we get an estimate on what this would cost if applied to our whole test suite? Could we do it for just a subset of the tests?
>
> (BTW I think this is better than the "virtual test suite" approach suggested by Dirk; running the test with all its resources cached from having loaded it immediately before is more reliable and better test coverage than running it later as part of some sequence that doesn't clear the cache.)
>
> Does anyone strongly object to this approach? It seems way better to me than other options discussed on this thread.
>

I would like to understand the proposal and thinking behind it a little better.

First, I'm not quite sure what the intent is here ... are you thinking
that maybe this would help ensure tests get more isolated environments
and yet we still exercise the cache all the time?

Second, today you could get close by saying NRWT --repeat-each=2
--batch-size=2; that would run each test twice in succession and then
restart DRT. Are you suggesting that we'd change something such that
we'd clear the cache rather than restarting DRT?

Third, it seems like it would probably introduce a *lot* of extra
redundant test-running. I would imagine the very large majority of our
tests are not sensitive to the cache contents at all and so running
them twice doesn't buy you anything. An advantage of the virtual test
suite approach is that you can specify which subsets of tests you'd
like to run; perhaps you could combine that with running them twice?

Lastly, it seems like this is focusing on the wrong thing; wouldn't it
be better to try and write specific tests to provoke different
configurations and code paths through the cache? Are you suggesting
we'd run all the tests twice as an interim measure until we had better
specific test suites?

-- Dirk

> Regards,
> Maciej
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev