[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Dirk Pranke]

I don't know that there was consensus that every port had to be
updated at the same time; in fact Balazs said Qt and EFL already clear
the cache.

I think you should just land the change for Chromium and let others
update their ports as needed. The value in reduced flakiness and more
predictability outweighs anything else in my book. Test coverage that
you can't explain or rely on doesn't count for much to me.

-- Dirk

On Fri, Oct 26, 2012 at 1:20 AM, Ami Fischman <fischman at chromium.org> wrote:
> This thread stalled out because although there seemed to be majority
> agreement that hermetic/repeatable tests are a good thing, there was a
> requirement that all ports be updated to the new behavior at the same time,
> and I'm only competent to do the chromium DRT (see
> https://bugs.webkit.org/show_bug.cgi?id=93195 for details).
>
> Is anyone interested in stepping up and doing the equivalent (clear caches
> between tests) for the mac and/or gtk ports' DRTs?
>
>
> On Wed, Aug 8, 2012 at 2:35 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> On Wed, Aug 8, 2012 at 10:47 AM, Ojan Vafai <ojan at chromium.org> wrote:
>> > See https://bugs.webkit.org/show_bug.cgi?id=93195.
>> >
>> > media/W3C/video/networkState/networkState_during_progress.html and
>> > media/video-poster-blocked-by-willsendrequest.html are flaky on all
>> > platforms because they behave differently if the loaded resource is
>> > cached.
>> >
>> > Every time I've taken a stab at reducing test flakiness, I've come
>> > across at
>> > least a few tests that pass when run as part of the test suite, but fail
>> > when run by themselves (or in parallel) because they accidentally expect
>> > an
>> > image or something to be in the cache.
>> >
>> > I think it would make the tests more maintainable if we cleared the
>> > cache
>> > before each test run. This is *not* before each page load though. So
>> > tests
>> > that do multiple page loads will still test cross-navigation caching
>> > behavior.
>> >
>> > While it's true that we could one-off fix each of these tests, it's
>> > usually
>> > very time consuming to figure out that caching is the problem, that's
>> > assuming anyone takes the time to look into why the test is flaky in the
>> > first place.
>> >
>> > Any objections?
>> >
>>
>> Given that the way we run tests in parallel in NRWT means that
>> different processes get different lists of tests each time, it sounds
>> like we may be getting a fair amount of nondeterminism from the cache
>> not being cleared between tests. That seems bad, so I'm in favor of
>> clearing the cache :)
>>
>> -- Dirk
>
>