[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Brian Holt]

I'm happy to take a look at clearing the cache for the GTK+ port, but I
won't be able to do anything for mac unfortunately.

 

Brian

 

From: webkit-dev-bounces at lists.webkit.org
[mailto:webkit-dev-bounces at lists.webkit.org] On Behalf Of Ami Fischman
Sent: 26 October 2012 09:20
To: Dirk Pranke
Cc: Adam Barth; WebKit Development
Subject: Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of
each test?

 

This thread stalled out because although there seemed to be majority
agreement that hermetic/repeatable tests are a good thing, there was a
requirement that all ports be updated to the new behavior at the same time,
and I'm only competent to do the chromium DRT (see
https://bugs.webkit.org/show_bug.cgi?id=93195 for details).

 

Is anyone interested in stepping up and doing the equivalent (clear caches
between tests) for the mac and/or gtk ports' DRTs?

 

On Wed, Aug 8, 2012 at 2:35 PM, Dirk Pranke <dpranke at chromium.org> wrote:

On Wed, Aug 8, 2012 at 10:47 AM, Ojan Vafai <ojan at chromium.org> wrote:

> See https://bugs.webkit.org/show_bug.cgi?id=93195.
>
> media/W3C/video/networkState/networkState_during_progress.html and
> media/video-poster-blocked-by-willsendrequest.html are flaky on all
> platforms because they behave differently if the loaded resource is
cached.
>
> Every time I've taken a stab at reducing test flakiness, I've come across
at
> least a few tests that pass when run as part of the test suite, but fail
> when run by themselves (or in parallel) because they accidentally expect
an
> image or something to be in the cache.
>
> I think it would make the tests more maintainable if we cleared the cache
> before each test run. This is *not* before each page load though. So tests
> that do multiple page loads will still test cross-navigation caching
> behavior.
>
> While it's true that we could one-off fix each of these tests, it's
usually
> very time consuming to figure out that caching is the problem, that's
> assuming anyone takes the time to look into why the test is flaky in the
> first place.
>
> Any objections?
>

Given that the way we run tests in parallel in NRWT means that
different processes get different lists of tests each time, it sounds
like we may be getting a fair amount of nondeterminism from the cache
not being cleared between tests. That seems bad, so I'm in favor of
clearing the cache :)

-- Dirk

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121026/94a3d5a7/attachment.html>