[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Ami Fischman]

This thread stalled out because although there seemed to be majority
agreement that hermetic/repeatable tests are a good thing, there was a
requirement that all ports be updated to the new behavior at the same time,
and I'm only competent to do the chromium DRT (see
https://bugs.webkit.org/show_bug.cgi?id=93195 for details).

Is anyone interested in stepping up and doing the equivalent (clear caches
between tests) for the mac and/or gtk ports' DRTs?


On Wed, Aug 8, 2012 at 2:35 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> On Wed, Aug 8, 2012 at 10:47 AM, Ojan Vafai <ojan at chromium.org> wrote:
> > See https://bugs.webkit.org/show_bug.cgi?id=93195.
> >
> > media/W3C/video/networkState/networkState_during_progress.html and
> > media/video-poster-blocked-by-willsendrequest.html are flaky on all
> > platforms because they behave differently if the loaded resource is
> cached.
> >
> > Every time I've taken a stab at reducing test flakiness, I've come
> across at
> > least a few tests that pass when run as part of the test suite, but fail
> > when run by themselves (or in parallel) because they accidentally expect
> an
> > image or something to be in the cache.
> >
> > I think it would make the tests more maintainable if we cleared the cache
> > before each test run. This is *not* before each page load though. So
> tests
> > that do multiple page loads will still test cross-navigation caching
> > behavior.
> >
> > While it's true that we could one-off fix each of these tests, it's
> usually
> > very time consuming to figure out that caching is the problem, that's
> > assuming anyone takes the time to look into why the test is flaky in the
> > first place.
> >
> > Any objections?
> >
>
> Given that the way we run tests in parallel in NRWT means that
> different processes get different lists of tests each time, it sounds
> like we may be getting a fair amount of nondeterminism from the cache
> not being cleared between tests. That seems bad, so I'm in favor of
> clearing the cache :)
>
> -- Dirk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121026/b8809e0e/attachment.html>