[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Ami Fischman]

>
> We can live in one of two worlds:
> 1) LayoutTests that concern themselves with specific network/loading
> concerns need to use unique URLs to refer to static data; or
> 2) DRT clears JS-visible state between tests.
> The pros/cons seem clear to me:
> Pro#1: loading/caching code is coincidentally tested by (unknown) tests
> that reuse URLs among themselves.
> Con#1: requires additional cognitive load for all webkit developers; the
> only way to write a test that won't be affected by future addition of
> unrelated tests is to use unique URLs
> Pro#2: principle of least-surprise is maintained; understanding DRT &
> reading a test (and not every other test) is enough to understand its
> behavior
> Con#2: loading/caching code needs to be tested explicitly.
> IMO (Pro#2 + -Con#1) >> (Pro#1 + -Con#2).
> Are you saying you believe the inequality goes a different way, or am I
> missing some other feature of your thesis?
>
> Yes, this is a fair description.
>

I'm going to assume you mean that yes, you believe the inequality goes the
other way:  (Pro#2 + -Con#1) << (Pro#1 + -Con#2)


> This accidental testing is not something to be neglected
>

I'm not neglecting it, I'm evaluating its benefit to be less than its cost.

To make concrete the cost/benefit tradeoff, would you add a random sleep()
into DRT execution to detect timing-related bugs?
It seems like a crazy thing to do, to me, but it would certainly catch
timing-related bugs quite effectively.
If you don't think we should do that, can you describe how you're
evaluating cost/benefit in each of the cases and why you arrive at
different conclusions?

(of course, adding such random sleeps under default-disabled flag control
for bug investigation could make a lot of sense; but here I'm talking about
what we do on the bots & by default)


> It's not humanly possible to have tests for everything in advance.
>

Of course.  But we should at least make it humanly possible to understand
our tests as written :)
Making understanding our tests not humanly possible isn't the way to make
up for the not-humanly-possible nature of testing everything in every way.
It just means we push off not knowing how much coverage we really have, and
derive a false sense of security from the fact that bugs have been found in
the past.

I completely agree with Maciej's idea that we should think about ways to
> make non-deterministic failures easier to work with, so that they would
> lead to discovering the root cause more directly, and without the costs
> currently associated with it.
>

I have no problem with that, but I'm not sure how it relates to this thread
unless one takes an XOR approach, in which case I guess I have low faith
that the bigger problem Maciej highlights will be solved in a reasonable
timeframe (weeks/months).

> Memory allocator state. Computer's real time clock. Hard drive's head
> position if you have a spinning hard drive, or SSD controller state if you
> have an SSD. HTTP cookies. Should I continue the list?
>
> These things are all outside of webkit.
>
> Yes, they are outside WebKit, but not outside WebKit control, if needed.
> Did you intend that to be an objection?
>

I imagine Balazs was pointing out that you included items that are not
JS-visible in an answer to my question about things that are JS-visible.
 But that was part of an earlier fork of this thread that went nowhere, so
let's let it go.

Cheers,
-a
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121028/2fc27e2d/attachment.html>