[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Ami Fischman]

On Sun, Oct 28, 2012 at 8:26 PM, Alexey Proskuryakov <ap at webkit.org> wrote:

> when you run the same set of tests over and over, you get the same
> behavior.
>

Not the way n-w-r-t does it; in particular the time each test takes to
finish affects which process it gets run in, which gets you flakiness.
Which is what started me down this path in the first place.

I think that you are hugely overstating this. Adding a random query to a
> URL does not make a test incomprehensible.
>

Of course not.  The problem is a test that does *not* randomize its URLs
may pass for a long time, and appear completely comprehensible, until an
entirely unrelated test is added that happens to use the same URL, and then
make the first test flaky.
My point is that any state that is shared between tests needs to be
known/understood to/by all test writers.


> A good test is usable in the aforementioned scenarios, and thus does not
> need special tricks in WebKitTestRunner.
>

...and yet we expect a repro'ing browser to not be running
extensions/plugins/etc that might mess with the repro case.
Clearing cookies/cache is probably the most common instruction included
with repro requests, followed by trying the case in a new/clean
profile/installation.

> I completely agree with Maciej's idea that we should think about ways to
>> make non-deterministic failures easier to work with, so that they would
>> lead to discovering the root cause more directly, and without the costs
>> currently associated with it.
>>
> I have no problem with that, but I'm not sure how it relates to this
> thread unless one takes an XOR approach, in which case I guess I have low
> faith that the bigger problem Maciej highlights will be solved in a
> reasonable timeframe (weeks/months).
>
> We have all the time in the world. There is no pressing problem that must
> be solved in months.
>

My attention span is not nearly as long as yours apparently is.
Gating fixing what appears to be an obvious bug to me (and many others) on
fundamental improvements to testing methodology that might take years is
madness.

Cheers,
-a
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121029/558bb0ac/attachment.html>