[webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

[Alexey Proskuryakov]

28.10.2012, в 12:25, Ami Fischman <fischman at chromium.org> написал(а):
> 
> To make concrete the cost/benefit tradeoff, would you add a random sleep() into DRT execution to detect timing-related bugs?
> It seems like a crazy thing to do, to me, but it would certainly catch timing-related bugs quite effectively. 
> If you don't think we should do that, can you describe how you're evaluating cost/benefit in each of the cases and why you arrive at different conclusions?

A huge difference between this idea and caches is that caches are deterministic - when you run the same set of tests over and over, you get the same behavior. So, you can find a small subset of tests that trigger a bug, or add logging that records state changes, and you can fix a bug in WebCore.

> Of course.  But we should at least make it humanly possible to understand our tests as written :)
> Making understanding our tests not humanly possible isn't the way to make up for the not-humanly-possible nature of testing everything in every way.
> It just means we push off not knowing how much coverage we really have, and derive a false sense of security from the fact that bugs have been found in the past.

I think that you are hugely overstating this. Adding a random query to a URL does not make a test incomprehensible.

Let me offer you another angle to look at this. As you said, ability to reason about tests is a priority. But reasoning does not include looking at test results alone. One needs to reason about tests in other cases, too:

- running a test in browser to update and/or debug it (e.g. when spec changes);
- sending a test to developers of another browser as part of a bug report.

If we have tests that strongly depend on tricks inside WebKitTestRunner, this becomes virtually impossible. You just reload the test in browser, or open it in two tabs, or load it after some other test, and it magically changes results.

A good test is usable in the aforementioned scenarios, and thus does not need special tricks in WebKitTestRunner.

> I completely agree with Maciej's idea that we should think about ways to make non-deterministic failures easier to work with, so that they would lead to discovering the root cause more directly, and without the costs currently associated with it.
> 
> I have no problem with that, but I'm not sure how it relates to this thread unless one takes an XOR approach, in which case I guess I have low faith that the bigger problem Maciej highlights will be solved in a reasonable timeframe (weeks/months).

We have all the time in the world. There is no pressing problem that must be solved in months.

>>> Memory allocator state. Computer's real time clock. Hard drive's head position if you have a spinning hard drive, or SSD controller state if you have an SSD. HTTP cookies. Should I continue the list?
>> 
>> These things are all outside of webkit.
> 
> Yes, they are outside WebKit, but not outside WebKit control, if needed.
> Did you intend that to be an objection?
> 
> I imagine Balazs was pointing out that you included items that are not JS-visible in an answer to my question about things that are JS-visible.  But that was part of an earlier fork of this thread that went nowhere, so let's let it go.

All these things are visible in that they can directly affect tests. Memory layout affects how you write tests for certain crashers, and the rest affects performance tests (we attempted to have regression tests for fixes that change asymptotic complexity of algorithms, and they were very flaky).

- WBR, Alexey Proskuryakov

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121028/0943c3c2/attachment.html>