<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 10/28/2012 08:25 PM, Ami Fischman
wrote:<br>
</div>
<blockquote
cite="mid:CAHuR8a-ynqR_TzK_diDLkBCO+pkUDEM4g3EiQ=owAz7=FVnQyQ@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>
<div class="im">
<blockquote type="cite">
<div class="gmail_extra">We can live in one of two
worlds: </div>
<div class="gmail_extra">1) LayoutTests that concern
themselves with specific network/loading concerns
need to use unique URLs to refer to static data;
or</div>
<div class="gmail_extra">2) DRT clears JS-visible
state between tests.</div>
<div class="gmail_extra">The pros/cons seem clear to
me:</div>
<div class="gmail_extra">Pro#1: loading/caching code
is coincidentally tested by (unknown) tests that
reuse URLs among themselves.</div>
<div class="gmail_extra">Con#1: requires additional
cognitive load for all webkit developers; the only
way to write a test that won't be affected by
future addition of unrelated tests is to use
unique URLs</div>
<div class="gmail_extra">
Pro#2: principle of least-surprise is maintained;
understanding DRT & reading a test (and not
every other test) is enough to understand its
behavior</div>
<div class="gmail_extra">Con#2: loading/caching code
needs to be tested explicitly.</div>
<div class="gmail_extra">IMO (Pro#2 + -Con#1)
>> (Pro#1 + -Con#2).</div>
<div class="gmail_extra">
Are you saying you believe the inequality goes a
different way, or am I missing some other feature
of your thesis?</div>
</blockquote>
</div>
</div>
<div>Yes, this is a fair description. </div>
</div>
</blockquote>
<div><br>
</div>
<div>I'm going to assume you mean that yes, you believe the
inequality goes the other way: (Pro#2 + -Con#1) <<
(Pro#1 + -Con#2)</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>This accidental testing is not something to be
neglected</div>
</div>
</blockquote>
<div><br>
</div>
<div>I'm not neglecting it, I'm evaluating its benefit to be
less than its cost.</div>
<div><br>
</div>
<div>To make concrete the cost/benefit tradeoff, would you add
a random sleep() into DRT execution to detect timing-related
bugs?</div>
<div>It seems like a crazy thing to do, to me, but it would
certainly catch timing-related bugs quite effectively. </div>
<div>If you don't think we should do that, can you describe
how you're evaluating cost/benefit in each of the cases and
why you arrive at different conclusions?</div>
<div><br>
</div>
<div>(of course, adding such random sleeps under
default-disabled flag control for bug investigation could
make a lot of sense; but here I'm talking about what we do
on the bots & by default)</div>
<div>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>It's not humanly possible to have tests for
everything in advance. </div>
</div>
</blockquote>
<div><br>
</div>
<div>Of course. But we should at least make it humanly
possible to understand our tests as written :)</div>
<div>Making understanding our tests not humanly possible isn't
the way to make up for the not-humanly-possible nature of
testing everything in every way.</div>
<div>It just means we push off not knowing how much coverage
we really have, and derive a false sense of security from
the fact that bugs have been found in the past.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>I completely agree with Maciej's idea that we should
think about ways to make non-deterministic failures
easier to work with, so that they would lead to
discovering the root cause more directly, and without
the costs currently associated with it.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I have no problem with that, but I'm not sure how it
relates to this thread unless one takes an XOR approach, in
which case I guess I have low faith that the bigger problem
Maciej highlights will be solved in a reasonable timeframe
(weeks/months).</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>
<div>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000">
<div class="im">
<blockquote type="cite">
<div>Memory allocator state. Computer's real
time clock. Hard drive's head position if
you have a spinning hard drive, or SSD
controller state if you have an SSD. HTTP
cookies. Should I continue the list?</div>
</blockquote>
</div>
<div class="im"><span style="">These things are
all outside of webkit.</span></div>
</div>
</blockquote>
</div>
<div bgcolor="#FFFFFF" text="#000000"><span style="">Yes,
they are outside WebKit, but not outside WebKit
control, if needed.</span></div>
<div bgcolor="#FFFFFF" text="#000000"><span style="">Did
you intend that to be an objection?</span></div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<blockquote
cite="mid:CAHuR8a-ynqR_TzK_diDLkBCO+pkUDEM4g3EiQ=owAz7=FVnQyQ@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>I imagine <span
style="font-family:arial,sans-serif;font-size:13px">Balazs </span>was
pointing out that you included items that are not JS-visible
in an answer to my question about things that are
JS-visible. But that was part of an earlier fork of this
thread that went nowhere, so let's let it go.</div>
</div>
</div>
</blockquote>
<br>
I was just meaning that it is not feasible to force every external
dependency to reset it's state, neither we want it. We just trust in
them. But the cache is in WebKit, and we can reset it's state. So
either resetting the cache is a good or a bad idea, I think it has
nothing to do with the fact that we cannot "reset" the OS and the
hardware (and external libs of course).<br>
<br>
</body>
</html>