<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 10/28/2012 08:25 PM, Ami Fischman
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAHuR8a-ynqR_TzK_diDLkBCO+pkUDEM4g3EiQ=owAz7=FVnQyQ@mail.gmail.com"
      type="cite">
      <div class="gmail_extra">
        <div class="gmail_quote">
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
            <div style="word-wrap:break-word">
              <div>
                <div class="im">
                  <blockquote type="cite">
                    <div class="gmail_extra">We can live in one of two
                      worlds:&nbsp;</div>
                    <div class="gmail_extra">1) LayoutTests that concern
                      themselves with specific network/loading concerns
                      need to use unique URLs to refer to static data;
                      or</div>
                    <div class="gmail_extra">2) DRT clears JS-visible
                      state between tests.</div>
                    <div class="gmail_extra">The pros/cons seem clear to
                      me:</div>
                    <div class="gmail_extra">Pro#1: loading/caching code
                      is coincidentally tested by (unknown) tests that
                      reuse URLs among themselves.</div>
                    <div class="gmail_extra">Con#1: requires additional
                      cognitive load for all webkit developers; the only
                      way to write a test that won't be affected by
                      future addition of unrelated tests is to use
                      unique URLs</div>
                    <div class="gmail_extra">
                      Pro#2: principle of least-surprise is maintained;
                      understanding DRT &amp; reading a test (and not
                      every other test) is enough to understand its
                      behavior</div>
                    <div class="gmail_extra">Con#2: loading/caching code
                      needs to be tested explicitly.</div>
                    <div class="gmail_extra">IMO (Pro#2 + -Con#1)
                      &gt;&gt; (Pro#1 + -Con#2).</div>
                    <div class="gmail_extra">
                      Are you saying you believe the inequality goes a
                      different way, or am I missing some other feature
                      of your thesis?</div>
                  </blockquote>
                </div>
              </div>
              <div>Yes, this is a fair description. </div>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div>I'm going to assume you mean that yes, you believe the
            inequality goes the other way: &nbsp;(Pro#2 + -Con#1) &lt;&lt;
            (Pro#1 + -Con#2)</div>
          <div>&nbsp;</div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
            <div style="word-wrap:break-word">
              <div>This accidental testing is not something to be
                neglected</div>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div>I'm not neglecting it, I'm evaluating its benefit to be
            less than its cost.</div>
          <div><br>
          </div>
          <div>To make concrete the cost/benefit tradeoff, would you add
            a random sleep() into DRT execution to detect timing-related
            bugs?</div>
          <div>It seems like a crazy thing to do, to me, but it would
            certainly catch timing-related bugs quite effectively.&nbsp;</div>
          <div>If you don't think we should do that, can you describe
            how you're evaluating cost/benefit in each of the cases and
            why you arrive at different conclusions?</div>
          <div><br>
          </div>
          <div>(of course, adding such random sleeps under
            default-disabled flag control for bug investigation could
            make a lot of sense; but here I'm talking about what we do
            on the bots &amp; by default)</div>
          <div>
            &nbsp;</div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
            <div style="word-wrap:break-word">
              <div>It's not humanly possible to have tests for
                everything in advance. </div>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div>Of course. &nbsp;But we should at least make it humanly
            possible to understand our tests as written :)</div>
          <div>Making understanding our tests not humanly possible isn't
            the way to make up for the not-humanly-possible nature of
            testing everything in every way.</div>
          <div>It just means we push off not knowing how much coverage
            we really have, and derive a false sense of security from
            the fact that bugs have been found in the past.</div>
          <div><br>
          </div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
            <div style="word-wrap:break-word">
              <div>I completely agree with Maciej's idea that we should
                think about ways to make non-deterministic failures
                easier to work with, so that they would lead to
                discovering the root cause more directly, and without
                the costs currently associated with it.<br>
              </div>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div>I have no problem with that, but I'm not sure how it
            relates to this thread unless one takes an XOR approach, in
            which case I guess I have low faith that the bigger problem
            Maciej highlights will be solved in a reasonable timeframe
            (weeks/months).</div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
            <div style="word-wrap:break-word">
              <div>
                <div>
                  <blockquote type="cite">
                    <div bgcolor="#FFFFFF" text="#000000">
                      <div class="im">
                        <blockquote type="cite">
                          <div>Memory allocator state. Computer's real
                            time clock. Hard drive's head position if
                            you have a spinning hard drive, or SSD
                            controller state if you have an SSD. HTTP
                            cookies. Should I continue the list?</div>
                        </blockquote>
                      </div>
                      <div class="im"><span style="">These things are
                          all outside of webkit.</span></div>
                    </div>
                  </blockquote>
                </div>
                <div bgcolor="#FFFFFF" text="#000000"><span style="">Yes,
                    they are outside WebKit, but not outside WebKit
                    control, if needed.</span></div>
                <div bgcolor="#FFFFFF" text="#000000"><span style="">Did
                    you intend that to be an objection?</span></div>
              </div>
            </div>
          </blockquote>
        </div>
      </div>
    </blockquote>
    <blockquote
cite="mid:CAHuR8a-ynqR_TzK_diDLkBCO+pkUDEM4g3EiQ=owAz7=FVnQyQ@mail.gmail.com"
      type="cite">
      <div class="gmail_extra">
        <div class="gmail_quote">
          <div><br>
          </div>
          <div>I imagine&nbsp;<span
              style="font-family:arial,sans-serif;font-size:13px">Balazs&nbsp;</span>was
            pointing out that you included items that are not JS-visible
            in an answer to my question about things that are
            JS-visible. &nbsp;But that was part of an earlier fork of this
            thread that went nowhere, so let's let it go.</div>
        </div>
      </div>
    </blockquote>
    <br>
    I was just meaning that it is not feasible to force every external
    dependency to reset it's state, neither we want it. We just trust in
    them. But the cache is in WebKit, and we can reset it's state. So
    either resetting the cache is a good or a bad idea, I think it has
    nothing to do with the fact that we cannot "reset" the OS and the
    hardware (and external libs of course).<br>
    <br>
  </body>
</html>