I don't believe that the test was checked in a flaky state. It was solid for a long time and then something happened...<div><br>I'll try to add more logging to this test this evening (after my turn at helping chromium stay up to date with WebKit is over).</div>
<div><br></div><div>I'll ping Drew about the other test.</div><div><br></div><div>Dave</div><div><br></div><div><br><div class="gmail_quote">On Mon, Sep 28, 2009 at 4:40 PM, Eric Seidel <span dir="ltr"><<a href="mailto:eric@webkit.org">eric@webkit.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">On Fri, Sep 25, 2009 at 2:42 PM, Maciej Stachowiak <<a href="mailto:mjs@apple.com">mjs@apple.com</a>> wrote:<br>
> I like Dave Levin's idea that the first action should be to instrument the<br>
> tests so we can find out why they intermittently fail. Especially if the<br>
> failure is reproducible on the bots but not on developer systems.<br>
<br>
</div>I like this idea too. I don't like the reality that flakey tests are<br>
a burden on all developers caused by one.<br>
<div class="im"><br>
> Using the<br>
> skip list should be a last resort, because that hides the failure instead of<br>
> helping us diangose the cause.<br>
<br>
</div>I (respectfully) disagree. I think we shouldn't be so afraid to skip<br>
tests. We don't allow people to check in compiles which fail. We<br>
don't allow people to check in tests which fail on other platforms<br>
(without skipping them) or on every other run. Why should we allow<br>
people to check in tests which fail every 10 runs? Or worse, why<br>
should we leave a known flakey test checked in/un-attended which fails<br>
every 10 runs?<br>
<br>
If we can't easily roll-out the failing tests (or the commit which<br>
cause them to start failing), we should skip them to keep the bots (a<br>
shared resource) green, so as not to block other work on the project.<br>
No?<br>
<br>
I very much like WebKit's "everyone is responsible for the whole<br>
project" culture, but I disagree that the burden of diagnosis should<br>
be on the person trying to make a completely unrelated checkin (as is<br>
the case when we leave flakey tests enabled in the tree).<br>
<br>
-eric<br>
<br>
p.s. I now have two "skipping flakey tests" changes up for review:<br>
<div class="im"><a href="https://bugs.webkit.org/show_bug.cgi?id=29322" target="_blank">https://bugs.webkit.org/show_bug.cgi?id=29322</a><br>
</div><a href="https://bugs.webkit.org/show_bug.cgi?id=29344" target="_blank">https://bugs.webkit.org/show_bug.cgi?id=29344</a><br>
<div><div></div><div class="h5">_______________________________________________<br>
webkit-dev mailing list<br>
<a href="mailto:webkit-dev@lists.webkit.org">webkit-dev@lists.webkit.org</a><br>
<a href="http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev" target="_blank">http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev</a><br>
</div></div></blockquote></div><br></div>