[Webkit-unassigned] [Bug 51272] commit-queue will report constant failures as flaky if other tests flake

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=51272





--- Comment #9 from Eric Seidel <eric at webkit.org>  2010-12-20 21:26:30 PST ---
(In reply to comment #8)
> (From update of attachment 77078 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=77078&action=review
> 
> The test ordering code is too fragile.  As the project evolves, its going to be subtly wrong.

I agree, that's a risk.  However, if we shared this code with NRWT it would be fine. :)

> > Tools/Scripts/webkitpy/common/net/layouttestresults.py:99
> > +    # This is intended to match run-webkit-tests behavior.
> > +    @classmethod
> > +    def test_order_compare(cls, test1, test2):
> 
> Really?  I don't think this function is right.  What about WebSocket tests?

Again, should be shared if we're going to go this way.

> > Tools/Scripts/webkitpy/tool/bot/commitqueuetask.py:181
> > +        compare_result = LayoutTestResults.test_order_compare(first_failures[-1], second_failures[-1])
> > +        if compare_result < 0:  # First run was shorter
> > +            return different_failures.difference(second_failures)
> > +        elif compare_result > 0:
> > +            return different_failures.difference(first_failures)
> 
> I don't know about this design.  You didn't like the idea of only reporting a flak if the second run was all-success?

Oh, that's fine.  Certainly simpler.  It just produces this state where we retry w/o reporting any flaky tests which is OK just less than idea.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.