[Webkit-unassigned] [Bug 51272] commit-queue will report constant failures as flaky if other tests flake

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=51272


Adam Barth <abarth at webkit.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #77078|review?, commit-queue?      |review-
               Flag|                            |




--- Comment #8 from Adam Barth <abarth at webkit.org>  2010-12-20 21:00:28 PST ---
(From update of attachment 77078)
View in context: https://bugs.webkit.org/attachment.cgi?id=77078&action=review

The test ordering code is too fragile.  As the project evolves, its going to be subtly wrong.

> Tools/Scripts/webkitpy/common/net/layouttestresults.py:99
> +    # This is intended to match run-webkit-tests behavior.
> +    @classmethod
> +    def test_order_compare(cls, test1, test2):

Really?  I don't think this function is right.  What about WebSocket tests?

> Tools/Scripts/webkitpy/tool/bot/commitqueuetask.py:181
> +        compare_result = LayoutTestResults.test_order_compare(first_failures[-1], second_failures[-1])
> +        if compare_result < 0:  # First run was shorter
> +            return different_failures.difference(second_failures)
> +        elif compare_result > 0:
> +            return different_failures.difference(first_failures)

I don't know about this design.  You didn't like the idea of only reporting a flak if the second run was all-success?

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.