[webkit-reviews] review denied: [Bug 51272] commit-queue will report constant failures as flaky if other tests flake : [Attachment 77078] Patch

[bugzilla-daemon at webkit.org]

Adam Barth <abarth at webkit.org> has denied Eric Seidel <eric at webkit.org>'s
request for review:
Bug 51272: commit-queue will report constant failures as flaky if other tests
flake
https://bugs.webkit.org/show_bug.cgi?id=51272

Attachment 77078: Patch
https://bugs.webkit.org/attachment.cgi?id=77078&action=review

------- Additional Comments from Adam Barth <abarth at webkit.org>
View in context: https://bugs.webkit.org/attachment.cgi?id=77078&action=review

The test ordering code is too fragile.	As the project evolves, its going to be
subtly wrong.

> Tools/Scripts/webkitpy/common/net/layouttestresults.py:99
> +    # This is intended to match run-webkit-tests behavior.
> +    @classmethod
> +    def test_order_compare(cls, test1, test2):

Really?  I don't think this function is right.	What about WebSocket tests?

> Tools/Scripts/webkitpy/tool/bot/commitqueuetask.py:181
> +	   compare_result =
LayoutTestResults.test_order_compare(first_failures[-1], second_failures[-1])
> +	   if compare_result < 0:  # First run was shorter
> +	       return different_failures.difference(second_failures)
> +	   elif compare_result > 0:
> +	       return different_failures.difference(first_failures)

I don't know about this design.  You didn't like the idea of only reporting a
flak if the second run was all-success?