[Webkit-unassigned] [Bug 199050] New: [perf.webkit.org] Update summary page calculations to use mean instead of median

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=199050

            Bug ID: 199050
           Summary: [perf.webkit.org] Update summary page calculations to
                    use mean instead of median
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: dean_johnson at apple.com

Median of a set of data points can see higher variation in each view of the summary page than when using mean. This means a benchmark + platform's status may see major shifts as new data comes in, or when a benchmark produces bi-modal results. The downside to using mean is that any major outliers will impact the score a benchmark sees for a longer period of time.

When doing comparisons between both median and mean-based calculations locally, results did not appear to shift wildly (i.e. from regressed => progressed or progressed => regressed). That said, I did see the resolution of a major regression cause incremental decreases to a single calculation as more data points came in, which seems desirable over an abrupt shift.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20190620/4ae068b7/attachment-0001.html>