[Webkit-unassigned] [Bug 199054] New: [perf.webkit.org] When calculating a ratio for the summary page, use a rolling window for last available data instead of only the last available data point

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=199054

            Bug ID: 199054
           Summary: [perf.webkit.org] When calculating a ratio for the
                    summary page, use a rolling window for last available
                    data instead of only the last available data point
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: dean_johnson at apple.com

Right now when calculating a ratio for the summary page, we do the following:
current = "last 24 hours of non-baseline data" if any data exists in the last 24 hours, otherwise "last available data point"
baseline = "last 24 hours of baseline data" if any data exists in the last 24 hours, otherwise "last available data point"
ratio = (median(current) / median(baseline))

Since we set the "last 24 hours" window to the current time, and it can take hours between a commit landing, build products becoming available, and testing to occur, this 24 hour window will always have a margin that contains no data to evaluate. Additionally, using only a single data point when no data is available in the past 24 hours can lead to misrepresented results due to a single outlier run before a configuration stopped producing results.

I propose that we change the calculation to:
current = "last 24 hours of non-baseline data since the last available data point"
baseline = "last 24 hours of baseline data since the last available data point"
ratio = (median(current) / median(baseline))

Overall, I think this should improve the analysis of data when representing results on Summary pages, since it will take more data points into account.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20190620/1c53f3f1/attachment-0001.html>