[Webkit-unassigned] [Bug 145762] New: JetStream should have a more rational story for jitter-oriented latency tests

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=145762

            Bug ID: 145762
           Summary: JetStream should have a more rational story for
                    jitter-oriented latency tests
    Classification: Unclassified
           Product: WebKit
           Version: 528+ (Nightly build)
          Hardware: All
                OS: All
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: fpizlo at apple.com

Currently we have some latency tests that are meant to measure jitter.  They do this by computing the RMS.  But the RMS is a pretty bad metric.  The thing that it rewards isn't really the thing that you'd want your browser to do.  These RMS-based tests involve taking the geomean of the RMS of some samples and the sample average.  The lower the geomean, the better (in the JetStream harness we then invert the scores so that higher is better, but let's ignore that for this discussion and assume that lower is better).  Here's an example of how this can go bad.  A browser that always computes a task in some horribly long time (say, 1000ms) but never varies that time will perform better than a browser that usually computes the task super quickly (say, 10ms) and sometimes just a little bit less quickly (say, 15ms).  The former browser will have an RMS of 0 and an average of 1000.  The latter will have a RMS somewhere around 3.5 and an average of 12.5 (assuming equal probability of 10ms and 

JetStream should not have this pathology.  The right way of avoiding it is to replace RMS with some other metric of how bad things get.  A good metric is the average of the worst percentile.  The worst 1% or the worst 5% would be good things to average.  This will catch cases where the VM jittered due to JIT or GC, but it never have the pathology that we end up giving the better score to a VM whose best case is worst than another VM's worst case.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-unassigned/attachments/20150608/30cfaeef/attachment.html>