[Webkit-unassigned] [Bug 43253] SunSpider run times are not reproducible

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=43253





--- Comment #2 from Paul Biggar <pbiggar at mozilla.com>  2010-08-06 16:43:02 PST ---
I've seen large variability with even 1000 runs. I run it on Linux, where I am told variability is worse.

I don't believe the number reported on the TOTAL line is accurate. It seems to be based on the difference between the two run-times, rather than the sum of the differences across all the benchmarks. As a simple example, 3bit-bits-in-byte benchmark runs in either 0ms or 1ms for me, and has massive variability, but that is completely masked by using the TOTAL value.

I wrote https://bug580532.bugzilla.mozilla.org/attachment.cgi?id=459618 as a better measure of variability. It's not perfect, but it allows you see how variable a test run is.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.