<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">I don’t think order of magnitudes tests ever quite worked JSC (it was added by Google so it obviously worked on V8). If adjusting the threshold was all that was needed to make the tests pass, I see no reason how to change it.</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">However, what you proposed will only reduce the likelihood of type I errors (false positives). You should also examine how it affects the likelihood of type II errors (false negative). You might want to make some API artificially O(n^2) and make sure the test starts failing, etc… You can probably look at the svn/git blame the change which added relevant tests in LayoutTests/perf/ and apply a reverse change locally to test that.</div> <br> <div class="bloop_sign" id="bloop_sign_1460528210226717952"><div>- R. Niwa</div></div><p class="airmail_on">On April 12, 2016 at 10:05:25 PM, Nikos Andronikos (<a href="mailto:nikos.andronikos@cisra.canon.com.au">nikos.andronikos@cisra.canon.com.au</a>) wrote:</p> <blockquote type="cite" class="clean_bq"><span><div><div></div><div>Hey all,
<br>
<br>We’ve noticed what we think is some flakiness with tests that attempt to
<br>determine the BigO characteristics (using magnitude-perf.js) of
<br>performance tests and would like some feedback from the community.
<br>
<br>In our particular case, we are seeing a failure on the iOS simulator of
<br>the following test:
<br>LayoutTests/perf/nested-combined-selectors.html
<br>
<br>See: https://webkit-queues.webkit.org/results/1107504
<br>From bug: https://bugs.webkit.org/show_bug.cgi?id=156096
<br>
<br>The result on the iOS sim is falling just short of being correlated to
<br>linear due to the tolerances chosen.
<br>
<br>The numbers for an example run are:
<br>rSquared 0.8817542004431037
<br>rSquaredXLog 0.44260332869587665
<br>rSquaredXYLog 0.8678643749155268
<br>rSquaredMax 0.8817542004431037
<br>
<br>
<br>And the relevant bit of code in magnitude-perf.js that determines
<br>correlation is:
<br>
<br> var bigO = Magnitude.INDETERMINATE;
<br>
<br> // FIXME: These numbers were chosen arbitrarily.
<br> // Are there a better value to check against? Do we need to check
<br>rSquaredMax?
<br> if (rSquared < 0.8 && rSquaredMax < 0.9)
<br> bigO = Magnitude.CONSTANT;
<br> else if (rSquaredMax > 0.9) {
<br> if (rSquared == rSquaredMax)
<br> bigO = Magnitude.LINEAR;
<br> else if (rSquaredXLog == rSquaredMax)
<br> bigO = Magnitude.LOGARITHMIC;
<br> else if (rSquaredXYLog == rSquaredMax)
<br> bigO = Magnitude.POLYNOMIAL;
<br> }
<br>
<br>
<br>So you can see we don’t quite pass the ‘else if (rSquaredMax 0.9)’ test
<br>and so the result is given as indeterminate.
<br>
<br>My feeling is that the numbers we are getting are correlated closely
<br>enough to Magnitude.LINEAR, and we should consider changing the values in
<br>magnitude-perf.js.
<br>
<br>So I’d like to propose that we lower the thresholds and change the following lines to check the slope of the linear correlation
<br>and the correlation coefficient:
<br>
<br>From:
<br> if (rSquared < 0.8 && rSquaredMax < 0.9)
<br> bigO = Magnitude.CONSTANT;
<br> else if (rSquaredMax 0.9) {
<br>To:
<br> if (rSquared > 0.7) {
<br> bigO = Magnitude.LINEAR;
<br> }
<br> else if (Math.abs(slope) < 1.0E-04)
<br> bigO = Magnitude.CONSTANT;
<br> else {
<br> // calculate and check correlation coefficient for rSquaredXLog and rSquaredXYLog
<br> }
<br>
<br>This check first tests the correlation coefficient (i.e. if linear order) – note the threshold is lowered to 0.7.
<br>Then it checks whether the performance is independent of the number of iterations/objects (i.e. a shallow slope).
<br>Then we calculate and check the other cases (i.e. polynomial or logarithmic).
<br>Failing all other checks, the order defaults to indeterminate.
<br>
<br>We are considering submitting a patch to make this change, but would like
<br>input from anyone who has experience with this test framework.
<br>
<br>Thanks,
<br>Nikos
<br>The information contained in this email message and any attachments may be confidential and may also be the subject to legal professional privilege. If you are not the intended recipient, any use, interference with, disclosure or copying of this material is unauthorised and prohibited. If you have received this email in error, please immediately advise the sender by return email and delete the information from your system.
<br>_______________________________________________<br>webkit-dev mailing list<br>webkit-dev@lists.webkit.org<br>https://lists.webkit.org/mailman/listinfo/webkit-dev<br></div></div></span></blockquote></body></html>