[webkit-dev] policy on perf/memory regressions?

[Stephanie Lewis]

The policy has never been clearly defined but in most cases follow this rule.  Upon discovering the perf. regression a P1 bug is filed.  It is expected that the bug will be a top priority, and handled in a timely manner.  With the exception of very large perf. regressions, we rarely roll patches out.  With very few exceptions (heavily discussed), it is never acceptable to check in something that will regress performance with no expectations of fixing it.

In general, when PLT regressions happen I often ask someone internally to look at it or work closely with the contributor to test.  It is rather unfair to ask someone to fix a regression they can't test.  While we strictly enforce the the zero regression policy internally, because many of the tests are painful to set up or not available externally we are not as strict with external contributions.  If Chromium has perf. tests they would like to share with us, I'd be willing to work with someone to learn those tests.

-- Stephanie

On Apr 28, 2010, at 3:30 PM, Ojan Vafai wrote:

> Do we have a policy on perf/memory regressions? I've been told that there's a "zero-regression" policy for page load tests. Is that actually the case? What about cases that are clear perf regressions that don't show up in PLT (e.g. they show up in one of the Chromium page load tests and/or on a real-world web page)? 
> 
> Perf and memory regressions seem much worse to me than test failures. Unlike test failures, there's isn't a binary right or wrong. If a patch regresses performance and then other patches come in that further improve/regress performance, it becomes nearly impossible to tell if the fix for the original patch actually addresses the entirety of the original regression.
> 
> The only case I can think of where perf regressions might be ok are:
> 1. The fix will be checked in reasonably soon (hours, not days/weeks). This matches our policy with failing tests.
> 2. We think there's no way to address the perf/memory regression and that the new functionality justifies it. This is extremely rare.
> 
> I'm asking because I've had considerable pushback dealing with a recent perf/memory regression on OS X that's been sitting in the tree for 3 weeks. I'm not linking to the bug in question because that issue is resolved. This is forward looking. I'd like to see us have a written down policy. Ideally, one day, we'll also have bots that run perf/memory tests and turn red when there is a regression.
> 
> Ojan
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev