[webkit-dev] policy on perf/memory regressions?

[Ojan Vafai]

Do we have a policy on perf/memory regressions? I've been told that there's
a "zero-regression" policy for page load tests. Is that actually the case?
What about cases that are clear perf regressions that don't show up in PLT
(e.g. they show up in one of the Chromium page load tests and/or on a
real-world web page)?

Perf and memory regressions seem much worse to me than test failures. Unlike
test failures, there's isn't a binary right or wrong. If a patch regresses
performance and then other patches come in that further improve/regress
performance, it becomes nearly impossible to tell if the fix for the
original patch actually addresses the entirety of the original regression.

The only case I can think of where perf regressions might be ok are:
1. The fix will be checked in reasonably soon (hours, not days/weeks). This
matches our policy with failing tests.
2. We think there's no way to address the perf/memory regression and that
the new functionality justifies it. This is extremely rare.

I'm asking because I've had considerable pushback dealing with a recent
perf/memory regression on OS X that's been sitting in the tree for 3 weeks.
I'm not linking to the bug in question because that issue is resolved. This
is forward looking. I'd like to see us have a written down policy. Ideally,
one day, we'll also have bots that run perf/memory tests and turn red when
there is a regression.

Ojan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100428/129a32c7/attachment.html>