[webkit-dev] Introducing run-perf-tests and Adding Performance Bots

[Ojan Vafai]

Very exciting!

On Thu, Jan 19, 2012 at 3:20 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

> I didn't merge it into run-webkit-tests because performance tests don't
> pass/fail but instead give us some values that fluctuate over time. While
> Chromium takes an approach to hard-code the rage of acceptable values, such
> an approach has a high maintenance cost and prone to problems such as
> having to increase the range periodically as the score slowly degrades over
> time. Also, as you can see on Chromium perf bots<http://build.chromium.org/p/chromium.perf/console>,
> the test results tend to fluctuate a lot so hard-coding a tight range of
> acceptable value is tricky.
>

While this isn't perfect, I still think it's worth doing. Turning the bot
red when a performance test fails badly is helpful for finding and
reverting regressions quickly, which in turn helps identify smaller
regressions more easily (large regressions mask smaller ones). This doesn't
help with small regressions as much because they're often in the range of
acceptable values.

In either case, we have to get the bots running the tests and work on
getting reliable data first.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120120/0133e82f/attachment.html>