Re: [webkit-changes] [24907] trunk
On Aug 7, 2007, at 2:15 PM, antti@webkit.org wrote:
LayoutTests:
- added performance test. With debug build on MBP this takes about 1.5s to run.
* fast/block/basic/stress-shallow-nested-expected.txt: Added. * fast/block/basic/stress-shallow-nested.html: Added. (emphases mine). Nothing about that makes sense to me.
On 8/7/07, Mitz Pettel <opendarwin.org@mitzpettel.com> wrote:
On Aug 7, 2007, at 2:15 PM, antti@webkit.org wrote: - added performance test. With debug build on MBP this takes about 1.5s to run.
* fast/block/basic/stress-shallow-nested-expected.txt: Added. * fast/block/basic/stress-shallow-nested.html: Added.
(emphases mine). Nothing about that makes sense to me.
Are you objecting to testing performance regressions as part of the test suite in general or particular method here? I'm open to suggestions. It does not take long to run (300ms on release, 1.5s on debug MBP) so I thought it would be appropriate for automatic test. There are similar cases in the suite already. It needs to have non-zero execution time so that O(n^2) nature of the bug shows up in testable way. antti
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
On Aug 7, 2007, at 5:22 PM, Antti Koivisto wrote:
On 8/7/07, Mitz Pettel <opendarwin.org@mitzpettel.com> wrote:
On Aug 7, 2007, at 2:15 PM, antti@webkit.org wrote: - added performance test. With debug build on MBP this takes about 1.5s to run.
* fast/block/basic/stress-shallow-nested-expected.txt: Added. * fast/block/basic/stress-shallow-nested.html: Added.
(emphases mine). Nothing about that makes sense to me.
Are you objecting to testing performance regressions as part of the test suite in general or particular method here? I'm open to suggestions.
Hi Antti, I do not understand the purpose of the test and how it is supposed to serve that purpose (two of the reasons that I am baffled are that it is my understanding that there are facilities to track performance within fractions of a percent, and that I am not aware of other tests that have use elapsed time internally as their success criterion).
It does not take long to run (300ms on release, 1.5s on debug MBP) so I thought it would be appropriate for automatic test. There are similar cases in the suite already. It needs to have non-zero execution time so that O(n^2) nature of the bug shows up in testable way.
On the computer I run the tests on, that single test alone takes roughly as much time as all the other 229 tests in fast/block together.
Another option for this test is to make it a PLT, like the nesting PLT that Hyatt recently checked in. Geoff On Aug 7, 2007, at 7:44 AM, Mitz Pettel wrote:
On Aug 7, 2007, at 5:22 PM, Antti Koivisto wrote:
On 8/7/07, Mitz Pettel <opendarwin.org@mitzpettel.com> wrote:
On Aug 7, 2007, at 2:15 PM, antti@webkit.org wrote: - added performance test. With debug build on MBP this takes about 1.5s to run.
* fast/block/basic/stress-shallow-nested-expected.txt: Added. * fast/block/basic/stress-shallow-nested.html: Added.
(emphases mine). Nothing about that makes sense to me.
Are you objecting to testing performance regressions as part of the test suite in general or particular method here? I'm open to suggestions.
Hi Antti,
I do not understand the purpose of the test and how it is supposed to serve that purpose (two of the reasons that I am baffled are that it is my understanding that there are facilities to track performance within fractions of a percent, and that I am not aware of other tests that have use elapsed time internally as their success criterion).
It does not take long to run (300ms on release, 1.5s on debug MBP) so I thought it would be appropriate for automatic test. There are similar cases in the suite already. It needs to have non-zero execution time so that O(n^2) nature of the bug shows up in testable way.
On the computer I run the tests on, that single test alone takes roughly as much time as all the other 229 tests in fast/block together. _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
On 8/7/07, Mitz Pettel <opendarwin.org@mitzpettel.com> wrote:
Hi Antti,
I do not understand the purpose of the test and how it is supposed to serve that purpose (two of the reasons that I am baffled are that it is my understanding that there are facilities to track performance within fractions of a percent, and that I am not aware of other tests that have use elapsed time internally as their success criterion).
In this case a particular narrow aspect of performance was testable in way that can be executed as part of the automatic suite. I can find only one directly timing related case, LayoutTests/fast/forms/slow-click.html (thought <video> tests in branch have a ton more). I suspect number of others are timing dependent indirectly.
On the computer I run the tests on, that single test alone takes roughly as much time as all the other 229 tests in fast/block together.
It does? That is suprising. For me fast/block takes 8.2s, 1.5s is spent in this particular test. Fast/block is generally one of the fastest sets, http set takes 50s for 184 tests, several of which are slower than this. But yes, I should probably remove the test since it has so much impact on slower systems. antti
On Aug 7, 2007, at 7:22 AM, Antti Koivisto wrote:
On 8/7/07, Mitz Pettel <opendarwin.org@mitzpettel.com> wrote:
On Aug 7, 2007, at 2:15 PM, antti@webkit.org wrote: - added performance test. With debug build on MBP this takes about 1.5s to run.
* fast/block/basic/stress-shallow-nested-expected.txt: Added. * fast/block/basic/stress-shallow-nested.html: Added.
(emphases mine). Nothing about that makes sense to me.
Are you objecting to testing performance regressions as part of the test suite in general or particular method here? I'm open to suggestions.
It does not take long to run (300ms on release, 1.5s on debug MBP) so I thought it would be appropriate for automatic test. There are similar cases in the suite already. It needs to have non-zero execution time so that O(n^2) nature of the bug shows up in testable way.
The main question for me would be whether this will fail if the performance bug was somehow reintroduced. Would it take so long that the watchdog timer would think the test had hanged and kick in? - Maciej
On Aug 7, 2007, at 8:09 PM, Maciej Stachowiak wrote:
The main question for me would be whether this will fail if the performance bug was somehow reintroduced. Would it take so long that the watchdog timer would think the test had hanged and kick in?
The “external” 10-second watchdog timer in DumpRenderTree only works if the test calls waitForDone(), which this test does not. The test uses an internal 8-second timer to decide if it has failed.
If I understood the bug fix and the test correctly, then a test like this might work: do 5000 iterations (instead of 30000) and look at the ratio between the time the first 500 and last 500 iterations take. On Aug 7, 2007, at 5:22 PM, Antti Koivisto wrote:
On 8/7/07, Mitz Pettel <opendarwin.org@mitzpettel.com> wrote:
On Aug 7, 2007, at 2:15 PM, antti@webkit.org wrote: - added performance test. With debug build on MBP this takes about 1.5s to run.
* fast/block/basic/stress-shallow-nested-expected.txt: Added. * fast/block/basic/stress-shallow-nested.html: Added.
(emphases mine). Nothing about that makes sense to me.
Are you objecting to testing performance regressions as part of the test suite in general or particular method here? I'm open to suggestions.
It does not take long to run (300ms on release, 1.5s on debug MBP) so I thought it would be appropriate for automatic test. There are similar cases in the suite already. It needs to have non-zero execution time so that O(n^2) nature of the bug shows up in testable way.
antti
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
I am just wondering whether it should be the other way round, in that how many iterations can be done in time x. The logic here is that as processors gets faster a given number of iterations will eventually reach a value near zero. On the other hand for a given time period there is much more room for expansion on the number of iterations possible. Andre On 7-Aug-07, at 19:04 , Mitz Pettel wrote:
If I understood the bug fix and the test correctly, then a test like this might work: do 5000 iterations (instead of 30000) and look at the ratio between the time the first 500 and last 500 iterations take.
On Aug 7, 2007, at 5:22 PM, Antti Koivisto wrote:
On 8/7/07, Mitz Pettel <opendarwin.org@mitzpettel.com> wrote:
On Aug 7, 2007, at 2:15 PM, antti@webkit.org wrote: - added performance test. With debug build on MBP this takes about 1.5s to run.
* fast/block/basic/stress-shallow-nested-expected.txt: Added. * fast/block/basic/stress-shallow-nested.html: Added.
(emphases mine). Nothing about that makes sense to me.
Are you objecting to testing performance regressions as part of the test suite in general or particular method here? I'm open to suggestions.
It does not take long to run (300ms on release, 1.5s on debug MBP) so I thought it would be appropriate for automatic test. There are similar cases in the suite already. It needs to have non-zero execution time so that O(n^2) nature of the bug shows up in testable way.
antti
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
That should be the total amount of time will reach a value near zero, not the total number of iterations - oops. On 7-Aug-07, at 23:29 , Andre-John Mas wrote:
I am just wondering whether it should be the other way round, in that how many iterations can be done in time x. The logic here is that as processors gets faster a given number of iterations will eventually reach a value near zero. On the other hand for a given time period there is much more room for expansion on the number of iterations possible.
Andre
On 7-Aug-07, at 19:04 , Mitz Pettel wrote:
If I understood the bug fix and the test correctly, then a test like this might work: do 5000 iterations (instead of 30000) and look at the ratio between the time the first 500 and last 500 iterations take.
On Aug 7, 2007, at 5:22 PM, Antti Koivisto wrote:
On 8/7/07, Mitz Pettel <opendarwin.org@mitzpettel.com> wrote:
On Aug 7, 2007, at 2:15 PM, antti@webkit.org wrote: - added performance test. With debug build on MBP this takes about 1.5s to run.
* fast/block/basic/stress-shallow-nested-expected.txt: Added. * fast/block/basic/stress-shallow-nested.html: Added.
(emphases mine). Nothing about that makes sense to me.
Are you objecting to testing performance regressions as part of the test suite in general or particular method here? I'm open to suggestions.
It does not take long to run (300ms on release, 1.5s on debug MBP) so I thought it would be appropriate for automatic test. There are similar cases in the suite already. It needs to have non-zero execution time so that O(n^2) nature of the bug shows up in testable way.
antti
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
These are good ideas. O(n^2) fixes really should have regression tests attached, like any other bug fixes. antti On 8/8/07, Andre-John Mas <ajmas@sympatico.ca> wrote:
I am just wondering whether it should be the other way round, in that how many iterations can be done in time x. The logic here is that as processors gets faster a given number of iterations will eventually reach a value near zero. On the other hand for a given time period there is much more room for expansion on the number of iterations possible.
Andre
On 7-Aug-07, at 19:04 , Mitz Pettel wrote:
If I understood the bug fix and the test correctly, then a test like this might work: do 5000 iterations (instead of 30000) and look at the ratio between the time the first 500 and last 500 iterations take.
On Aug 7, 2007, at 5:22 PM, Antti Koivisto wrote:
On 8/7/07, Mitz Pettel <opendarwin.org@mitzpettel.com> wrote:
On Aug 7, 2007, at 2:15 PM, antti@webkit.org wrote: - added performance test. With debug build on MBP this takes about 1.5s to run.
* fast/block/basic/stress-shallow-nested-expected.txt: Added. * fast/block/basic/stress-shallow-nested.html: Added.
(emphases mine). Nothing about that makes sense to me.
Are you objecting to testing performance regressions as part of the test suite in general or particular method here? I'm open to suggestions.
It does not take long to run (300ms on release, 1.5s on debug MBP) so I thought it would be appropriate for automatic test. There are similar cases in the suite already. It needs to have non-zero execution time so that O(n^2) nature of the bug shows up in testable way.
antti
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
participants (5)
-
Andre-John Mas
-
Antti Koivisto
-
Geoffrey Garen
-
Maciej Stachowiak
-
Mitz Pettel