If you look at DumpRenderTree on Win32, we just made the results append -win to the file. Unfortunately we can't really generate good results for Win32 yet, since someone needs to rewrite DumpRenderTree to be more of a windowed app (like Spinneret). dave On Aug 31, 2006, at 1:08 PM, Krzysztof Kowalczyk wrote:

On 8/31/06, Jean-Charles VERDIE <jcverdie@origyn.fr> wrote:

...

We imagine a solution which is to drop our goal to fix the problem, and instead generate "linux"-based -expected.txt files and to modify dump render tree to either chose osx or linux -based expected files, depending on the platform we are testing on.

This solution has a major drawback, which is that it will force the community to maintain two version of expected files for every single test.

Before starting to work in this direction, I'd appreciate some feedback on the feeling about this solution, and may be, fortunately, others ideas on how to remove this roadblock.

It would probably be an ugly hack, but looking at the diff it looks like all differences are for sizes and fall within 1-2 pixels. How about a fuzz factor (either as percent of the total original size or in pixels) and accepting failures for sizes orginated from text rendering if they fall within fuzz factor? Fuzz factor would be best determined empirically (i.e. by comparing current linux vs. mac differences and choosing a factor that makes them pass).

The thinking is that a major breakage would still be detected as falling outside of fuzz factor.

-- kjk _______________________________________________ webkit-dev mailing list webkit-dev@opendarwin.org http://www.opendarwin.org/mailman/listinfo/webkit-dev

Hi Darin & all, On 9/1/06, Darin Adler <darin@apple.com> wrote:

To improve the situation, I'd like to see some of the following happen:

1) Reorganize the tests so that tests that should work across all platforms are separate from ones that might need different results per platform. (Perhaps we could rationalize the organization and naming of tests in some other ways as well.)

2) Change more tests to use dumpAsText as a way of making them platform-independent.

3) Come up with some techniques to make tests independent of font widths to make them platform-independent; for example, perhaps we can create a font with widths that are consistent on all platforms and use it for most tests.

It would surely be the best plan, but I'm not sure that it will be possible (in a measurable human-based time). May be the fastest way of having the tests more cross-platform is to split them into three categories: 1- those which already are cross-platform. 2- those which are not cross-platform, but which do not need a font. E.g. comparing 2 cells of a table. We could replace the texts by pictures with a specific size. The test would not be self-documented any more, but we can solve this by a 2-windows solution, one with the explanation, and one with the actual test. 3.1- those which are not cross-platform, but which need a font. Some can be addressed by replacing DumpRenderTree by DumpAsText. 3.2 - ...But other tests in this third will hardly need a font, e.g. css manipulation or font replacement. For this latest part, I don't imagine a solution right now. May be some can be addressed by your proposal #4 But one question is: how much of these tests remain? if we address 1- to 3.1 ? IMHO here is the list : - DOM: already x-platform - plugins / svg / http / editing: we've not started looking at them - Tables, fast, traversal: all should be replaceable by images - CSS 1, 2.1: here is the list of our problems :) There are 362 tests in these folders, but may be only 30% of them are belonging to the problematic category so something like a big hundred of tests would need to be re-thinked... 4) Consider alternate dumping formats that would include what's

relevant to check if a test succeeded that don't dump the entire layout and position of each element -- something in between "dump as text" and "dump render tree".

Yes! Also, I think the DumpRenderTree tool is going to need a rename

eventually. That was a good name for the original tool back when I first wrote it and it was only about render trees, but now it's more like "test engine" or something along those lines. Similarly, I think the LayoutTests directory might need a rename for similar reasons.

Very good idea, for sure. Best regards, -- Jean-Charles Verdié Origyn Web Browser for Embedded Systems Team CTO

Darin, We'll schedule a task for identifying in CSS 1/2.1 folders which tests should be replaced by picture-based ones, and which could not. For the latter, we'll try to think about workarounds to propose. Have you an accurate placeholder to suggest for these thoughts? I was not sure whether bugzilla or wiki would be better.... For others tests folders, I think that if the proposed plan fits WebKit community's interest, we should file bugs about replacing Tables, Fast and Traversal by image-based tests, and also bugs about exploring the parts we haven't done yet (plugin, svg, http, editing)... Let me know what you think, Regards Jean-Charles On 9/1/06, Darin Adler <darin@apple.com> wrote:

On Sep 1, 2006, at 1:38 AM, Jean-Charles VERDIE wrote:

...

It would surely be the best plan, but I'm not sure that it will be possible (in a measurable human-based time). May be the fastest way of having the tests more cross-platform is to split them into three categories: 1- those which already are cross-platform. 2- those which are not cross-platform, but which do not need a font. E.g. comparing 2 cells of a table. We could replace the texts by pictures with a specific size. The test would not be self- documented any more, but we can solve this by a 2-windows solution, one with the explanation, and one with the actual test. 3.1- those which are not cross-platform, but which need a font. Some can be addressed by replacing DumpRenderTree by DumpAsText. 3.2 - ...But other tests in this third will hardly need a font, e.g. css manipulation or font replacement.

For this latest part, I don't imagine a solution right now. May be some can be addressed by your proposal #4 But one question is: how much of these tests remain? if we address 1- to 3.1 ? IMHO here is the list : - DOM: already x-platform - plugins / svg / http / editing: we've not started looking at them - Tables, fast, traversal: all should be replaceable by images - CSS 1, 2.1: here is the list of our problems :) There are 362 tests in these folders, but may be only 30% of them are belonging to the problematic category so something like a big hundred of tests would need to be re-thinked...

Sounds good. If you would like to start doing some work in this area, please let us know about your specific ideas.

Maybe for your (2) above we could make tests where certain parts of the page wouldn't dump. That way the test could be self documenting but the part that dumps wouldn't be affected by the text.

-- Darin

-- Jean-Charles Verdié Origyn Web Browser for Embedded Systems Team CTO

Hi Darin & Dave & Krzystof & Jean-Claude, During the Qt/KDE platform porting, I've basically run into the same problem, though I also dislike the idea of a fuzz factor.

1) Reorganize the tests so that tests that should work across all platforms are separate from ones that might need different results per platform. (Perhaps we could rationalize the organization and naming of tests in some other ways as well.)

I still think this is the best solution, having -expected-qt.txt, -expected-gdk.txt, -expected-osx.txt, -expected-win.txt, to be able to find out regressions pixel-wise, so that 1px does make a difference.

Also, I think the DumpRenderTree tool is going to need a rename eventually. That was a good name for the original tool back when I first wrote it and it was only about render trees, but now it's more like "test engine" or something along those lines. Similarly, I think the LayoutTests directory might need a rename for similar reasons.

Something like RegressionTester may be better, indeed. My suggestions is to create subdirectories for the specific platforms, where the -expected-<PLATFORM>.txt files & image diffs etc. will go. Maybe it would be even wiser to adjust the "Get-WebKit-From-SVN" scripts to just svn co the specific subdirectory needed. For example it doesn't make sense for you guys to get our -expected-gdk/qt.txt files. And we also don't need for instance the -win.txt files. Please let me know what you think. Niko