[webkit-dev] CSS 2.1 Test Suite

[Alan Stearns]

The best way to judge whether a reference result is correct is to submit
the result to the W3C CSS 2.1 test suite and have it reviewed. The only
way this test suite will get more reference results is if people like us
volunteer to submit references. If it's useful to us to have these
'homebrew reference results' then it will be useful to everyone else who
uses the suite.

The previous thread mentioned checking Mozilla to see if their test suite
had references for particular tests. If that's the case, then we should
either encourage them to submit the references to the W3C, or just do that
ourselves on their behalf.

Alan

From:  Ryosuke Niwa <rniwa at webkit.org>

As we have previous discussed following
https://lists.webkit.org/pipermail/webkit-dev/2012-March/019782.html, it's
hard to judge whether a given reference result is enough to cover
everything the test intends to test. e.g. you can have a bug such that
both the test and the reference file ends up having the same rendering
result.
- Ryosuke

On Tue, Apr 10, 2012 at 2:26 PM, Robert Hogan <lists at roberthogan.net>
wrote:


Hi there,

We currently add tests from the CSS 2.1 suite as we fix them. They get
added
to the css2.1/20110323 folder in LayoutTests. Most of them don't have
native reference tests yet in the suite so we (mostly I) have been adding
homebrew reference results to the folder to avoid generating pixel results
on all platforms. (see http://webkitmemes.tumblr.com/post/20788159625 !)

These reference-results are easily removed once superseded but it might be
cleaner just to move them, and the associated css tests, to a folder of
their own in fast/css. That will allow css2.1/20110323 to be a clean import
that the 8500 or so passing tests can be added to in 20 or 30 batches.[1]
It will also make NRWT's reftests harness work with the suite.

Does anyone object to that approach? The only thing going against it seems
to be the principle that imported tests should be stored separately and
together but this is more a case of using them to fix bugs and prevent
future regressions while allowing a clean import of the CSS 2.1 test suite
to take place in parallel.

The problem this does not solve is how we avoid creating pixel results for
tests that already pass but which do not have reftests of their own. Again
I would be in favour of putting these in fast/css and keeping them there
until reftests are added to the suite. This would allow us to prevent them
regressing and come up with a reftest for them that can be submitted to the
CSS test suite guys.

The end result would be that we only directly import to the css2.1 folder
those tests in the CSS test suite that have ref tests native to the suite.

Thanks,
Robert


[1] I keep a local and relatively up to date copy of the passing and
failing tests in separate folders in my checkout.  Yes I know I should
create bugs for them and get started on landing the passes.

_______________________________________________
webkit-dev mailing list
webkit-dev at lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev