[webkit-dev] CSS 2.1 Test Suite

[Robert Hogan]

On Wednesday 11 April 2012 20:11:23 Alan Stearns wrote:
> The best way to judge whether a reference result is correct is to submit
> the result to the W3C CSS 2.1 test suite and have it reviewed. The only
> way this test suite will get more reference results is if people like us
> volunteer to submit references. If it's useful to us to have these
> 'homebrew reference results' then it will be useful to everyone else who
> uses the suite.

Agreed. This will help us land the tests that already pass and won't slow 
down the effort to fix the css tests that we don't. We can agree to only 
import css tests with reftests and get NRWT working on them. I hope to do 
this in:

https://bugs.webkit.org/show_bug.cgi?id=83048

However I do think we need a decision on how we:

1. Land fixes for currently broken tests that don't have a reftest.
2. Clean up the existing css2.1/20110323 folder.

If it's just a case of living with pixel results for now I'm happy with 
that. But I think allowing them to live outside css2.1/20110323 would 
encourage me and others to write reftests while we're fixing the tests' 
results on WebKit.

From listening to Maciej, Ojan and Ryosuke it is not an option to keep 
homebrew reftests in css2.1/20110323 for two good reasons: it breaks 
important assumptions in the way the reftest harness works, and it is 
better to keep imported test suites clean and unmodified.

So it's either 1 or 2 above I think. I would prefer 2 as it won't bloat git 
checkouts so much and will make fixes easier to land.