[webkit-dev] Making changes to our copy of the W3C CSS1 test suite

[Darin Adler]

On Jun 3, 2010, at 2:30 PM, Ojan Vafai wrote:

> On Wed, Jun 2, 2010 at 6:20 PM, Darin Adler <darin at apple.com> wrote:
>    1) There’s one directory with a pristine copy of the W3C test suite, with no WebKit changes.
>    2) If there are some tests that need to be fixed, fixed copies of those individual tests would go into another directory.
> 
> So we would know that if css3-fixed-up/foo.html exists to skip css3-pristine/foo.html? 

Not necessarily. We can skip the pristine one if we have to, but generally speaking I think we should run the pristine one and the fixed up one too.

>    3) The broken tests can be run as-is, and we can land expected results to reflect what the broken tests do.
> 
> Perhaps I would think differently about some aspect of this if we had introduced the “test expectations” concept for platforms other than Chromium. 
> 
> It's a tradeoff. On the one hand, the test expectations approach lets you have a list of failing tests that you drive to 0. On the other hand, checking in failing expectations lets you know if you ever unintentionally change the the type of failure (e.g. it was failing for one reason and is now failing for a different reason due to your patch). If we intend to have a concerted, short-term effort to drive the failing tests to 0, then an expectations approach seems better.

Maybe we should come up with a system that does both.
 
> There should be some set of tests that are faster to run that omits the slower thorough tests. This was the original goal of “fast” but we have put tests outside “fast” more or less at random. Why are “editing” tests outside “fast”? Just the whim of the person who added them. Same comment on directories like “accessibility”.
> 
> I don't like the concept of putting fast tests in a separate top-level directory.

Neither do I. I said this was the original concept, but I meant that we should accomplish this a new way now.

> If we want a way to run just the fast tests, we should just come up with a way of annotating tests as slow. Another option is that we could have a "slow" subdirectory.

Agreed. This was what I meant to say.

    -- Darin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100603/68edd4d2/attachment.html>