[webkit-dev] Making changes to our copy of the W3C CSS1 test suite

[Ojan Vafai]

On Wed, Jun 2, 2010 at 6:20 PM, Darin Adler <darin at apple.com> wrote:
>
>    1) There’s one directory with a pristine copy of the W3C test suite,
> with no WebKit changes.
>    2) If there are some tests that need to be fixed, fixed copies of those
> individual tests would go into another directory.
>

So we would know that if css3-fixed-up/foo.html exists to skip
css3-pristine/foo.html?

   3) The broken tests can be run as-is, and we can land expected results to
> reflect what the broken tests do.
>
> Perhaps I would think differently about some aspect of this if we had
> introduced the “test expectations” concept for platforms other than
> Chromium.


It's a tradeoff. On the one hand, the test expectations approach lets you
have a list of failing tests that you drive to 0. On the other hand,
checking in failing expectations lets you know if you ever unintentionally
change the the type of failure (e.g. it was failing for one reason and is
now failing for a different reason due to your patch). If we intend to have
a concerted, short-term effort to drive the failing tests to 0, then an
expectations approach seems better.


> There should be some set of tests that are faster to run that omits the
> slower thorough tests. This was the original goal of “fast” but we have put
> tests outside “fast” more or less at random. Why are “editing” tests outside
> “fast”? Just the whim of the person who added them. Same comment on
> directories like “accessibility”.
>

I don't like the concept of putting fast tests in a separate top-level
directory. I want the tests to be grouped by what they're testing. If we
want a way to run just the fast tests, we should just come up with a way of
annotating tests as slow. Another option is that we could have a "slow"
subdirectory.

A real-world example, we have some very slow editing tests, but most of them
are fast. We could move the slow ones into editing/slow or we could annotate
them as slow in an expectations file. Then we add a --fast flag to
run-webkit-tests that skips over slow tests.

This does mean that we'll have slow subdirectories in many places (or a
single file listing the slow tests), but at least tests will all be grouped
by what they're testing.

The test suites that are imported from elsewhere should go into directories
> with clear names such as:
>
>    ImportedTestSuites/W3C-CSS1
>
> Rather than names like:
>
>    css1
>

This sounds great. It will make understanding the tests much easier.

Ojan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100603/6bdf8075/attachment.html>