[webkit-dev] Process for importing third party tests

Ryosuke Niwa rniwa at webkit.org
Tue May 8 13:30:17 PDT 2012

On Tue, May 8, 2012 at 1:04 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>  On Tue, May 8, 2012 at 12:42 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> > On Wed, Apr 25, 2012 at 2:18 PM, Dirk Pranke <dpranke at chromium.org>
> wrote:
> >> I am also more than a little leery of mixing -expected.{txt,png}
> >> results with -expected.html results; I feel like it would be very
> >> confusing, and it would lose many of the advantages of reftests, since
> >> we'd presumably have to update the reference every time as often as we
> >> update pixel tests. [We could create a "fake" reftest that just
> >> contained the 800x600 pixel dump, but I'm not sure if that's better or
> >> not].
> >
> > I don't think we will lose advantages. Without some sort of the current
> > result, we will not be able to catch regressions.
> > According to Maciej, we've caught many regressions by using conformance
> > tests this way.
> I have no doubt that you can catch regressions by using conformance
> tests this way. My concern -- which is expressed throughout and which
> you seemed to either miss or downplay -- is that adding more tests
> creates more work, especially for pixel tests (and it slows down build
> and test cycles, obviously). I don't think we should just add tests
> because they exist somewhere on the web; they may provide no
> additional coverage beyond the tests we already have.

Yes, that's why we want reviews. And that's why we should only import tests
from W3C instead of other browser vendors. We expect W3C to have some
guideline on avoiding test duplicates.

I feel like we need a stronger mechanism to either check that new test
> suites do cover more functionality or we move to obsolete tests we
> already have.

We can't really verify that two tests test same functionality, etc...
automatically. Also, the most general form of this question is undecidable.

However, there are a couple of ways to mitigate this issue:

   1. Upstream as many layout tests as possible to W3C
   2. Delete duplicate layout tests as we import more tests from W3C

To be clear, I am all for importing test suites when we believe they
> are comprehensive or do cover things we don't cover well now.

This should probably be judged by individual reviewers.

But, for example, rather than having four different test suites for
> flexbox, I
> would rather see us have one good one.

Sure but I'd that's a really hard problem to solve.

 >> Also, I was under the impression that (a) the W3C is mostly focused on
> >> ref tests going forward and (b) we had agreed in that discussion that
> >> we wouldn't import non-ref tests? Did something change in a discussion
> >> after that session?
> >
> > No. We had agreed to import all tests regardless of whether they're
> reftests
> > or not because non-reftests can still catch future regressions or
> > progressions. And the number of PNG files added to the repository wasn't
> > considered as a valid counter-argument due to this utility.
> Well, I certainly didn't agree to it :) My concern is not the # of
> PNGs so much as the cost of maintenance.

Sure, that's a valid concern but the overwhelming majority of the people in
the room (e.g. Darin, Maciej, etc...) seemed to agree that this is a good

> I don't understand your proposal about adding platform/webkit. Why do we
> > want that? As far as I know, there are no files in W3C test directories
> that
> > end with -expected.txt or -expected.png.
> The idea would be that no webkit-specific files would live in the test
> directory, only files received from upstream. My thinking was that it
> would make importing new versions easier and it would be easier to
> understand what was ours vs. what was theirs. I don't feel that
> strongly about this, though, it was just an idea.

That'll be nice indeed. But if we're going this route, we should probably
move all existing -expected.* to this directory as well. So this is
probably a tangential issue. Similar to:

>> By "ultimately move all existing tests", I assume you're including
> >> tests that are currently in LayoutTests that have not come from (or
> >> been submitted to) the W3C, e.g., the tests in fast/ ?
> >
> > Yes.
> I think reorganizing our existing test tree is an entirely different
> discussion. I'm all for it, I just don't want to confuse it with the
> discussion about importing test suites.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120508/b50c7864/attachment-0001.html>

More information about the webkit-dev mailing list