[webkit-dev] Process for importing third party tests

[Ryosuke Niwa]

On Tue, May 8, 2012 at 1:04 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>
>  On Tue, May 8, 2012 at 12:42 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> > On Wed, Apr 25, 2012 at 2:18 PM, Dirk Pranke <dpranke at chromium.org>
> wrote:
> >> I am also more than a little leery of mixing -expected.{txt,png}
> >> results with -expected.html results; I feel like it would be very
> >> confusing, and it would lose many of the advantages of reftests, since
> >> we'd presumably have to update the reference every time as often as we
> >> update pixel tests. [We could create a "fake" reftest that just
> >> contained the 800x600 pixel dump, but I'm not sure if that's better or
> >> not].
> >
> > I don't think we will lose advantages. Without some sort of the current
> > result, we will not be able to catch regressions.
> > According to Maciej, we've caught many regressions by using conformance
> > tests this way.
>
> I have no doubt that you can catch regressions by using conformance
> tests this way. My concern -- which is expressed throughout and which
> you seemed to either miss or downplay -- is that adding more tests
> creates more work, especially for pixel tests (and it slows down build
> and test cycles, obviously). I don't think we should just add tests
> because they exist somewhere on the web; they may provide no
> additional coverage beyond the tests we already have.
>

Yes, that's why we want reviews. And that's why we should only import tests
from W3C instead of other browser vendors. We expect W3C to have some
guideline on avoiding test duplicates.

I feel like we need a stronger mechanism to either check that new test
> suites do cover more functionality or we move to obsolete tests we
> already have.
>

We can't really verify that two tests test same functionality, etc...
automatically. Also, the most general form of this question is undecidable.

However, there are a couple of ways to mitigate this issue:

   1. Upstream as many layout tests as possible to W3C
   2. Delete duplicate layout tests as we import more tests from W3C

To be clear, I am all for importing test suites when we believe they
> are comprehensive or do cover things we don't cover well now.


This should probably be judged by individual reviewers.

But, for example, rather than having four different test suites for
> flexbox, I
> would rather see us have one good one.
>

Sure but I'd that's a really hard problem to solve.

 >> Also, I was under the impression that (a) the W3C is mostly focused on
> >> ref tests going forward and (b) we had agreed in that discussion that
> >> we wouldn't import non-ref tests? Did something change in a discussion
> >> after that session?
> >
> > No. We had agreed to import all tests regardless of whether they're
> reftests
> > or not because non-reftests can still catch future regressions or
> > progressions. And the number of PNG files added to the repository wasn't
> > considered as a valid counter-argument due to this utility.
>
> Well, I certainly didn't agree to it :) My concern is not the # of
> PNGs so much as the cost of maintenance.
>

Sure, that's a valid concern but the overwhelming majority of the people in
the room (e.g. Darin, Maciej, etc...) seemed to agree that this is a good
idea.

> I don't understand your proposal about adding platform/webkit. Why do we
> > want that? As far as I know, there are no files in W3C test directories
> that
> > end with -expected.txt or -expected.png.
>
> The idea would be that no webkit-specific files would live in the test
> directory, only files received from upstream. My thinking was that it
> would make importing new versions easier and it would be easier to
> understand what was ours vs. what was theirs. I don't feel that
> strongly about this, though, it was just an idea.


That'll be nice indeed. But if we're going this route, we should probably
move all existing -expected.* to this directory as well. So this is
probably a tangential issue. Similar to:

>> By "ultimately move all existing tests", I assume you're including
> >> tests that are currently in LayoutTests that have not come from (or
> >> been submitted to) the W3C, e.g., the tests in fast/ ?
> >
> > Yes.
>
> I think reorganizing our existing test tree is an entirely different
> discussion. I'm all for it, I just don't want to confuse it with the
> discussion about importing test suites.
>

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120508/b50c7864/attachment-0001.html>