[webkit-dev] Supporting <link rel=> for finding ref tests

[Robert Ma]

WPT has recently passed an RFC
<https://github.com/web-platform-tests/rfcs/blob/master/rfcs/reftest_simplification.md>
trying to simplify the reftest graph (although it has not been implemented
yet), which eliminates a lot of the complexities and concerns.

On Fri, Nov 8, 2019 at 5:07 PM Ryosuke Niwa <rniwa at webkit.org> wrote:

> On Fri, Nov 8, 2019 at 2:01 PM Simon Fraser <simon.fraser at apple.com>
> wrote:
> >
> > I'd like to land a patch to support finding test references via <link
> rel="match/mismatch">:
> > https://bugs.webkit.org/show_bug.cgi?id=203784
> >
> > There has been some discussion about this in the past:
> > https://lists.webkit.org/pipermail/webkit-dev/2011-November/018470.html
> >
> > But I think the benefits outweigh the drawbacks. As that mail states:
> >
> > *Link element approach*
> > Pros:
> >
> >    - Can reuse same ref. file for multiple tests
> >
> > Still true.
> >
> >    - Can have multiple ref. files for single test
> >
> > True but no something that we support, and I haven't see any WPT use
> this (our importer throws an error if it sees this)
> >
> >    - Information is self-contained in the test file
> >
> > Still true
> >
> >    - We may get away with test suite build step
> >
> > It certainly simplifies WPT test import.
> >
> > Currently importing some CSS suites (e.g. css-backgrounds) results in
> broken -expected.html files because copying them breaks references to sub
> resources.
> >
> > (It turns out that we can't convert W3C ref tests to use WebKit
> conventions
> > due to the first two points.)
> >
> > We're doing this much more now, and the "multiple references" point is
> moot, so I think we can import WPT tests mostly as-is.
> >
> > Cons:
> >
> >    - Requires us modifying each port's DRT to support this format
> >
> > No, it just requires webkitpy hacking which I've done in the patch.
>
> I'm not certain writing a bunch of regular expressions in webkitpy is
> a reliable mechanism to find expected results. Another issue I found
> back then was that it significantly slowed run-webkit-tests' startup
> time because WPT has a workflow to find all tests & their expected
> results upfront before any tests could run.
> >
> >    - Adding link elements itself may affect tests (all W3C tests are
> >    required to have link elements at the moment)
> >
> > I haven't seen this be an issue.
>
> Another issue is that if you were to modify a test which happens to be
> also used as a reference or a mismatch result (worse) for some other
> test, then you may not notice that without inspecting every other test
> in existence.
>
> >    - Hard to understand relationship between files. e.g. if we want to
> >    figure out which tests use ref.html, we must look at all test files
> >
> > This is true, but I don't really see it being a problem in practice.
>
> This definitely is an issue. It's possible WPT has improved things but
> we've definitely had an experience where tests were used as reference
> for other tests, etc... and having to think about this issue every
> time I touch test drove me nuts.
>
> > What I have seen is us importing CSS 2.1 tests that have foo.html and
> foo-ref.html, and treating foo-ref.html as a test so generating
> foo-expected.txt and foo-ref-expected.txt. That seems worse.
>
> Seems like we can treat "-ref" as a special suffix like we already do
> with support directory and resources directory.
>
> > So now that WPT is heavily invested in <link rel=> I think we should
> follow suite. It will simplify WPT import, and reduced the number of cloned
> -expected.html files significantly.
>
> I really don't want to deal with tests being used as references for
> other tests. I'm okay with this approach if we forbid that.
>
> - R. Niwa
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20191108/db0d0272/attachment.htm>