[webkit-dev] Where do we put WPT tests to be exported

[youenn fablet]

Le lun. 15 mai 2017 à 23:15, Maciej Stachowiak <mjs at apple.com> a écrit :

>
> On May 15, 2017, at 9:08 PM, youenn fablet <youennf at gmail.com> wrote:
>
> I see two main cases:
> - Writer of the patch is making sure to upstream WPT test changes at
> WebKit landing time. It is ok to make the changes directly in
> LayoutTests/imported/w3c/web-platform-tests/
> - Writer plans to upstream WPT test changes at some point but wants more
> time. It is better to develop the tests in LayoutTests/http/wpt and then
> migrate them later on.
>
>
> My proposal was different from either of these, it was to have a directory
> specifically for tests meant to be upstreamed (LayoutTests/http/wpt should
> contain only tests not meant
>

The possibility to have LayoutTests/http/wpt/to-be-imported was mentionned
previously.
I think it makes sense to run to-be-imported tests behind WPT as this is
the way they will end up being run when upstreamed.


>
> I think adding new tests directly to
> LayoutTests/imported/w3c/web-platform-tests/ is needlessly messy. Most
> stuff in the imported/ directory is an exact copy of an upstream test
> suite, so if you run only a specific directory of tests, you know you are
> running an official conformance suite. With this proposal, it might also
> contain random tests that will hopefully be upstreamed but maybe not, or
> might be changed before the PR lands upstream, or might get renamed, or
> whatever. There's no guarantee that updating from the official version will
> ever fully resolve the delta.
>

LayoutTests/imported/w3c/web-platform-tests is mostly about regression
testing.
For conformance testing, it is probably more accurate to either use
w3c-test.org (probably not reliable enough though) or make a clone of W3C
WPT, do the set up as specified in their README (etc/hosts...) and use
wptrunner.


>
> I think it would be more elegant to have a parallel directory
> (LayoutTests/for-export/w3c/web-platform-tests). Then when something is
> actually upstreamed and then pulled back down, we could delete the staged
> version. Directly modifying our local copy seems like it could easily lead
> to long-term divergences slipping through the cracks.
>

A parallel directory is fine when starting a test suite.
Often, changes are limited to modifying a test, which might be best done
in-place.
Often, adding a test is easier by adding it in an already existing file.

Tooling should protect us from diverging.
Authoring in LayoutTests/for-export/w3c/web-platform-tests might not allow
you to use WPT tools and common resources.
At upstream time, reviewers will probably tell to use that and that
existing resource.

There is agreement that a imported/w3c/web-platform-test layout test change
that lands in WebKit can be merged upstreamed.
I am not sure what would be the process with
LayoutTests/for-export/w3c/web-platform-tests tests.


>
> Of course, people could always go run the official copy from w3c-test.org .
> But we usually leave imported conformance test suites unmodified except the
> minimum necessary to make them run.
>
> I would start with an experimental phase with some of us making direct
> changes in LayoutTests/imported/w3c/web-platform-tests/.
> When we are happy with the tools and think the risk for issues is low
> enough (or when the bots can handle most of it for us), hacking
> LayoutTests/imported/w3c/web-platform-tests/ could be the default.
>
>
> I think the problem with this plan is not tools risk, it's the fact that a
> directory of imported tests can no longer be trusted to actually just be
> imported tests. So a smaller number of people doing it to start would not
> address my concern. It might be that other people don't care about this. My
> opinion counts no more than anyone else's, and I'd be interested in hearing
> from others.
>

I am not sure to understand precisely what the consequences of "imported
tests can no longer be trusted to actually just be imported tests" are.
Ideally, a WPT test change landing in WebKit should be merged at the same
time on W3C WPT.

There was a suggestion that LayoutTests/imported/w3c/web-platform-tests be
moved to a shorter path like LayoutTests/web-platform-tests. That would
also make it clear that this folder is not only about one-way-sync.


>
> Regards,
> Maciej
>
>
> Le lun. 15 mai 2017 à 21:02, Ryosuke Niwa <rniwa at webkit.org> a écrit :
>
>> Hi all,
>>
>> Youenn is working on a patch to automatically create a GitHub PR to
>> export tests from a WebKit patch which includes changes to
>> web-platform-tests [1].
>>
>> That raises a question as to where we should put new tests or modified
>> tests intended to be exported to web-platform-tests from WebKit.
>>
>> I think the most obvious option is to use
>> LayoutTests/imported/w3c/web-platform-tests/.  However, in the other
>> thread about adopting testharness.js (titled Another WPT bite), Maciej
>> briefly expressed the preference for creating a new directory:
>> https://lists.webkit.org/pipermail/webkit-dev/2017-May/029022.html
>>
>> Do other people have strong opinions about this?
>>
>> - R. Niwa
>>
>> [1] https://bugs.webkit.org/show_bug.cgi?id=169462
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170516/8f97f074/attachment.html>