[webkit-dev] Proposal for Device-Specific Layout Tests

[Ryosuke Niwa]

Are those directories along side tests? If so, that seems very different
from the approach we take with platforms. Why the discrepancy?

Can't we just treat them as specific type of iOS platforms?

- R. Niwa


On Mon, Dec 3, 2018 at 4:05 PM Jonathan Bedard <jbedard at apple.com> wrote:

> Hello everyone,
>
> I have a proposal to make writing layout tests for specific types of
> devices more straight-forward.
>
> Currently, we name directories with the beginning of a device name, such
> as ‘iphone7’ or ‘ipad’ and hope that those directory names match the name
> of a device type in the associated port’s CUSTOM_DEVICE_CLASS. This causes
> some problems, however, in a few circumstances. The first problem arises
> when two ports have associated devices (such as watchOS and iOS), in this
> case, a directory which maps to a device type on one port might not on
> another. The second problem is that our current code for managing devices
> assumes that devices can be booted on-demand, which is not the case in
> on-device testing, where we need to run different sets of tests depending
> on the type of device attached. The last (and most important) problem is
> that we cannot run the same test twice on two different devices in a single
> test run (ie, run test.html once on an iPhone and once on an iPad).
>
> To resolve this, I propose 3 changes to how we organize layout tests.
>
> 1) Allow *-expected.txt files to be device-specific <
> https://bugs.webkit.org/show_bug.cgi?id=192162>
>
> The idea here would be to optionally allow *-expected.txt files to be a
> directory looking something like this:
>
> dir/
>     test.html
>     test-expected/
>         iPhone-7.txt
>
> Under this scheme, the name of the .txt file would map directly to a
> device type. Multiple device-specific expectations would be permitted, but
> making sense of these results requires change #2.
>
> 2) Support multiple results for a specific test <
> https://bugs.webkit.org/show_bug.cgi?id=192163>
>
> Similar to #1, we need to allow test results to be attributed to a device
> type. This would be done with a similar scheme, where results for a test
> would optionally be saved as a directory looking like this:
>
> dir/
>     test/
>         iPhone-7-actual.txt
>         iPhone-7-diff.txt
>         iPhone-7-expected.txt
>         iPhone-7-diff.html
>         iPhone-8-actual.txt
>         iPhone-8-diff.txt
>         iPhone-8-expected.txt
>         iPhone-8-diff.html
>
> When reporting, results would be reported to as a new ‘queue’ of the form
> '<queue>-<device>’.
>
> 3) Provide device specific tags in test expectations <
> https://bugs.webkit.org/show_bug.cgi?id=192164>
>
> We already have Debug/Release as well as version tags in our test
> expectations. The addition here would be device-type tags, those tags would
> look like this:
>
> [ iPhone 7 ] dir/test.html [ Pass ]
> [ iPad ] dir/other-test.html [ Pass ]
>
> These tags would allow for 2 features. The first is similar to existing
> tags, allowing a certain test (or group of tests) to have different
> expectations or to be skipped entirely on certain device types. The second
> feature allows for tests and groups of tests to be run on multiple
> different devices in a single instantiation of run-webkit-tests. For
> example,
>
> [ iPhone 7, iPad ] dir/some-dir/ [ Pass ]
>
> would indicate that every test in dir/some-dir/ will be run on BOTH iPhone
> 7 and iPad.
>
> I would welcome anyone else’s thoughts on this.
>
> Jonathan
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20181203/05790e90/attachment.html>