[webkit-dev] new: experimental support for 'virtual test suites' in new-run-webkit-tests

[Dirk Pranke]

Hi all,

I have recently landed support for 'virtual test suites' in
new-run-webkit-tests (as the subject suggests ;) ). A virtual test
suite tells NRWT to run all the tests in directory 'foo' with an
additional set of command line arguments being passed to DRT, and to
look for baselines in directory 'bar' before looking in directory
'foo'.

As a concrete example, we're going to try and use this to remove the
'chromium-gpu' ports and just view gpu-specific tests as a virtual
suite on top of the existing tests. So, we have a mapping that says:

run all the tests in fast/canvas with --accelerated-2d-canvas being
passed to DRT (as well as the normal flags), and look for results in
LayoutTests/platform/chromium/virtual/gpu/fast/canvas before looking
in the normal places.

The mapping of virtual test suites is currently hardcoded in the NRWT
python code (in the webkitpy/layout_tests/port/*.py files, to be
precise), but it would be easy to make the mappings live as files in
the tree as well.

All of the normal tools (webkit-patch garden-o-matic, webkit-patch
rebaseline, the flakiness dashboard) should work as if the virtual
tests are regular tests. I doubt they do at the moment - I'm still
testing and bug hunting.

I believe this feature will be generally useful down the road. For
example, one can imagine running a set of tests with a mocked-out set
of controls (e.g., fake scrollbars) so that we can have
platform-independent baselines, and then re-running a subset of the
tests with native controls to ensure that they also work.

I realize that this is a bit different than the normal practice for
LayoutTests, where you often run tests differently by setting up a
test differently and then including a SCRIPT to do the real test; the
downside of the latter approach is that there's more boilerplate and
you have to remember to do this every time a new test is added. That
said, there are definitely still times (most times?) when that is the
better approach.

At any rate, like I said, it's an experiment and feedback is welcome.
If you have ideas for how something like this might be useful but need
additional features, etc., do let me know.

Thanks,

-- Dirk