[webkit-dev] The --test-list option behavior in NRWT

[Žan Doberšek]

Hi WebKitties,

I'd like to see in what scale the --test-list option in NRWT is used and
whether anyone would object a change in how its usage behaves.

Currently the test gathering takes into account any paths that were passed
in through the command line and any paths listed in the file to which a
possible --test-file option points[1]. These paths are then expanded if
necessary (due to platform-specific tests and virtual test suites) and all
the tests to which these paths apply are gathered[2]. All the gathered
tests are then either sorted into a natural order or randomized (if the
--randomize-order option is used)[3].

I'd like to propose a change in the behavior when using the --test-list
option. When used, the tests would be gathered only from the paths listed
in the file to which the option points. Also, the gathered tests would be
sorted to follow the order of the paths in the test list file. For
instance, consider the following two entries being placed in the test list
file:

c/d/e.html
a/b/

The current behavior would gather all the tests to which these two paths
apply and sort them in this order (on the premise that no other paths are
passed through the command line arguments):

a/b/1.html
a/b/2.html
c/d/e.html

The new behavior would place the c/d/e.html test at the top of the list but
sort the other two tests in order, like this:

c/d/e.html
a/b/1.html
a/b/2.html

The idea behind this is to be able to specify the exact order in which the
tests should be run. I'm currently playing around with a tool that randomly
chooses a certain number of tests, runs them and bisects down the first
regression that occurs (if any) to the smallest possible ordered list of
tests that reproduces that regression. The proposed change makes this a
simple task from the test listing perspective and I'd argue that exact test
ordering is the main point of an option such as --test-list.

Currently I'm using an untested workaround of adding a new option that
specifies the tests should be reordered back to the original order that the
test list file provided, but I'd like to move on with implementing the
change in behavior if everyone feels it's OK and won't tamper with his/her
workflow.

-Z


[1]
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46
[2]
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576
[3]
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121106/0dea2321/attachment.html>