[webkit-dev] The --test-list option behavior in NRWT

[Dirk Pranke]

On Tue, Nov 6, 2012 at 11:58 PM, Žan Doberšek <zandobersek at gmail.com> wrote:
> Hi WebKitties,
>
> I'd like to see in what scale the --test-list option in NRWT is used and
> whether anyone would object a change in how its usage behaves.
>
> Currently the test gathering takes into account any paths that were passed
> in through the command line and any paths listed in the file to which a
> possible --test-file option points[1]. These paths are then expanded if
> necessary (due to platform-specific tests and virtual test suites) and all
> the tests to which these paths apply are gathered[2]. All the gathered tests
> are then either sorted into a natural order or randomized (if the
> --randomize-order option is used)[3].
>
> I'd like to propose a change in the behavior when using the --test-list
> option. When used, the tests would be gathered only from the paths listed in
> the file to which the option points. Also, the gathered tests would be
> sorted to follow the order of the paths in the test list file. For instance,
> consider the following two entries being placed in the test list file:
>
> c/d/e.html
> a/b/
>
> The current behavior would gather all the tests to which these two paths
> apply and sort them in this order (on the premise that no other paths are
> passed through the command line arguments):
>
> a/b/1.html
> a/b/2.html
> c/d/e.html
>
> The new behavior would place the c/d/e.html test at the top of the list but
> sort the other two tests in order, like this:
>
> c/d/e.html
> a/b/1.html
> a/b/2.html
>
> The idea behind this is to be able to specify the exact order in which the
> tests should be run. I'm currently playing around with a tool that randomly
> chooses a certain number of tests, runs them and bisects down the first
> regression that occurs (if any) to the smallest possible ordered list of
> tests that reproduces that regression. The proposed change makes this a
> simple task from the test listing perspective and I'd argue that exact test
> ordering is the main point of an option such as --test-list.
>
> Currently I'm using an untested workaround of adding a new option that
> specifies the tests should be reordered back to the original order that the
> test list file provided, but I'd like to move on with implementing the
> change in behavior if everyone feels it's OK and won't tamper with his/her
> workflow.
>
> -Z
>
>
> [1]
> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46
> [2]
> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576
> [3]
> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309
>

Hi,

I have used the option from time to time, and I have also wanted from
time to time the ability to run tests in a specific order. This change
would be fine by me. If people want to ensure that tests run in the
old order, they can always sort the test list.

I would also be fine changing the code so that order is preserved for
tests passed in on the command line as well. (e.g. so
"run-webkit-tests foo.html bar.html" runs them in that order).

I think it would be good to keep the behavior consistent across the
two methods of specifying names.

-- Dirk