[webkit-dev] The --test-list option behavior in NRWT

[Balazs Kelemen]

On 11/08/2012 02:51 AM, Ojan Vafai wrote:
> On Wed, Nov 7, 2012 at 12:41 PM, Dirk Pranke <dpranke at chromium.org 
> <mailto:dpranke at chromium.org>> wrote:
>
>     On Tue, Nov 6, 2012 at 11:58 PM, Z(an Dobers(ek
>     <zandobersek at gmail.com <mailto:zandobersek at gmail.com>> wrote:
>     > Hi WebKitties,
>     >
>     > I'd like to see in what scale the --test-list option in NRWT is
>     used and
>     > whether anyone would object a change in how its usage behaves.
>     >
>     > Currently the test gathering takes into account any paths that
>     were passed
>     > in through the command line and any paths listed in the file to
>     which a
>     > possible --test-file option points[1]. These paths are then
>     expanded if
>     > necessary (due to platform-specific tests and virtual test
>     suites) and all
>     > the tests to which these paths apply are gathered[2]. All the
>     gathered tests
>     > are then either sorted into a natural order or randomized (if the
>     > --randomize-order option is used)[3].
>     >
>     > I'd like to propose a change in the behavior when using the
>     --test-list
>     > option. When used, the tests would be gathered only from the
>     paths listed in
>     > the file to which the option points. Also, the gathered tests
>     would be
>     > sorted to follow the order of the paths in the test list file.
>     For instance,
>     > consider the following two entries being placed in the test list
>     file:
>     >
>     > c/d/e.html
>     > a/b/
>     >
>     > The current behavior would gather all the tests to which these
>     two paths
>     > apply and sort them in this order (on the premise that no other
>     paths are
>     > passed through the command line arguments):
>     >
>     > a/b/1.html
>     > a/b/2.html
>     > c/d/e.html
>     >
>     > The new behavior would place the c/d/e.html test at the top of
>     the list but
>     > sort the other two tests in order, like this:
>     >
>     > c/d/e.html
>     > a/b/1.html
>     > a/b/2.html
>     >
>     > The idea behind this is to be able to specify the exact order in
>     which the
>     > tests should be run. I'm currently playing around with a tool
>     that randomly
>     > chooses a certain number of tests, runs them and bisects down
>     the first
>     > regression that occurs (if any) to the smallest possible ordered
>     list of
>     > tests that reproduces that regression. The proposed change makes
>     this a
>     > simple task from the test listing perspective and I'd argue that
>     exact test
>     > ordering is the main point of an option such as --test-list.
>     >
>     > Currently I'm using an untested workaround of adding a new
>     option that
>     > specifies the tests should be reordered back to the original
>     order that the
>     > test list file provided, but I'd like to move on with
>     implementing the
>     > change in behavior if everyone feels it's OK and won't tamper
>     with his/her
>     > workflow.
>     >
>     > -Z
>     >
>     >
>     > [1]
>     >
>     http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46
>     > [2]
>     >
>     http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576
>     > [3]
>     >
>     http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309
>     >
>
>     Hi,
>
>     I have used the option from time to time, and I have also wanted from
>     time to time the ability to run tests in a specific order. This change
>     would be fine by me. If people want to ensure that tests run in the
>     old order, they can always sort the test list.
>
>
> I have used this in the past to identify which test caused test 
> flakiness (e.g. I have 100 tests and I know the last test depends on 
> one of the other 99 to run first in order to pass). The problem with 
> this last sentence is that it's hard to know what order 
> run-webkit-tests would choose, so it's hard to replicate it. It would 
> make that use-case a little harder if we made this change. Otherwise, 
> I don't care either way.

Exactly for the reason Ojan mentioned I think the best would be to add 
an option for defining the desired ordering behavior. Paths on command 
line and in --test-list should imply to use the ordering as it is passed 
but it would be possible to force the current method. I hope it would 
not add too much complexity.

-kbalazs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121108/4180e5bd/attachment.html>