[webkit-dev] The --test-list option behavior in NRWT

[Žan Doberšek]

I'd propose an --ordering option, with three possible values:
- random, randomizes the test list
- natural, orders the test list in natural order, which is the current
behavior
- none, keeps the order that was specified in the arguments or test list,
default (not certain about the name, though)

The --randomized option would be kept for backwards compatibility, but
perhaps deprecated. I'll build up a patch that's based on the consensus.

-Z


On Thu, Nov 8, 2012 at 10:43 AM, Balazs Kelemen <kbalazs at webkit.org> wrote:

>  On 11/08/2012 02:51 AM, Ojan Vafai wrote:
>
>  On Wed, Nov 7, 2012 at 12:41 PM, Dirk Pranke <dpranke at chromium.org>wrote:
>
>>  On Tue, Nov 6, 2012 at 11:58 PM, Žan Doberšek <zandobersek at gmail.com>
>> wrote:
>> > Hi WebKitties,
>> >
>> > I'd like to see in what scale the --test-list option in NRWT is used and
>> > whether anyone would object a change in how its usage behaves.
>> >
>> > Currently the test gathering takes into account any paths that were
>> passed
>> > in through the command line and any paths listed in the file to which a
>> > possible --test-file option points[1]. These paths are then expanded if
>> > necessary (due to platform-specific tests and virtual test suites) and
>> all
>> > the tests to which these paths apply are gathered[2]. All the gathered
>> tests
>> > are then either sorted into a natural order or randomized (if the
>> > --randomize-order option is used)[3].
>> >
>> > I'd like to propose a change in the behavior when using the --test-list
>> > option. When used, the tests would be gathered only from the paths
>> listed in
>> > the file to which the option points. Also, the gathered tests would be
>> > sorted to follow the order of the paths in the test list file. For
>> instance,
>> > consider the following two entries being placed in the test list file:
>> >
>> > c/d/e.html
>> > a/b/
>> >
>> > The current behavior would gather all the tests to which these two paths
>> > apply and sort them in this order (on the premise that no other paths
>> are
>> > passed through the command line arguments):
>> >
>> > a/b/1.html
>> > a/b/2.html
>> > c/d/e.html
>> >
>> > The new behavior would place the c/d/e.html test at the top of the list
>> but
>> > sort the other two tests in order, like this:
>> >
>> > c/d/e.html
>> > a/b/1.html
>> > a/b/2.html
>> >
>> > The idea behind this is to be able to specify the exact order in which
>> the
>> > tests should be run. I'm currently playing around with a tool that
>> randomly
>> > chooses a certain number of tests, runs them and bisects down the first
>> > regression that occurs (if any) to the smallest possible ordered list of
>> > tests that reproduces that regression. The proposed change makes this a
>> > simple task from the test listing perspective and I'd argue that exact
>> test
>> > ordering is the main point of an option such as --test-list.
>> >
>> > Currently I'm using an untested workaround of adding a new option that
>> > specifies the tests should be reordered back to the original order that
>> the
>> > test list file provided, but I'd like to move on with implementing the
>> > change in behavior if everyone feels it's OK and won't tamper with
>> his/her
>> > workflow.
>> >
>> > -Z
>> >
>> >
>> > [1]
>> >
>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46
>> > [2]
>> >
>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576
>> > [3]
>> >
>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309
>> >
>>
>>  Hi,
>>
>> I have used the option from time to time, and I have also wanted from
>> time to time the ability to run tests in a specific order. This change
>> would be fine by me. If people want to ensure that tests run in the
>> old order, they can always sort the test list.
>>
>
>  I have used this in the past to identify which test caused test flakiness
> (e.g. I have 100 tests and I know the last test depends on one of the other
> 99 to run first in order to pass). The problem with this last sentence is
> that it's hard to know what order run-webkit-tests would choose, so it's
> hard to replicate it. It would make that use-case a little harder if we
> made this change. Otherwise, I don't care either way.
>
>
> Exactly for the reason Ojan mentioned I think the best would be to add an
> option for defining the desired ordering behavior. Paths on command line
> and in --test-list should imply to use the ordering as it is passed but it
> would be possible to force the current method. I hope it would not add too
> much complexity.
>
> -kbalazs
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121108/0558596c/attachment.html>