[webkit-dev] The --test-list option behavior in NRWT

[Ojan Vafai]

On Thu, Nov 8, 2012 at 2:00 AM, Žan Doberšek <zandobersek at gmail.com> wrote:

> I'd propose an --ordering option, with three possible values:
> - random, randomizes the test list
> - natural, orders the test list in natural order, which is the current
> behavior
> - none, keeps the order that was specified in the arguments or test list,
> default (not certain about the name, though)
>
> The --randomized option would be kept for backwards compatibility, but
> perhaps deprecated. I'll build up a patch that's based on the consensus.
>

Sounds good to me. I don't think we need to keep the --randomized option
though. Fewer flags == better. We already have too many flags.


>
> -Z
>
>
> On Thu, Nov 8, 2012 at 10:43 AM, Balazs Kelemen <kbalazs at webkit.org>wrote:
>
>>  On 11/08/2012 02:51 AM, Ojan Vafai wrote:
>>
>>  On Wed, Nov 7, 2012 at 12:41 PM, Dirk Pranke <dpranke at chromium.org>wrote:
>>
>>>  On Tue, Nov 6, 2012 at 11:58 PM, Žan Doberšek <zandobersek at gmail.com>
>>> wrote:
>>> > Hi WebKitties,
>>> >
>>> > I'd like to see in what scale the --test-list option in NRWT is used
>>> and
>>> > whether anyone would object a change in how its usage behaves.
>>> >
>>> > Currently the test gathering takes into account any paths that were
>>> passed
>>> > in through the command line and any paths listed in the file to which a
>>> > possible --test-file option points[1]. These paths are then expanded if
>>> > necessary (due to platform-specific tests and virtual test suites) and
>>> all
>>> > the tests to which these paths apply are gathered[2]. All the gathered
>>> tests
>>> > are then either sorted into a natural order or randomized (if the
>>> > --randomize-order option is used)[3].
>>> >
>>> > I'd like to propose a change in the behavior when using the --test-list
>>> > option. When used, the tests would be gathered only from the paths
>>> listed in
>>> > the file to which the option points. Also, the gathered tests would be
>>> > sorted to follow the order of the paths in the test list file. For
>>> instance,
>>> > consider the following two entries being placed in the test list file:
>>> >
>>> > c/d/e.html
>>> > a/b/
>>> >
>>> > The current behavior would gather all the tests to which these two
>>> paths
>>> > apply and sort them in this order (on the premise that no other paths
>>> are
>>> > passed through the command line arguments):
>>> >
>>> > a/b/1.html
>>> > a/b/2.html
>>> > c/d/e.html
>>> >
>>> > The new behavior would place the c/d/e.html test at the top of the
>>> list but
>>> > sort the other two tests in order, like this:
>>> >
>>> > c/d/e.html
>>> > a/b/1.html
>>> > a/b/2.html
>>> >
>>> > The idea behind this is to be able to specify the exact order in which
>>> the
>>> > tests should be run. I'm currently playing around with a tool that
>>> randomly
>>> > chooses a certain number of tests, runs them and bisects down the first
>>> > regression that occurs (if any) to the smallest possible ordered list
>>> of
>>> > tests that reproduces that regression. The proposed change makes this a
>>> > simple task from the test listing perspective and I'd argue that exact
>>> test
>>> > ordering is the main point of an option such as --test-list.
>>> >
>>> > Currently I'm using an untested workaround of adding a new option that
>>> > specifies the tests should be reordered back to the original order
>>> that the
>>> > test list file provided, but I'd like to move on with implementing the
>>> > change in behavior if everyone feels it's OK and won't tamper with
>>> his/her
>>> > workflow.
>>> >
>>> > -Z
>>> >
>>> >
>>> > [1]
>>> >
>>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46
>>> > [2]
>>> >
>>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576
>>> > [3]
>>> >
>>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309
>>> >
>>>
>>>  Hi,
>>>
>>> I have used the option from time to time, and I have also wanted from
>>> time to time the ability to run tests in a specific order. This change
>>> would be fine by me. If people want to ensure that tests run in the
>>> old order, they can always sort the test list.
>>>
>>
>>  I have used this in the past to identify which test caused test
>> flakiness (e.g. I have 100 tests and I know the last test depends on one of
>> the other 99 to run first in order to pass). The problem with this last
>> sentence is that it's hard to know what order run-webkit-tests would
>> choose, so it's hard to replicate it. It would make that use-case a little
>> harder if we made this change. Otherwise, I don't care either way.
>>
>>
>> Exactly for the reason Ojan mentioned I think the best would be to add an
>> option for defining the desired ordering behavior. Paths on command line
>> and in --test-list should imply to use the ordering as it is passed but it
>> would be possible to force the current method. I hope it would not add too
>> much complexity.
>>
>> -kbalazs
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121108/9b0c2dec/attachment.html>