[webkit-dev] The --test-list option behavior in NRWT

[Ojan Vafai]

On Wed, Nov 7, 2012 at 12:41 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> On Tue, Nov 6, 2012 at 11:58 PM, Žan Doberšek <zandobersek at gmail.com>
> wrote:
> > Hi WebKitties,
> >
> > I'd like to see in what scale the --test-list option in NRWT is used and
> > whether anyone would object a change in how its usage behaves.
> >
> > Currently the test gathering takes into account any paths that were
> passed
> > in through the command line and any paths listed in the file to which a
> > possible --test-file option points[1]. These paths are then expanded if
> > necessary (due to platform-specific tests and virtual test suites) and
> all
> > the tests to which these paths apply are gathered[2]. All the gathered
> tests
> > are then either sorted into a natural order or randomized (if the
> > --randomize-order option is used)[3].
> >
> > I'd like to propose a change in the behavior when using the --test-list
> > option. When used, the tests would be gathered only from the paths
> listed in
> > the file to which the option points. Also, the gathered tests would be
> > sorted to follow the order of the paths in the test list file. For
> instance,
> > consider the following two entries being placed in the test list file:
> >
> > c/d/e.html
> > a/b/
> >
> > The current behavior would gather all the tests to which these two paths
> > apply and sort them in this order (on the premise that no other paths are
> > passed through the command line arguments):
> >
> > a/b/1.html
> > a/b/2.html
> > c/d/e.html
> >
> > The new behavior would place the c/d/e.html test at the top of the list
> but
> > sort the other two tests in order, like this:
> >
> > c/d/e.html
> > a/b/1.html
> > a/b/2.html
> >
> > The idea behind this is to be able to specify the exact order in which
> the
> > tests should be run. I'm currently playing around with a tool that
> randomly
> > chooses a certain number of tests, runs them and bisects down the first
> > regression that occurs (if any) to the smallest possible ordered list of
> > tests that reproduces that regression. The proposed change makes this a
> > simple task from the test listing perspective and I'd argue that exact
> test
> > ordering is the main point of an option such as --test-list.
> >
> > Currently I'm using an untested workaround of adding a new option that
> > specifies the tests should be reordered back to the original order that
> the
> > test list file provided, but I'd like to move on with implementing the
> > change in behavior if everyone feels it's OK and won't tamper with
> his/her
> > workflow.
> >
> > -Z
> >
> >
> > [1]
> >
> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46
> > [2]
> >
> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576
> > [3]
> >
> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309
> >
>
> Hi,
>
> I have used the option from time to time, and I have also wanted from
> time to time the ability to run tests in a specific order. This change
> would be fine by me. If people want to ensure that tests run in the
> old order, they can always sort the test list.
>

I have used this in the past to identify which test caused test flakiness
(e.g. I have 100 tests and I know the last test depends on one of the other
99 to run first in order to pass). The problem with this last sentence is
that it's hard to know what order run-webkit-tests would choose, so it's
hard to replicate it. It would make that use-case a little harder if we
made this change. Otherwise, I don't care either way.


> I would also be fine changing the code so that order is preserved for
> tests passed in on the command line as well. (e.g. so
> "run-webkit-tests foo.html bar.html" runs them in that order).
>
> I think it would be good to keep the behavior consistent across the
> two methods of specifying names.
>
> -- Dirk
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121107/2cd10125/attachment.html>