[webkit-dev] can we stop using Skipped files?

[Ryosuke Niwa]

On Fri, Jun 8, 2012 at 12:50 PM, Filip Pizlo <fpizlo at apple.com> wrote:

> On Jun 8, 2012, at 12:31 PM, Dirk Pranke wrote:
>  > On Fri, Jun 8, 2012 at 10:56 AM, Filip Pizlo <fpizlo at apple.com> wrote:
> >>
> >> It's a lot harder to dive into, a lot more cumbersome to improve, and
> not
> >> any easier to maintain.
> >>
> >
> > I definitely agree that NRWT is more complicated than it seems like it
> > should be; it got contorted as we added all the features we needed to
> > add, and I have been in a "simplify" mode over the past few months. I
> > would welcome any feedback where you think things are overly complex.
>
> This is a difficult question - it's unfortunately easier to observe that
> something is complex, than it is to pinpoint why it is complex.  But I will
> try.
>
> 1) Code locality.  I can open Tools/Scripts/old-run-webkit-tests and
> pretty rapidly discover (a) how options are parsed, (b) how platform
> differences are handled, (c) how tests are found, and (d) how tests are
> run.  I can hack all of this code because it's all in one place.  I don't
> have to be a domain expert to do it.  Hell, I don't even have to be good a
> Perl to find my way around.
>
> 2) Code size:
>
> [pizlo at wartooth OpenSource] wc Tools/Scripts/old-run-webkit-tests
>    2796   10316   98733 Tools/Scripts/old-run-webkit-tests
>
> [pizlo at wartooth OpenSource] wc Tools/Scripts/new-run-webkit-tests
> Tools/Scripts/webkitpy/layout_tests/*.py
> Tools/Scripts/webkitpy/layout_tests/*/*.py
> .... bunch of stuff
>   23197 91897 1049914 total
>
> That's a *HUGE* difference.  Consider that NRWT just adds only one thing
> that most people care about: parallelism.  Is an 8x increase in code size
> justified?
>
> I know that LoC metrics are evil in most cases.  But this is not most
> cases.  This is an order-of-magnitude difference.  That's 8x more code I
> have to look at to find what I want.  That's 8x more code that I
> potentially have to edit to add a feature.  That's 8x more code that could
> have a bug.  And so on.  Badness!
>

Part of that comes from the fact layout_tests directory also includes port
classes, which knows various aspects of each port, and they're also used in
webkit-patch and other Python tools.

But the act of running layout tests is in itself a regression test of the
> layout test harness just as much as it is a regression test of WebKit.
>
> So the unit tests are superfluous.  In particular, if I had to pick
> between only having unit tests or only having regression tests, I might
> pick unit tests.  But if I already have regression tests then I'm unlikely
> to want to incur technical debt to build unit tests, particularly since
> unit tests requiring changing the infrastructure to make the code more
> testable, which then leads to the problems listed above.
>

I agree that we've accumulated way too many unit tests in webkitpy and some
of our unit test code is hideous but I think it's quite unrealistic not to
have any unit tests. We've had many regressions in the past, and it's hard
to make sure your code works on all platforms without tests. So it's the
balance that's important.

I am intrigued by the notion of using Erlang, but worry that it would
> reduce effective hackability due to there being less Erlang experience in
> the universe.  I also don't want the porting of Erlang's runtime to be a
> gating factor for porting WebKit itself.  Hence, I fear that we should
> stick to broadly accepted languages, like Python, or Perl, or if need be,
> C++.
>

If I were to write a new test runner, I'd use C++ without a question. It
has solid threading support in almost all operating systems, is statically
typed, and virtually everyone who contributes to WebKit can read/write well.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120608/50c94662/attachment.html>