[webkit-dev] can we stop using Skipped files?

[Filip Pizlo]

On Jun 8, 2012, at 1:04 PM, Ryosuke Niwa wrote:

> On Fri, Jun 8, 2012 at 12:50 PM, Filip Pizlo <fpizlo at apple.com> wrote:
> On Jun 8, 2012, at 12:31 PM, Dirk Pranke wrote:
>  > On Fri, Jun 8, 2012 at 10:56 AM, Filip Pizlo <fpizlo at apple.com> wrote:
> >>
> >> It's a lot harder to dive into, a lot more cumbersome to improve, and not
> >> any easier to maintain.
> >>
> >
> > I definitely agree that NRWT is more complicated than it seems like it
> > should be; it got contorted as we added all the features we needed to
> > add, and I have been in a "simplify" mode over the past few months. I
> > would welcome any feedback where you think things are overly complex.
> 
> This is a difficult question - it's unfortunately easier to observe that something is complex, than it is to pinpoint why it is complex.  But I will try.
> 
> 1) Code locality.  I can open Tools/Scripts/old-run-webkit-tests and pretty rapidly discover (a) how options are parsed, (b) how platform differences are handled, (c) how tests are found, and (d) how tests are run.  I can hack all of this code because it's all in one place.  I don't have to be a domain expert to do it.  Hell, I don't even have to be good a Perl to find my way around.
> 
> 2) Code size:
> 
> [pizlo at wartooth OpenSource] wc Tools/Scripts/old-run-webkit-tests
>    2796   10316   98733 Tools/Scripts/old-run-webkit-tests
> 
> [pizlo at wartooth OpenSource] wc Tools/Scripts/new-run-webkit-tests Tools/Scripts/webkitpy/layout_tests/*.py Tools/Scripts/webkitpy/layout_tests/*/*.py
> .... bunch of stuff
>   23197 91897 1049914 total
> 
> That's a *HUGE* difference.  Consider that NRWT just adds only one thing that most people care about: parallelism.  Is an 8x increase in code size justified?
> 
> I know that LoC metrics are evil in most cases.  But this is not most cases.  This is an order-of-magnitude difference.  That's 8x more code I have to look at to find what I want.  That's 8x more code that I potentially have to edit to add a feature.  That's 8x more code that could have a bug.  And so on.  Badness!
> 
> Part of that comes from the fact layout_tests directory also includes port classes, which knows various aspects of each port, and they're also used in webkit-patch and other Python tools.
> 
> But the act of running layout tests is in itself a regression test of the layout test harness just as much as it is a regression test of WebKit.
> 
> So the unit tests are superfluous.  In particular, if I had to pick between only having unit tests or only having regression tests, I might pick unit tests.  But if I already have regression tests then I'm unlikely to want to incur technical debt to build unit tests, particularly since unit tests requiring changing the infrastructure to make the code more testable, which then leads to the problems listed above.
> 
> I agree that we've accumulated way too many unit tests in webkitpy and some of our unit test code is hideous but I think it's quite unrealistic not to have any unit tests. We've had many regressions in the past, and it's hard to make sure your code works on all platforms without tests. So it's the balance that's important.

I think you've had regressions because the code is too complex.

> 
> I am intrigued by the notion of using Erlang, but worry that it would reduce effective hackability due to there being less Erlang experience in the universe.  I also don't want the porting of Erlang's runtime to be a gating factor for porting WebKit itself.  Hence, I fear that we should stick to broadly accepted languages, like Python, or Perl, or if need be, C++.
> 
> If I were to write a new test runner, I'd use C++ without a question. It has solid threading support in almost all operating systems, is statically typed, and virtually everyone who contributes to WebKit can read/write well.

I would try to avoid using C++ unless we really had to.  The goal here is simplicity, and C++ is not the best for that.

> 
> - Ryosuke
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120608/c8787f3d/attachment.html>