[webkit-dev] run-bindings-tests

[James Robinson]

On Thu, Sep 8, 2011 at 11:49 AM, Alexey Proskuryakov <ap at webkit.org> wrote:

>
> 08.09.2011, в 11:32, Eric Seidel написал(а):
>
> > FYI:  As many of you already know, the build.webkit.org bots run
> > "run-bindings-tests" on (almost) all platforms.
> >
> > They've been running (mostly w/o incident) on the bots since 6/20:
> > http://trac.webkit.org/changeset/89267
> >
> >
> > These just make sure that our generated bindings look sane, by
> > comparing the generated results against checked-in baselines.
> >
> > run-bindings-tests makes it easier to make cross-platform bindings
> > changes w/o needing a Gtk/Qt/V8/etc. port of WebKit.
> >
> > If you're changing binding generation you should be aware of this script.
>
> As discussed on IRC, I do not think that bots should run this test at all.
> It has a non-trivial maintenance cost, but provides very little benefit.
> Even the time spent by multiple engineers on IRC today discussing bot
> complaints is likely more than the test could save in the lifetime of the
> project, at my guesstimate.
>
> A test like this is almost like keeping a separate text file with a number
> of space characters in WebKit sources, and chastising anyone who fails to
> update this text file with their commit. Why would we care about the number
> of spaces, or about the exact look of generated code?
>
> Specifically, this is today's failure: <
> http://build.webkit.org/builders/SnowLeopard%20Intel%20Release%20%28Tests%29/builds/32923/steps/bindings-generation-tests/logs/stdio>
> - a test that complains about such changes doesn't test for the right thing.
>
> A script like this might be useful to run locally when making bindings
> changes if in doubt, comparing "before" and "after" results. There is no
> need to check in most recent results for this though. I'm not sure if this
> gives you more than manually copying DerivedSources directory and diffing
> new derived sources to that, but if someone finds a little automation
> valuable, then why not.
>


We used to not run these tests on the bots.  This meant that people would
change the bindings code and not update the expected results, so the
expected results were always massively out of date.  This meant when
patching the bindings scripts you could not rely on run-bindings-tests at
all, because the expectations were already broken before you made any
changes!  This it not theoretical, it happened to be multiple times and I
know I'm not the only one.

The real problem here is that people check in without looking at the bots
and then do not respond when the bots go red.  That's a people problem.

- James



> - WBR, Alexey Proskuryakov
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110908/02e2863a/attachment.html>