[webkit-dev] Testing changes to CodeGenerator*.pm

[Adam Barth]

On Thu, Apr 29, 2010 at 12:43 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> To repeat, I think this is a useful tool, but not necessarily as a test.
> The mode of operation is that when you run this test, if the generated
> bindings for the text IDL file change in any way, it reports a failure. Then
> you must manually take the step of regenerating the golden master file. It
> doesn't seem like the failure report itself will result in a decision. It
> doesn't provide interesting information until you regenerate the test file
> and look at the diffs, except in the highly unusual case of changing the
> output of the codegen script unintentially. Most changes to the codegen
> script are to intentionally change the output.

The failure report shows you the diff.  In hacking on CodeGeneratorJS,
I often run the script many times to see how my changes are affecting
the generated code, much in the same way I would run all the
LayoutTests constantly if they took <5s to run.

> It seems to me a better model would be to regenerate the bindings test file
> automatically as part of the build. Then the changes can still be reviewed
> by you, or as part of a diff, but there would be no extra manual steps
> involved. And people making behaviorally transparent changes to codegen
> output would perhaps feel a little less burdened.

That sounds like a good improvement.  I think it would be fine to
regenerate the output as part of the build.  However, I think we
should still preserve the ability to run the script along it "test"
mode because that's about three orders of magnitude faster than
performing a build after touching CodeGeneratorJS.

> That makes more sense to me than treating it as a regression test. It also
> seems like this model would still have all the benefits you cite above.

What I hear from this conversation is the following:

1) A bunch of people who've used the tool saying that they've found it useful.
2) A bunch of people who haven't used the tool suggesting improvements.

This tool impacts the handful of people who work on
CodeGeneratorJS.pm.  Everyone else in the project can safely ignore
it.  I'm all for improvements, and I invite anyone interested to
either improve the tool or write a new tool that does the job better.

Adam