[webkit-dev] Testing changes to CodeGenerator*.pm

[Adam Barth]

On Thu, Apr 29, 2010 at 10:54 AM, Adam Barth <abarth at webkit.org> wrote:
> On Thu, Apr 29, 2010 at 10:39 AM, Alexey Proskuryakov <ap at webkit.org> wrote:
>> On 29.04.2010, at 10:27, Jeremy Orlow wrote:
>>> It's great to test end-to-end behavior, and unit tests can also also
>>> useful sometimes, but why test that source code stays byte to byte
>>> identical?
>>>
>>> When you make a change to the code generator, you should make a
>>> corresponding change to the generated test code.  This allows the reviewer
>>> to see how the change in CodeGenerator*.pm affects the generated code and
>>> documents the change in SVN.  The hope is that it'll be obvious if your
>>> change is going to have unintended consequences on the generated code.
>>>  We've found this useful when working on the V8 bindings.
>>
>> For these goals to be achieved, we'd need to check all generated code, not
>> just one custom test file for each language.
>
> The idea is that the test IDL file exercises the features of the code
> generator.  Sure, it's not 100% coverage, but it's much better not
> testing it at all.
>
>> It would be great to have a tool that generates a diff of derived sources
>> for inspection, but making it into a test for everyone to maintain feels
>> like unnecessary burden. I certainly would feel bad about having to maintain
>> a test that verifies source file content instead of behavior.
>
> You should feel free to develop a better testing harness.  This one
> certainly isn't best conceivable tool, but it's better than what we
> had previously, which was essentially the C++ compiler.
>
> The maintenance is super easy.  I've been doing a lot of development
> work on the code generator in the past few days, and it amounts to
> typing a single command:
>
> ./WebKitTools/Scripts/run-bindings-tests --reset-results
>
> The harness has been super useful in working on the code generator
> because the tests run in a few seconds.  That lets me iterate on the
> script much more quickly compared to rebuilding the world every time I
> want to try a tweak.

As an additional datapoint on maintenance, it looks like you've
touched CodeGeneratorJS.pm twice in the past year and neither of those
change would have resulted in a large diff to the expected test
results.

By contrast, I've made five changes in the past two days and plan to
made dozens more changes shortly to reduce the amount of custom
bindings code we have in the tree, which is a serious maintenance
problem because it is full of subtle, quirky behavior.

Adam