[webkit-dev] Testing changes to CodeGenerator*.pm

[Adam Barth]

On Thu, Apr 29, 2010 at 10:39 AM, Alexey Proskuryakov <ap at webkit.org> wrote:
> On 29.04.2010, at 10:27, Jeremy Orlow wrote:
>> It's great to test end-to-end behavior, and unit tests can also also
>> useful sometimes, but why test that source code stays byte to byte
>> identical?
>>
>> When you make a change to the code generator, you should make a
>> corresponding change to the generated test code.  This allows the reviewer
>> to see how the change in CodeGenerator*.pm affects the generated code and
>> documents the change in SVN.  The hope is that it'll be obvious if your
>> change is going to have unintended consequences on the generated code.
>>  We've found this useful when working on the V8 bindings.
>
> For these goals to be achieved, we'd need to check all generated code, not
> just one custom test file for each language.

The idea is that the test IDL file exercises the features of the code
generator.  Sure, it's not 100% coverage, but it's much better not
testing it at all.

> It would be great to have a tool that generates a diff of derived sources
> for inspection, but making it into a test for everyone to maintain feels
> like unnecessary burden. I certainly would feel bad about having to maintain
> a test that verifies source file content instead of behavior.

You should feel free to develop a better testing harness.  This one
certainly isn't best conceivable tool, but it's better than what we
had previously, which was essentially the C++ compiler.

The maintenance is super easy.  I've been doing a lot of development
work on the code generator in the past few days, and it amounts to
typing a single command:

./WebKitTools/Scripts/run-bindings-tests --reset-results

The harness has been super useful in working on the code generator
because the tests run in a few seconds.  That lets me iterate on the
script much more quickly compared to rebuilding the world every time I
want to try a tweak.

Adam