[webkit-dev] Testing changes to CodeGenerator*.pm

[Maciej Stachowiak]

On Apr 29, 2010, at 11:38 AM, Ojan Vafai wrote:

> I don't really follow the argument against supporting this test.  
> What is the maintenance overhead? Isn't this just like having a  
> layout test with expected results? It's a small isolated test  
> instead of testing everything. That seems like a good thing.
>
> More importantly, it lets you be sure that every feature of the code  
> generator has some testing. In the real IDLs, a feature might stop  
> getting used temporarily and then changes to the code generator  
> would not be readily apparent.

I don't think your comments above are responsive to my paragraph below:

>
> On Thu, Apr 29, 2010 at 11:05 AM, Maciej Stachowiak <mjs at apple.com>  
> wrote:
>
> It also strikes me as odd to do testing by doing exact comparison of  
> the generated source. But I can also see side benefits. I think the  
> real issue here may be one of naming. If the use model is that you  
> fully regenerate the results every time you make a change to the  
> bindings generator, then it's not really a regression test. The  
> purpose is not to catch unintentional changes but rather to be able  
> to observe changes to generated code, and new kinds of generated  
> code, while working on a change and when reviewing code. Perhaps the  
> tool should have a name that reflects that, instead of implying that  
> the purpose is to catch bugs accidentally introduced by changes. It  
> doesn't seem like an efficient or effective way to do the latter.
>

To repeat, I think this is a useful tool, but not necessarily as a test.

The mode of operation is that when you run this test, if the generated  
bindings for the text IDL file change in any way, it reports a  
failure. Then you must manually take the step of regenerating the  
golden master file. It doesn't seem like the failure report itself  
will result in a decision. It doesn't provide interesting information  
until you regenerate the test file and look at the diffs, except in  
the highly unusual case of changing the output of the codegen script  
unintentially. Most changes to the codegen script are to intentionally  
change the output.

It seems to me a better model would be to regenerate the bindings test  
file automatically as part of the build. Then the changes can still be  
reviewed by you, or as part of a diff, but there would be no extra  
manual steps involved. And people making behaviorally transparent  
changes to codegen output would perhaps feel a little less burdened.

That makes more sense to me than treating it as a regression test. It  
also seems like this model would still have all the benefits you cite  
above.

Regards,
Maciej

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100429/09a29255/attachment.html>