[webkit-dev] run-bindings-tests

[Oliver Hunt]

On Sep 8, 2011, at 11:55 AM, James Robinson wrote:
> 
> We used to not run these tests on the bots.  This meant that people would change the bindings code and not update the expected results, so the expected results were always massively out of date.  This meant when patching the bindings scripts you could not rely on run-bindings-tests at all, because the expectations were already broken before you made any changes!  This it not theoretical, it happened to be multiple times and I know I'm not the only one.
> 
> The real problem here is that people check in without looking at the bots and then do not respond when the bots go red.  That's a people problem.

The real problem is that we have a test suite: run-webkit-tests, that everyone runs (it's even mentioned at step 4 on http://www.webkit.org/coding/contributing.html , which is apparently not running all the tests.

I run-webkit-tests before i commit (new-run-webkit-tests has ensured that any prior complains about time taken no longer exist, so kudos to those folk \o/ ), and yet I end up breaking the build in a way that would show up locally if the tests were simply run.  run-webkit-tests should run all of the webkit tests -- not some subset, all of them.  If failing a cross platform test can turn the bots red, then that test should be covered by run-webkit-tests.

The other problem is "people check in without looking at the bots".  I do try to watch the bots, but the time between me landing and the bots actually going red can literally be hours.  Of course i'm away from irc/email whatever when I land a patch at 4pm and it turns the bots red at 11pm.

I appreciate this isn't as much of a problem for people who don't work on code for which all changes heralds a world rebuild and effect (apparently) every single test that exists in the repository, but it's certainly frustrating for those of us who do.

(This is ignoring the overly aggressive rollouts of large patches the break only single platforms due to platform specific code that is difficult for anyone outside of that platform to fix)

--Oliver

> - James
> 
> 
> 
> - WBR, Alexey Proskuryakov
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110908/4530e48f/attachment.html>