[webkit-dev] run-bindings-tests

[Eric Seidel]

If the objection against run-bindings-tests is that they're not part
of some larger test script which developers can run locally, it's very
easy to add a wrapper script which runs all known testing harnesses.

The test tests which currently run on the bots include:
run-webkit-tests (minutes)
run-javascriptcore-tests (45s)
test-webkitpy (32s)
test-webkitperl (2.0s)
run-binding-tests (2.4s)
run-api-tests (failed on my machine, so I can't tell you how long)

There are a couple other scripts which run on specific platforms, but
that's the core set.

run-webkit-tests is the bulk of the time, taking multiple minutes on a
modern machine (even with NRWT).

I'm happy to write a run-all-tests script which runs all known tests
that platform can handle. :)

-eric


On Thu, Sep 8, 2011 at 12:19 PM, Oliver Hunt <oliver at apple.com> wrote:
>
> On Sep 8, 2011, at 11:55 AM, James Robinson wrote:
>
> We used to not run these tests on the bots.  This meant that people would
> change the bindings code and not update the expected results, so the
> expected results were always massively out of date.  This meant when
> patching the bindings scripts you could not rely on run-bindings-tests at
> all, because the expectations were already broken before you made any
> changes!  This it not theoretical, it happened to be multiple times and I
> know I'm not the only one.
> The real problem here is that people check in without looking at the bots
> and then do not respond when the bots go red.  That's a people problem.
>
> The real problem is that we have a test suite: run-webkit-tests, that
> everyone runs (it's even mentioned at step 4 on
> http://www.webkit.org/coding/contributing.html , which is apparently not
> running all the tests.
> I run-webkit-tests before i commit (new-run-webkit-tests has ensured that
> any prior complains about time taken no longer exist, so kudos to those folk
> \o/ ), and yet I end up breaking the build in a way that would show up
> locally if the tests were simply run.  run-webkit-tests should run all of
> the webkit tests -- not some subset, all of them.  If failing a cross
> platform test can turn the bots red, then that test should be covered by
> run-webkit-tests.
> The other problem is "people check in without looking at the bots".  I do
> try to watch the bots, but the time between me landing and the bots actually
> going red can literally be hours.  Of course i'm away from irc/email
> whatever when I land a patch at 4pm and it turns the bots red at 11pm.
> I appreciate this isn't as much of a problem for people who don't work on
> code for which all changes heralds a world rebuild and effect (apparently)
> every single test that exists in the repository, but it's certainly
> frustrating for those of us who do.
> (This is ignoring the overly aggressive rollouts of large patches the break
> only single platforms due to platform specific code that is difficult for
> anyone outside of that platform to fix)
> --Oliver
>
> - James
>
>>
>> - WBR, Alexey Proskuryakov
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>