[Webkit-unassigned] [Bug 185783] test262/Runner.pm: add unit tests

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=185783

--- Comment #32 from valerie <valerie at bocoup.com> ---
(In reply to Daniel Bates from comment #31)
> Comment on attachment 341105 [details]
> Patch
> 
> I am not happy with this patch. I do not understand the need to add tests
> for a runner as it seems excessive to do. With the exception of the webkitpy
> code we (the WebKit OpenSource Project) tends to not write tests for driver
> code (e.g. build-webkit, DumpRenderTree, et cetera). Instead we prefer to
> write tests for the framework/library/module code that the driver calls.
> There is a careful balancing act between adding testing to catch regressions
> and hindering hackability because the tests actually make the program's
> design more rigid. Writing tests for the driver itself (i.e. writing
> end-to-end integration tests) tends to hurt hack-ability more than the
> benefit it provides.

I thought about this when writing these tests, which is why, in the end, they are pretty simple. The important thing (from my perspective) is that we get an error code/failure under the appropriate conditions. If we change the scenario on purpose, then these tests can be updated. Otherwise I hope the test will catch anyone editing the test262-harness script output without realizing the infrastructure that depends on it.

For example: someone might edit the output because they are using the harness as part of their local development practice, but the script is also used in buildbot to surface failures to jsc devs. If those edits break buildbot's ability to surface those errors, I don't want that to be silent.

> Can we factor out more of the driver code into an
> existing Perl module(s) or new modules and then write tests for them instead
> of writing an end-to-end integration test for the driver itself?

The test262-runner script had some strong motivation to be stand alone, and has a bizarre compatibility on 5.8.8 (bizarre as in nothing else in webkit does). 

> If we chose
> to move forward with adding tests for the driver then can we at least mock
> out the calls to the jsc binary and their results so that we can avoid the
> need to use the real jsc binary and hence get into the situation with have
> now of what jsc binary to use (debug/release/system)? Can we make more use
> of webkitdirs.pm?

Interesting idea. I hope my colleague Leo Balter can look into this. He'll take over future patches on this bug. As of this moment, I'm out on vacation for two weeks!

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20180524/f64b16de/attachment.html>