[Webkit-unassigned] [Bug 185783] test262/Runner.pm: add unit tests

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=185783

--- Comment #36 from Daniel Bates <dbates at webkit.org> ---
(In reply to Leo Balter from comment #35)
> As Valerie mentioned, I'm taking over this patch as she's in a vacation time.
> 
> > It is still beneficial to add Buildbot unit tests to catch parsing regressions in how we "surface those errors" in the Buildbot UI.
> 
> The unit tests for the buildbot contains only tests for how the buildbot
> will interpret the results, isolating from anything else, including the
> results from the test262-runner. 
> It contains a static provided expectation text and asserts how the buildbot
> script read it.
> 
> This is for me an important reason to maintain tests for the runner that
> asserts the output will be compatible with the buildbot, 

I do not think tests are the appropriate way to enforce this compatibility. End-to-end buildbot tests would, but this may not be feasible. You will need an English explanation somewhere in either the test runner or unit tests (or both) to explain that the formatting of the output must never change without changing buildbot parsing code. Otherwise, the tests themselves only assert the output was compatible with the output buildbot expected as of the some date. Lack of such English could lead to someone changing the output format AND the tests only to cause Buildbot breakage once a WebKit.org admin pushes a change to Builbot.

If we are still interested in pursuing tests for the output then can we at least mock out the calls to JSC so that we do not need to have a dependency on it?

> > If the former, then I suggest we make more use of webkitdirs.pm as it will simplify the code and make it more consistent with our other Perl code. 
> 
> I sent a patch for this already and it got rejected. We also considered it
> would be a unwelcome burden having to tell everyone to keep consistency for
> it with such an old version of Perl as it is not directly necessary for this
> script.
> 

Do you have a bug # for this?

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20180529/7f6f32af/attachment.html>