[Webkit-unassigned] [Bug 54855] Let NRWT print a detailed summary of expected results

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=54855





--- Comment #3 from Dirk Pranke <dpranke at chromium.org>  2011-02-21 18:46:24 PST ---
(From update of attachment 83136)
View in context: https://bugs.webkit.org/attachment.cgi?id=83136&action=review

> Tools/Scripts/webkitpy/layout_tests/layout_package/test_runner.py:897
> +                details[expectation]['now' if test in now else 'wontfix'] += 1

Nit: We don't often use the ternary operator in our code, so you might be better off rewriting this as an if loop. But it's kind of a judgement call. I hate to ban a perfectly good python construct for now real reason.

Technically, the patch looks fine.

However, I wonder how useful this output will really be to anyone but you. I've increasingly found myself thinking that new-run-webkit-tests has too many knobs for displaying statistics different ways, and we log too many statistics and too much stuff in our verbose output. One could easily already argue that no one but me (and maybe one or two other people) know what all the existing --print options do, let alone what another added one would do.

So, while not intending to directly derail this patch, I'd like to ask what you're using these numbers for? Is there some other way we could be getting these numbers that wouldn't require additional command line flags, or for this stuff to be computed and logged at runtime on every run?

For example, do you really need this to be printed on every run? Or is this something that you just want to track occasionally, as a report, or something like that. Or is it perhaps better to generate this as a dashboard report across revisions, without actually running NRWT (since this data can be generated statically) (Much like the LTTF dashboard I wrote that you've probably never seen does).

Or is it maybe better to have RWT output a full table of test results in some form for a run, that can then be post-processed into whatever format you want? 

What do others think?

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.