[Webkit-unassigned] [Bug 62425] new-run-webkit-tests: results.html don't list the same list of failed tests as bots do

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=62425





--- Comment #2 from Dirk Pranke <dpranke at chromium.org>  2011-06-09 22:28:47 PST ---
Hm. I may not have really looked at the results.html format since Ojan redid everything.

I agree the output is highly confusing. It looks like the results.html file is showing tests that crash, have MISSING expectations, etc. even if those are expected? Moreover, it does not actually tell you that any of those things are expected, nor does it distinguish those sections from the "unexpected results" section, which is empty and hence not displaying.

Frankly, I was surprised that we were generating a results.html at all when the test run is passing (i.e., there are no unexpected results), since that's not what ORWT does nor what NRWT used to do.

Ojan, are these all intended changes? Can we fix things so that it's at least clearer that the stuff that is being displayed is expected (but admittedly bad)?

I would actually argue that the *only* things that should be displayed by default are the unexpected results, and you should have to toggle the "expected crashes" into view, just like you have to toggle all of the other expected things.

Also, there may be another issue causing confusion. I'll mention it here for continuity but it probably should be a separate bug.

If I look at the chromium.webkit canary bot and find the most recent failing Linux build (which I was doing for comparison), I went from here:

http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Linux

to:

http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Linux/builds/8949

which claimed that there was 1 unexpected failure and 1 unexpectedly flaky test (which matches the stdio output). However, if I click on the layout_test_results archive link, it takes me to:

http://build.chromium.org/f/chromium/layout_test_results/Webkit_Linux/88633/

and downloading that .zip file gives me:

Desktop $ shasum layout-test-results.zip 
e86ad69aff7b2dc53e3a5cf07a4eaff287d5700e  layout-test-results.zip

assuming I'm not getting some weird caching issue from Chrome, that zip file does not show the failures listed in the stdio log at all; is it possible that the wrong layout-test-results.zip file is getting archived?

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.