[Webkit-unassigned] [Bug 105636] New: run-webkit-tests summary seems incomplete

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=105636

           Summary: run-webkit-tests summary seems incomplete
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: ojan at chromium.org
                CC: eric at webkit.org, dpranke at chromium.org,
                    dbates at webkit.org


>From http://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux/builds/3240/steps/webkit_tests/logs/stdio: 

=> Results: 33706/34160 tests passed (98.7%)

=> Tests to be fixed (894):
      6 crashes                  ( 0.7%)
     27 timeouts                 ( 3.0%)
      4 missing results          ( 0.4%)
     98 image-only failures      (11.0%)

=> Tests that will only be fixed if they crash (WONTFIX) (6753):


Expected to fail, but passed: (7)
  fast/forms/button-generated-content.html
  fast/forms/button-inner-block-reuse.html
  fast/text/hyphenate-character.html
  fast/text/hyphenate-first-word.html
  fast/text/hyphenate-limit-lines.html
  fast/text/hyphenate-locale.html
  fast/text/hyphens.html


The numbers under "Tests to be fixed" don't add up to 100%. Are we leaving out TEXT/IMAGE+TEXT failures? Do we leave out flaky passes there?

Also, 33706 + 894 != 34160. So, it's not clear to me what those numbers are supposed to be. Maybe this is due to tests listed as flaky that actually pass?

Some other nits I may as well mention as I look at this:
1) The colon after the WONTFIX line is confusing since nothing is printed after it.
2) The text for WONTFIX is no longer accurate since we skip these tests now. We should just say something like "Tests that will *never* be fixed".
3) I find the "Tests to be fixed" wording confusing. I'd prefer something like "Expected failures". At this point, everyone needs to know the concept of expected vs unexpected failures.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.