[Webkit-unassigned] [Bug 63104] New: new-run-webkit-tests: TEXT failures are getting reported as MISSING

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=63104

           Summary: new-run-webkit-tests: TEXT failures are getting
                    reported as MISSING
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Text
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: dpranke at chromium.org
                CC: eric at webkit.org, aroben at apple.com, tony at chromium.org,
                    abarth at webkit.org, ojan at chromium.org


See bug 63057 for the initial bug report ...

A problem occurs when a text-only test fails before it gets a chance to call layoutTestController.dumpAsText() ... DRT will output an image block, and NRWT will then look for (and not find) an -expected.png file.

Older versions of NRWT used to print out what kind of image test failure was occuring (missing hash, missing text, missing checksum), but we've lumped them all together now into the one bucket. In addition, because the result is MISSING, the results.html file doesn't pull in the (actually existing) -expected.txt, and we can't see in the results that there is a text diff.

I think this is causing unnecessary (and preventable) confusion, because it will cause people to think that the -expected.txt file couldn't be found.

I'm not quite sure what to do about this, though, since there's no way to tell from the test itself whether it's text only or not. Possible options:

1) reserve MISSING for only tests that really have no expected text output, and possibly add a MISSING_IMAGE result type.

2) lump missing image files into an IMAGE failure?

3) modify the results.json file to know what kind of missing results we actually got.

4) ??

As another example, I recently did a test run and got output that included:

Unexpected flakiness: no expected results found (3)
  fast/forms/form-associated-element-crash.html = MISSING PASS
  fast/forms/form-attribute-elements-order.html = MISSING PASS
  fast/forms/form-attribute-elements-order2.html = MISSING PASS

Which, from a normal user's point of view, shouldn't be possible.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.