[Webkit-unassigned] [Bug 113223] New: REGRESSION(r146657): Test failures reported incorrectly

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=113223

           Summary: REGRESSION(r146657): Test failures reported
                    incorrectly
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Keywords: NRWT
          Severity: Normal
          Priority: P2
         Component: New Bugs
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: ossy at webkit.org
                CC: dpranke at chromium.org, rniwa at webkit.org,
                    zarvai at inf.u-szeged.hu, kadam at inf.u-szeged.hu
            Blocks: 112898


After http://trac.webkit.org/changeset/146657 simple test 
failures reported incorrectly by NRWT and buildbot.

Here is a simple example: css3/filters/effect-reference.html 
was broken by http://trac.webkit.org/changeset/146779.
(It seems it needs only rebaseline.)

But the problem is the following NRWT output:
...
09:52:43.778 10859 [4003/28958] css3/filters/effect-reference.html failed unexpectedly (text diff)
09:52:43.778 10859 worker/0 css3/filters/effect-reference.html failed:
09:52:43.778 10859 worker/0  text diff
...
10:11:28.026 10859 Retrying 6 unexpected failure(s) ...
...
10:20:17.156 10859 [4/6] css3/filters/effect-reference.html failed unexpectedly (text diff, image diff)
10:20:17.156 10859 worker/0 css3/filters/effect-reference.html failed:
10:20:17.157 10859 worker/0  text diff
10:20:17.157 10859 worker/0  image diff
...
10:20:27.659 10859 Testing completed, Exit status: 1 <-------- correct exit code
...
Unexpected flakiness: text-only failures (4)
  css3/filters/effect-reference.html [ Failure Pass Failure ]   <------ This line is absolutely wrong!
...


and the reported result by the bot: (It is wrong, because its input is wrong)
9 new passes 6 flakes 7 missing results 

See http://build.webkit.org/builders/Qt%20Linux%20Release/builds/58743 for details

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.