[Webkit-unassigned] [Bug 161003] New: Many of the expected results from imported/w3c are wrong (contain FAIL strings)

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=161003

            Bug ID: 161003
           Summary: Many of the expected results from imported/w3c are
                    wrong (contain FAIL strings)
    Classification: Unclassified
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: clopez at igalia.com
                CC: achristensen at apple.com, bugs-noreply at webkitgtk.org,
                    lforschler at apple.com, youennf at gmail.com
            Blocks: 160446

I have been checking that many (~328) of the expected results from imported/w3c contain FAIL strings. I understand this means that this tests are actually failing.

$ find LayoutTests/imported/w3c -type f -name '*-expected.txt'|xargs grep -l FAIL|wc -l
328

I wonder:

Wouldn't it have been better to mark this tests as failing in the TestExpectations file instead of uploading a -expected.txt file with FAIL results?

On the GTK+ port some of this tests PASS, but the tooling thing they are failing because the expected file is different than the one that it should be.


Check this example:

Running imported/w3c/web-platform-tests/XMLHttpRequest/send-entity-body-empty.htm on the GTK port I get:

Regressions: Unexpected text-only failures (1)
  imported/w3c/web-platform-tests/XMLHttpRequest/send-entity-body-empty.htm [ Failure ]


But then, checking the diff result I see:

--- /home/clopez/webkit/webkit/layout-test-results/imported/w3c/web-platform-tests/XMLHttpRequest/send-entity-body-empty-expected.txt
+++ /home/clopez/webkit/webkit/layout-test-results/imported/w3c/web-platform-tests/XMLHttpRequest/send-entity-body-empty-actual.txt
@@ -1,5 +1,5 @@

 PASS XMLHttpRequest: send("") - empty entity body (POST) 
 PASS XMLHttpRequest: send("") - empty entity body (PUT) 
-FAIL XMLHttpRequest: send("") - empty entity body (HEAD) assert_equals: expected "NO" but got "0"
+PASS XMLHttpRequest: send("") - empty entity body (HEAD) 



Which means that the test is passing on the GTK+ port, but not on the port where the result was generated (Mac?).

I think the right thing to do is to mark this tests as failing on the corresponding TestExpectations, rather than committing as expected results the result of a test that FAILS.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-unassigned/attachments/20160819/dec8b76b/attachment.html>