[Webkit-unassigned] [Bug 73603] nrwt shouldn't blow up when there are errors in test_expectations.txt

Thu Dec 1 17:00:30 PST 2011

https://bugs.webkit.org/show_bug.cgi?id=73603

--- Comment #3 from Ryosuke Niwa <rniwa at webkit.org>  2011-12-01 17:00:30 PST ---
(In reply to comment #1)
> NRWT fails by design; having errors like duplicate expectations is bad and shouldn't be ignored; in other words, if there is an error in the expectations file, it should fail the build step. But I don't think you're suggesting the errors should be ignored completely (i.e., what would happen if we just turned them into warnings).

No, I'm not suggesting to ignore the failure. I'm just suggesting to make nrwt not stop running tests when encountering errors.

> I think one problem now is that when this does fail, the reason for the failure isn't obvious on the waterfall ... you get a ??? rather than "bad expectations" or something. It may partially because (IIRC) duplicate expectations actually cause a stack trace rather than an orderly exit.

We can just ignore the line and assume it's PASS unless otherwise specified elsewhere.

> Another possibility would be to add a new --lint-test-files builds step that turned red separately from the actual test run. Then the test run would be free to treat the parsing failures as warnings and proceed on.

nrwt can still fail by returning -1 at the end. It's not great to lose the entire test coverage just for one typo in text_expectations.txt.

(In reply to comment #2)
> We should just make it harder to check in failures.  webkit-patch land and webkit-patch land-cowboy should both check to see if test_expectations.txt has changed, if it has, run --lint-test-files.

I don't think that's a good idea (I'll be opposed to such an idea). I don't want webkit-patch land get slowed down by such checks.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.