[Webkit-unassigned] [Bug 48629] Need additional test expectations for the chromium multi load of layout tests

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=48629





--- Comment #7 from Dirk Pranke <dpranke at chromium.org>  2010-11-08 13:50:52 PST ---
(In reply to comment #6)
> It would be good to have dpranke review this patch since he knows this code best.
> 
> I'm slightly worried that our test expectation complexity is growing geometrically.  Is there a plan to remove some of these files?  Who's going to maintain them?  Is there a bot that goes red when they're wrong?

Søren, it's probably easiest if you copy me on these patches first. 

There's three sets of comments here.

First, to Adam's concerns, yes, I'm concerned about them as well. I hope to modify the test_expectations file syntax to support generic modifiers to help address this. That doesn't directly address the combinatorial explosion of test configurations, however. 

Second, to the main thrust of the patch, would it be acceptable to just mark these as flaky tests in the main file, rather than creating a separate file with non-flaky results? 

E.g., instead of having 

BUG_ML : fast/images/animated-svg-as-image.html = IMAGE

in the ml_expectations.txt file, you could just mark:

BUG_ML : fast/images/animated-svg-as-image.html = IMAGE PASS

in the main file. This isn't ideal, since like all flaky test expectations, it will conflate actual flakiness with tests that fail consistently and tests that pass consistently. Also, there's the danger that you'll mark it flaky because of the ML flag, but someone else might look at the file and confuse it as a non-flaky test.

If this was an acceptable short-term workaround until I can get a more general solution in place that'll help you, the GPU tests, and whatever else we come up with, I'd prefer to do that first (that should happen this week or next).

But, if that isn't acceptable, I am open to putting a patch like this in place instead as a temporary workaround until I can fix it properly. Comments on the patch in the next comment.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.