[Webkit-unassigned] [Bug 90892] results.html should handle flaky tests differently

Wed Jul 11 15:11:13 PDT 2012

https://bugs.webkit.org/show_bug.cgi?id=90892

--- Comment #3 from Dirk Pranke <dpranke at chromium.org>  2012-07-11 15:11:13 PST ---
(In reply to comment #2)
> (In reply to comment #1)
> > I'm sure you realize this already but we don't currently have a way to compare the output from the two runs to see if they are different. I don't know that it would be particularly hard to add that.
> 
> full_results.json, which is what results.html uses, has this information and shows it in the UI already. We don't technically know which run was first and which was second and we don't have the -actual.* files for the first run, but we have the type of failure for each run.
> 

True. Shouldn't we have the -actuals for the first run as well?

> > Also, I'm a bit concerned that implementing this just makes it even easier to ignore the tests in list 1, that seem to be should either be marked as expected flaky or actually be fixed.
> 
> That's true. In practice, I think that this is already ignored. So the cost is that other non-flaky failures get missed.
> 
> In fact, upon further thought, I think we should hide list 1 by default. There's just too much noise right now in the results.html output.

That would be hiding unexpected behavior, which seems kinda bad. If others thought this was a good idea, I'd be willing to give it a shot, though.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.