[Webkit-unassigned] [Bug 90892] results.html should handle flaky tests differently

bugzilla-daemon at webkit.org bugzilla-daemon at webkit.org
Wed Jul 11 15:18:29 PDT 2012


--- Comment #4 from Ojan Vafai <ojan at chromium.org>  2012-07-11 15:18:28 PST ---
(In reply to comment #3)
> (In reply to comment #2)
> > (In reply to comment #1)
> > > I'm sure you realize this already but we don't currently have a way to compare the output from the two runs to see if they are different. I don't know that it would be particularly hard to add that.
> > 
> > full_results.json, which is what results.html uses, has this information and shows it in the UI already. We don't technically know which run was first and which was second and we don't have the -actual.* files for the first run, but we have the type of failure for each run.
> > 
> True. Shouldn't we have the -actuals for the first run as well?

Do we? Where do we store them? I thought the second run overwrote the first one.

> > > Also, I'm a bit concerned that implementing this just makes it even easier to ignore the tests in list 1, that seem to be should either be marked as expected flaky or actually be fixed.
> > 
> > That's true. In practice, I think that this is already ignored. So the cost is that other non-flaky failures get missed.
> > 
> > In fact, upon further thought, I think we should hide list 1 by default. There's just too much noise right now in the results.html output.
> That would be hiding unexpected behavior, which seems kinda bad. If others thought this was a good idea, I'd be willing to give it a shot, though.

I suppose. I see a slew (~12) of flaky passes everytime I run the tests. Maybe I just encounter this more because I usually run with -f and I'm just getting what I ask for.

Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

More information about the webkit-unassigned mailing list