[Webkit-unassigned] [Bug 96718] mac-future ignores fallback test expectations

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=96718





--- Comment #6 from Stephanie Lewis <slewis at apple.com>  2012-09-13 22:40:36 PST ---
(In reply to comment #5)
> (In reply to comment #4)
> > There were comments in the code that suggested we need to refactor the approach as well.  I'm open to that, but if this is going to be a large refactoring I'd like to get my tests running first.
> > 
> > My understanding was that mac-future was added as a placeholder for platforms still in development.  Running the layout tests against changes to mac os early on helps us find regressions in underlying code.
> > 
> 
> Not exactly. "mac-future" is a placeholder for platforms still in development, but really it's meant to always resolve to platform/mac (as opposed to something like "mac-lion"  which may point to platform/mac one day and platform/mac-lion the next. So, the theory was that platform/mac would always contain the results of whatever was under development and, if the currently shipping version produces different results, it would go in mac-lion or whatever (mac-mountainlion now).

I think our issue is that we now have two "macs".  The one that webkit considers the most recent and the one that we consider the most recent.  I'm confused as to how you think we should differentiate between the two.

> 
> I believe I worked this all out w/ Mark Rowe a while ago. That said, I'm not particularly wedded to this and we can certainly change it to whatever works for you.
> 
> > We use 
> > 
> > --additional-platform-directory=ADDITIONAL_PLATFORM_DIRECTORY
> >                         Additional directory where to look for test baselines
> >                         (will take precendence over platform baselines).
> >                         Specify multiple times to add multiple search path
> >                         entries.
> > --additional-expectations=ADDITIONAL_EXPECTATIONS
> >                         Path to a test_expectations file that will override
> >                         previous expectations. Specify multiple times for
> >                         multiple sets of overrides.
> > 
> > to specify the location of those results and the TestExpectations file.
> > 
> > The bug is that since there is no match for the platform in VERSION_FALLBACK_ORDER the expectations for older platforms don't match configurations.
> 
> I'm having trouble parsing this sentence. Why do you need mac-future to be in VERSION_FALLBACK_ORDER if you're also using --additional-expectations and --additional-platform-directory? Does run-webkit-tests --platform mac-future --additional-platform-directory=X --additional-expectations=Y not work? I tried this quickly before I left the office tonight and --platform mac-future seemed to work ok for me, but I might've missed something.

passing --platform doesn't resolve the TestExpectation issue.

So what I tracked down the problem to is that we create a TestExpectation for each port that we want to use the testexpectations files for.  Those are mac-future and  mac.  mac is what we are interested in here.  TestExpectation has a list of configurations that the mac testexpectation would apply to.  The function to create that list is port._generate_all_test_configurations.  The configurations end up being VERSION_FALLBACK_ORDER x BUILD_TYPES x ARCHITECTURES.  Since the platform we're running is mac-future and the VERSION_FALLBACK_ORDER doesn't contain mac-future TestExpectation decides that the mac testexpectation doesn't apply to our current platform.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.