[Webkit-unassigned] [Bug 96718] mac-future ignores fallback test expectations

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=96718


Dirk Pranke <dpranke at chromium.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mrowe at apple.com




--- Comment #5 from Dirk Pranke <dpranke at chromium.org>  2012-09-13 21:24:52 PST ---
(In reply to comment #4)
> There were comments in the code that suggested we need to refactor the approach as well.  I'm open to that, but if this is going to be a large refactoring I'd like to get my tests running first.
> 
> My understanding was that mac-future was added as a placeholder for platforms still in development.  Running the layout tests against changes to mac os early on helps us find regressions in underlying code.
> 

Not exactly. "mac-future" is a placeholder for platforms still in development, but really it's meant to always resolve to platform/mac (as opposed to something like "mac-lion"  which may point to platform/mac one day and platform/mac-lion the next. So, the theory was that platform/mac would always contain the results of whatever was under development and, if the currently shipping version produces different results, it would go in mac-lion or whatever (mac-mountainlion now).

I believe I worked this all out w/ Mark Rowe a while ago. That said, I'm not particularly wedded to this and we can certainly change it to whatever works for you.

> We use 
> 
> --additional-platform-directory=ADDITIONAL_PLATFORM_DIRECTORY
>                         Additional directory where to look for test baselines
>                         (will take precendence over platform baselines).
>                         Specify multiple times to add multiple search path
>                         entries.
> --additional-expectations=ADDITIONAL_EXPECTATIONS
>                         Path to a test_expectations file that will override
>                         previous expectations. Specify multiple times for
>                         multiple sets of overrides.
> 
> to specify the location of those results and the TestExpectations file.
> 
> The bug is that since there is no match for the platform in VERSION_FALLBACK_ORDER the expectations for older platforms don't match configurations.

I'm having trouble parsing this sentence. Why do you need mac-future to be in VERSION_FALLBACK_ORDER if you're also using --additional-expectations and --additional-platform-directory? Does run-webkit-tests --platform mac-future --additional-platform-directory=X --additional-expectations=Y not work? I tried this quickly before I left the office tonight and --platform mac-future seemed to work ok for me, but I might've missed something.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.