[webkit-dev] the new TestExpectations syntax is landing soon

[Dirk Pranke]

Also, for anyone wondering, the "check in failing test baselines"
feature will be landing shortly after this sticks. I'll send out
another note with details on that when we get there.

-- Dirk

On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> Hi all,
>
> The new format of the much-debated TestExpectations syntax will be
> landing soon (hopefully in the next couple days).
>
> For those of who have forgotten / repressed the earlier debates, the
> new syntax looks something like:
>
> webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [ ImageOnlyFailure ]
>
> Andis documented in full at
> https://trac.webkit.org/wiki/TestExpectations#NewSyntaxNotquiteyetlanded
> .
>
> ( The "[" and "]" characters are delimiters, not EBNF "optional"
> markers, although those sections are in fact optional :) ).
>
> Note that the new syntax means that Skipped files are a syntactic
> subset of TestExpectations files, and I plan to convert all of the
> Skipped files to TestExpectations files via copy and paste shortly
> after the new syntax is landed, and then drop support for Skipped
> files (I will update ORWT to use the new files and treat any entry as
> a Skip).
>
> The plan for landing these changes is:
>
> 1) Add support for parsing the new lines and converting them back into
> the old format (internally) so that both syntaxes are supported
> 2) Convert all the existing files over
> 3) Make sure things aren't broken :)
> 4) Drop support for the old syntax
>
> I plan for this to all happen quickly, in less than a day. This means
> that if you have patches posted that modify those files they may
> become stale and need to be updated.
>
> Changes from the old syntax:
>
> 1. We use URLs (a specific whitelisted set; let me know if you want to
> add to it) instead of BUGWK12345 etc.
> 2. We use "bug(dpranke) instead of BUGDPRANKE
> 3. We use CamelCase instead of SHOUTING
> 4. We use Failure to represent what used to be TEXT, IMAGE+TEXT, and
> AUDIO - these failures will be indistinguishable in the new world,
> meaning that you can't distinguish between "text only" and "both image
> and text". Since only Chromium runs pixel tests by default, this
> shouldn't be a big deal.
> 5. We use ImageOnlyFailure to represent what used to be IMAGE
> 6. We use "[" and "]" for delimiters instead of ":" and "="
> 7. We use "#" instead of "//" as a comment
> 7. WontFix will now imply Skip, i.e., tests marked WontFix will
> automatically be Skipped
> 8. WontFix and Skip will not require (or even allow) any other
> expectations, i.e., you can't say "[ WontFix Crash ]". If you want to
> indicate that the test will crash if you actually run it, use a
> comment.
> 9. WontFix, Skip, Slow, and Rebaseline all move from the left hand
> side to the right. The only keywords on the left restrict which
> configurations the lines apply to.
>
> I will send out follow-up emails as this stuff lands. Please let me
> know if you have any questions. Thanks!
>
> -- Dirk