[webkit-dev] the new TestExpectations syntax is landing soon

[Dirk Pranke]

Hi all,

The new format of the much-debated TestExpectations syntax will be
landing soon (hopefully in the next couple days).

For those of who have forgotten / repressed the earlier debates, the
new syntax looks something like:

webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [ ImageOnlyFailure ]

Andis documented in full at
https://trac.webkit.org/wiki/TestExpectations#NewSyntaxNotquiteyetlanded
.

( The "[" and "]" characters are delimiters, not EBNF "optional"
markers, although those sections are in fact optional :) ).

Note that the new syntax means that Skipped files are a syntactic
subset of TestExpectations files, and I plan to convert all of the
Skipped files to TestExpectations files via copy and paste shortly
after the new syntax is landed, and then drop support for Skipped
files (I will update ORWT to use the new files and treat any entry as
a Skip).

The plan for landing these changes is:

1) Add support for parsing the new lines and converting them back into
the old format (internally) so that both syntaxes are supported
2) Convert all the existing files over
3) Make sure things aren't broken :)
4) Drop support for the old syntax

I plan for this to all happen quickly, in less than a day. This means
that if you have patches posted that modify those files they may
become stale and need to be updated.

Changes from the old syntax:

1. We use URLs (a specific whitelisted set; let me know if you want to
add to it) instead of BUGWK12345 etc.
2. We use "bug(dpranke) instead of BUGDPRANKE
3. We use CamelCase instead of SHOUTING
4. We use Failure to represent what used to be TEXT, IMAGE+TEXT, and
AUDIO - these failures will be indistinguishable in the new world,
meaning that you can't distinguish between "text only" and "both image
and text". Since only Chromium runs pixel tests by default, this
shouldn't be a big deal.
5. We use ImageOnlyFailure to represent what used to be IMAGE
6. We use "[" and "]" for delimiters instead of ":" and "="
7. We use "#" instead of "//" as a comment
7. WontFix will now imply Skip, i.e., tests marked WontFix will
automatically be Skipped
8. WontFix and Skip will not require (or even allow) any other
expectations, i.e., you can't say "[ WontFix Crash ]". If you want to
indicate that the test will crash if you actually run it, use a
comment.
9. WontFix, Skip, Slow, and Rebaseline all move from the left hand
side to the right. The only keywords on the left restrict which
configurations the lines apply to.

I will send out follow-up emails as this stuff lands. Please let me
know if you have any questions. Thanks!

-- Dirk