[webkit-dev] the new TestExpectations syntax is landing soon

Dirk Pranke dpranke at chromium.org
Wed Sep 12 16:30:48 PDT 2012

Also, for anyone wondering, the "check in failing test baselines"
feature will be landing shortly after this sticks. I'll send out
another note with details on that when we get there.

-- Dirk

On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> Hi all,
> The new format of the much-debated TestExpectations syntax will be
> landing soon (hopefully in the next couple days).
> For those of who have forgotten / repressed the earlier debates, the
> new syntax looks something like:
> webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [ ImageOnlyFailure ]
> Andis documented in full at
> https://trac.webkit.org/wiki/TestExpectations#NewSyntaxNotquiteyetlanded
> .
> ( The "[" and "]" characters are delimiters, not EBNF "optional"
> markers, although those sections are in fact optional :) ).
> Note that the new syntax means that Skipped files are a syntactic
> subset of TestExpectations files, and I plan to convert all of the
> Skipped files to TestExpectations files via copy and paste shortly
> after the new syntax is landed, and then drop support for Skipped
> files (I will update ORWT to use the new files and treat any entry as
> a Skip).
> The plan for landing these changes is:
> 1) Add support for parsing the new lines and converting them back into
> the old format (internally) so that both syntaxes are supported
> 2) Convert all the existing files over
> 3) Make sure things aren't broken :)
> 4) Drop support for the old syntax
> I plan for this to all happen quickly, in less than a day. This means
> that if you have patches posted that modify those files they may
> become stale and need to be updated.
> Changes from the old syntax:
> 1. We use URLs (a specific whitelisted set; let me know if you want to
> add to it) instead of BUGWK12345 etc.
> 2. We use "bug(dpranke) instead of BUGDPRANKE
> 3. We use CamelCase instead of SHOUTING
> 4. We use Failure to represent what used to be TEXT, IMAGE+TEXT, and
> AUDIO - these failures will be indistinguishable in the new world,
> meaning that you can't distinguish between "text only" and "both image
> and text". Since only Chromium runs pixel tests by default, this
> shouldn't be a big deal.
> 5. We use ImageOnlyFailure to represent what used to be IMAGE
> 6. We use "[" and "]" for delimiters instead of ":" and "="
> 7. We use "#" instead of "//" as a comment
> 7. WontFix will now imply Skip, i.e., tests marked WontFix will
> automatically be Skipped
> 8. WontFix and Skip will not require (or even allow) any other
> expectations, i.e., you can't say "[ WontFix Crash ]". If you want to
> indicate that the test will crash if you actually run it, use a
> comment.
> 9. WontFix, Skip, Slow, and Rebaseline all move from the left hand
> side to the right. The only keywords on the left restrict which
> configurations the lines apply to.
> I will send out follow-up emails as this stuff lands. Please let me
> know if you have any questions. Thanks!
> -- Dirk

More information about the webkit-dev mailing list