[webkit-dev] Should we close the tree? (was: Re: the new TestExpectations syntax is landing soon)

[Dirk Pranke]

This should be fixed now. I apologize for the problems. As Ojan said,
rolling it out would've been tricky because the changes to the
expectation syntax depended on this change. In retrospect it might've
been better to have more time elapse between the initial change and
the changes to the syntax but I didn't want to impose multiple changes
to the format on people for an extended period of time.

Some might wonder why this got through ... we had unit tests for both
the python side and the html side of this, but not an integration test
that ensured that the right thing went all the way through (it's not
actually clear how to test this without being overly invasive on your
checkout, but I'm going to keep thinking about it). Unfortunately,
this also wasn't caught in a review and the tree happened to be fairly
green yesterday so I missed it in garden-o-matic and on the bots
(which don't really use results.html anyway) :(.

-- Dirk

On Thu, Sep 20, 2012 at 11:14 AM, Ojan Vafai <ojan at chromium.org> wrote:
> Fixing the problem will likely take less time at this point than rolling the
> patch out as there are a number of dependent patches. The fix should be in
> soon.
>
>
> On Thu, Sep 20, 2012 at 11:10 AM, Geoffrey Garen <ggaren at apple.com> wrote:
>>
>> I'd prefer to see the patch rolled out.
>>
>> Geoff
>>
>> On Sep 20, 2012, at 11:07 AM, Alexey Proskuryakov <ap at webkit.org> wrote:
>>
>> >
>> > Now tracked as <https://bugs.webkit.org/show_bug.cgi?id=97182>.
>> >
>> > I think that we should close the tree if resolving this takes any
>> > significant time. Not being able to see how exactly tests are failing on
>> > other platforms is unacceptable.
>> >
>> > - WBR, Alexey Proskuryakov
>> >
>> >
>> > 20.09.2012, в 3:54, Osztrogonac Csaba <oszi at inf.u-szeged.hu> написал(а):
>> >
>> >> Unfortunately r129047 broke the results.html, see
>> >> https://bugs.webkit.org/show_bug.cgi?id=96845#c9 for details.
>> >>
>> >> Dirk Pranke írta:
>> >>> These changes are now starting to land ...
>> >>> as of r129047, TEXT, IMAGE+TEXT, and AUDIO are no longer legal
>> >>> keywords in the TestExpectations syntax ... you should use "FAIL"
>> >>> instead.
>> >>> I will be landing the support for the new syntax as quickly as I can
>> >>> to minimize the transition period. Apologies for the inconvenience.
>> >>> -- Dirk
>> >>> On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke <dpranke at chromium.org>
>> >>> wrote:
>> >>>> Hi all,
>> >>>>
>> >>>> The new format of the much-debated TestExpectations syntax will be
>> >>>> landing soon (hopefully in the next couple days).
>> >>>>
>> >>>> For those of who have forgotten / repressed the earlier debates, the
>> >>>> new syntax looks something like:
>> >>>>
>> >>>> webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [
>> >>>> ImageOnlyFailure ]
>> >>>>
>> >>>> Andis documented in full at
>> >>>>
>> >>>> https://trac.webkit.org/wiki/TestExpectations#NewSyntaxNotquiteyetlanded
>> >>>> .
>> >>>>
>> >>>> ( The "[" and "]" characters are delimiters, not EBNF "optional"
>> >>>> markers, although those sections are in fact optional :) ).
>> >>>>
>> >>>> Note that the new syntax means that Skipped files are a syntactic
>> >>>> subset of TestExpectations files, and I plan to convert all of the
>> >>>> Skipped files to TestExpectations files via copy and paste shortly
>> >>>> after the new syntax is landed, and then drop support for Skipped
>> >>>> files (I will update ORWT to use the new files and treat any entry as
>> >>>> a Skip).
>> >>>>
>> >>>> The plan for landing these changes is:
>> >>>>
>> >>>> 1) Add support for parsing the new lines and converting them back
>> >>>> into
>> >>>> the old format (internally) so that both syntaxes are supported
>> >>>> 2) Convert all the existing files over
>> >>>> 3) Make sure things aren't broken :)
>> >>>> 4) Drop support for the old syntax
>> >>>>
>> >>>> I plan for this to all happen quickly, in less than a day. This means
>> >>>> that if you have patches posted that modify those files they may
>> >>>> become stale and need to be updated.
>> >>>>
>> >>>> Changes from the old syntax:
>> >>>>
>> >>>> 1. We use URLs (a specific whitelisted set; let me know if you want
>> >>>> to
>> >>>> add to it) instead of BUGWK12345 etc.
>> >>>> 2. We use "bug(dpranke) instead of BUGDPRANKE
>> >>>> 3. We use CamelCase instead of SHOUTING
>> >>>> 4. We use Failure to represent what used to be TEXT, IMAGE+TEXT, and
>> >>>> AUDIO - these failures will be indistinguishable in the new world,
>> >>>> meaning that you can't distinguish between "text only" and "both
>> >>>> image
>> >>>> and text". Since only Chromium runs pixel tests by default, this
>> >>>> shouldn't be a big deal.
>> >>>> 5. We use ImageOnlyFailure to represent what used to be IMAGE
>> >>>> 6. We use "[" and "]" for delimiters instead of ":" and "="
>> >>>> 7. We use "#" instead of "//" as a comment
>> >>>> 7. WontFix will now imply Skip, i.e., tests marked WontFix will
>> >>>> automatically be Skipped
>> >>>> 8. WontFix and Skip will not require (or even allow) any other
>> >>>> expectations, i.e., you can't say "[ WontFix Crash ]". If you want to
>> >>>> indicate that the test will crash if you actually run it, use a
>> >>>> comment.
>> >>>> 9. WontFix, Skip, Slow, and Rebaseline all move from the left hand
>> >>>> side to the right. The only keywords on the left restrict which
>> >>>> configurations the lines apply to.
>> >>>>
>> >>>> I will send out follow-up emails as this stuff lands. Please let me
>> >>>> know if you have any questions. Thanks!
>> >>>>
>> >>>> -- Dirk
>> >>> _______________________________________________
>> >>> webkit-dev mailing list
>> >>> webkit-dev at lists.webkit.org
>> >>> http://lists.webkit.org/mailman/listinfo/webkit-dev
>> >>
>> >> _______________________________________________
>> >> webkit-dev mailing list
>> >> webkit-dev at lists.webkit.org
>> >> http://lists.webkit.org/mailman/listinfo/webkit-dev
>> >
>> >
>> >
>> >
>> > - WBR, Alexey Proskuryakov
>> >
>> >
>> > _______________________________________________
>> > webkit-dev mailing list
>> > webkit-dev at lists.webkit.org
>> > http://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>