[webkit-dev] Merging Skipped and test_expectations.txt formats WAS: Simplifying syntax in test_expectations.txt (bug 86691)

[Dirk Pranke]

On Thu, May 17, 2012 at 7:27 PM, Ojan Vafai <ojan at chromium.org> wrote:
> On Thu, May 17, 2012 at 4:29 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>>
>>
>> On May 17, 2012, at 1:42 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>
>> On Thu, May 17, 2012 at 1:37 PM, Peter Kasting <pkasting at chromium.org>
>> wrote:
>>>
>>> On Thu, May 17, 2012 at 1:34 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>>>
>>>> 2. Make outcomes optional. If they are left out, then the test is
>>>> skipped (unless the test is marked SLOW, in which case it's expected to
>>>> pass). There is no SKIP modifier.
>>>
>>>
>>> I don't think we should do this.  It seems very subtle.  I'd rather be
>>> explicit.
>>>
>>> I'm OK with the rest of your numbered proposals.
>>
>>
>> I disagree, but I'm fine with punting this to the list of controversial
>> changes that we should discuss separately. FWIW, my main motivation here is
>> that it allows us to unify the Skipped file format with the
>> test_expectations.txt format. But again, we can discuss that separately.
>>
>>
>> Adding SKIP (or whatever) to every line of skipped files is not a big
>> hurdle, I think we could live with that is a transitions tep. I think the
>> bigger hurdle is supporting chaining across multiple directories.
>
>
> That's great. I don't think anyone is opposed to adding chaining and I think
> that's on Dirk short-list of todos.
>
> The only potentially tricky thing here is figuring out what the platform
> modifiers mean for non-Chromium ports, e.g. I imagine Qt will want similar
> modifiers to Chromium (mac, linux, win, debug, release, etc). But I think
> the difficulty here is more in getting the python code right than agreeing
> on what the correct behavior is.
>

Yes, I am implementing cascading expectations files as soon as this
thread dies :).

As far as sharing modifiers goes, I am of two minds. One: switch to
one big list of modifiers that everyone shares, and add support for
"implementation" as a dimension of modifier alongside operating system
version and debug/release. Two: allow for per-implementation specific
sets of modifiers but require all files to share the same sets.
Practically speaking, the latter would mean that if multiple
implementations shared a single file, then that file would only be
allowed to have lines without any modifier, or modifiers that all
implementations implemented.

This is perhaps better illustrated by two use cases. The first: we
should be able to mark a set of tests that will fail across all ports
while we're waiting for new baselines for a test. The second: it would
be nice to put expectaions where everyone that meets a specific
criteria will fail in one place. Obvious examples: WebKit2-specific
bugs (or WebKit1-specific bugs), and DEBUG assertions.

The former can be met by having a top-level expectations file that
contains lines with no modifiers at all. The latter can be met by
having a lines in that same file with DEBUG or WEBKIT2 as long as
everyone who imports that file recognizes those modifiers. The latter
can also be met by having platform/wk2/test_expectations.txt with no
modifiers, which is okay but not great. Creating a separate
platform/debug/test_expectations.txt (or something) feel more awkward,
and this clearly doesn't scale all that well.

I think there is a point in filing bugs even for WONTFIX features,
just so there is a record of the rationale for why it's WONTFIX and
that rationale doesn't clutter up the expectations file.