[webkit-dev] Merging Skipped and test_expectations.txt formats WAS: Simplifying syntax in test_expectations.txt (bug 86691)
dpranke at chromium.org
Thu May 17 19:47:11 PDT 2012
On Thu, May 17, 2012 at 7:27 PM, Ojan Vafai <ojan at chromium.org> wrote:
> On Thu, May 17, 2012 at 4:29 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>> On May 17, 2012, at 1:42 PM, Ojan Vafai <ojan at chromium.org> wrote:
>> On Thu, May 17, 2012 at 1:37 PM, Peter Kasting <pkasting at chromium.org>
>>> On Thu, May 17, 2012 at 1:34 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>>> 2. Make outcomes optional. If they are left out, then the test is
>>>> skipped (unless the test is marked SLOW, in which case it's expected to
>>>> pass). There is no SKIP modifier.
>>> I don't think we should do this. It seems very subtle. I'd rather be
>>> I'm OK with the rest of your numbered proposals.
>> I disagree, but I'm fine with punting this to the list of controversial
>> changes that we should discuss separately. FWIW, my main motivation here is
>> that it allows us to unify the Skipped file format with the
>> test_expectations.txt format. But again, we can discuss that separately.
>> Adding SKIP (or whatever) to every line of skipped files is not a big
>> hurdle, I think we could live with that is a transitions tep. I think the
>> bigger hurdle is supporting chaining across multiple directories.
> That's great. I don't think anyone is opposed to adding chaining and I think
> that's on Dirk short-list of todos.
> The only potentially tricky thing here is figuring out what the platform
> modifiers mean for non-Chromium ports, e.g. I imagine Qt will want similar
> modifiers to Chromium (mac, linux, win, debug, release, etc). But I think
> the difficulty here is more in getting the python code right than agreeing
> on what the correct behavior is.
Yes, I am implementing cascading expectations files as soon as this
thread dies :).
As far as sharing modifiers goes, I am of two minds. One: switch to
one big list of modifiers that everyone shares, and add support for
"implementation" as a dimension of modifier alongside operating system
version and debug/release. Two: allow for per-implementation specific
sets of modifiers but require all files to share the same sets.
Practically speaking, the latter would mean that if multiple
implementations shared a single file, then that file would only be
allowed to have lines without any modifier, or modifiers that all
This is perhaps better illustrated by two use cases. The first: we
should be able to mark a set of tests that will fail across all ports
while we're waiting for new baselines for a test. The second: it would
be nice to put expectaions where everyone that meets a specific
criteria will fail in one place. Obvious examples: WebKit2-specific
bugs (or WebKit1-specific bugs), and DEBUG assertions.
The former can be met by having a top-level expectations file that
contains lines with no modifiers at all. The latter can be met by
having a lines in that same file with DEBUG or WEBKIT2 as long as
everyone who imports that file recognizes those modifiers. The latter
can also be met by having platform/wk2/test_expectations.txt with no
modifiers, which is okay but not great. Creating a separate
platform/debug/test_expectations.txt (or something) feel more awkward,
and this clearly doesn't scale all that well.
I think there is a point in filing bugs even for WONTFIX features,
just so there is a record of the rationale for why it's WONTFIX and
that rationale doesn't clutter up the expectations file.
More information about the webkit-dev