[webkit-dev] generic test expectations?

[Dirk Pranke]

We don't currently support port-specific reftests (or at least, not
very well), and we certainly should be trying to minimize where they
occur. Perhaps we should discuss why you need them (in a separate
thread with a separate subject line)? It sounds like this largely has
to do with what features are enabled and/or supported? Perhaps said
tests should just be skipped in the meantime?

At any rate, we encourage people these days to check in expected
failures rather than suppressing them using the TestExpectations
files. I am hoping this week to finally get back to working on the
"-failing" feature so you can at least distinguish expectations that
are known to be failing/incorrect more easily.

-- Dirk

On Tue, Nov 13, 2012 at 11:02 AM, Glenn Adams <glenn at skynav.com> wrote:
> That would seem to work only if the test(s) fail the same way on all ports.
> In the case that I'm working from, I'm using reftests, where I know the
> correct expectations, but the actual behavior will (does) differ on
> different ports (when the corresponding feature is committed).
>
> I would like to be able to (independently) commit new reftests *and* their
> known good expectation counterparts (that should apply on all ports), and
> then subsequently commit an independent patch that implements the expected
> behavior (on some but not all ports), and the comment further follow-on
> patches that fix the failing ports (possibly by writing new expectations for
> those specific ports).
>
> On Tue, Nov 13, 2012 at 11:52 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>>
>> It is customary to add a failing test expectation (i.e. *-expected.txt
>> file that contains the said failure) in such cases.
>>
>> On Tue, Nov 13, 2012 at 10:49 AM, Glenn Adams <glenn at skynav.com> wrote:
>>>
>>> Just checking, but I don't see a way to add test expectations that apply
>>> generically (to all ports).
>>>
>>> It would be nice to have something like
>>> LayoutTests/platform/generic/TestExpectations to which one could add new
>>> tests that are known to fail everywhere (e.g., because the code that
>>> implements a feature that is tested by those tests is not yet committed),
>>> but which will (at some point in the near future) not fail (when the code
>>> that is to be tested is committed).
>>>
>>> At present, it seems that if one wishes to do this, then it is necessary
>>> to add entries to the each base port expectations (i.e., chromium, mac, win,
>>> etc), which is rather annoying.
>>>
>>> If there is no objection to adding such a "generic" platform expectations
>>> file, then I will undertake to do so.
>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>