[webkit-dev] generic test expectations?

[Dirk Pranke]

On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams <glenn at skynav.com> wrote:
>
> On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> We don't currently support port-specific reftests (or at least, not
>> very well), and we certainly should be trying to minimize where they
>> occur.
>
>
> Hmm, I actually used port specific reftest expectation files in a recent
> patch [1] (since rolled out), and it appeared to work (as a way to
> effectively rebaseline those expectations). So something seems to be
> working.
>
> [1] http://trac.webkit.org/changeset/133529
>

I expect it'll sort of work, but it won't be robust; you may hit weird
behavior and/or bugs. We really haven't beaten on this aspect of
things, and I don't know yet how much we want to.

>>
>> At any rate, we encourage people these days to check in expected
>> failures rather than suppressing them using the TestExpectations
>> files.
>
>
> The problem is essentially a chicken and egg problem. I don't know what the
> per-port failures will be ahead of time, but I do know the set of "correct"
> expectations. Since I am (independently) unable to build/test all ports run
> by build bots, I would like to commit the set of tests plus known good
> expectations as a preliminary step (with a generic skip all tests for all
> ports), and then subsequently commit the feature itself, and then
> subsequently override the generic skip on a port specific basis, effectively
> re-enabling the tests on a port by port basis as I refine the feature patch
> (as needed) to handle port specific behavioral differences.
>

I think this is a reasonable approach. I would be interested to hear
if others had alternatives they preferred.

-- Dirk