[webkit-dev] test_expectations.txt for non-chromium ports

[Ojan Vafai]

On Mon, Feb 13, 2012 at 3:09 PM, Maciej Stachowiak <mjs at apple.com> wrote:

> On Feb 13, 2012, at 1:40 PM, Ojan Vafai wrote:
>
> On Mon, Feb 13, 2012 at 1:06 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
>> On Feb 10, 2012, at 4:20 PM, Dirk Pranke wrote:
>>
>>  >
>> > For example, we might want to use only Skipped files for tests that
>> > are always planned to be skipped, and test_expectations for things
>> > that are supposed to be temporary workarounds to keep the tree green
>> > until bugs can be filed and new baselines generated. Or, we might want
>> > to do something else ...
>>
>> I'd much prefer a single file, but perhaps with different status and uses
>> of states than the current test_expectations.txt format.
>>
>
> I think agreeing on the list of states and whether to have a single file
> or cascading files are the two controversial points. While (if?) we figure
> this out, it seems better to me if we do the simple work of only allowing
> either of a test_expectations.txt file or Skipped lists. Until someone does
> the work of unifying the two systems, it will continue to be a source of
> confusion to allow both for a given port.
>
>
> This seems like mainly an issue for ports that have both files present. If
> only one is actually used, then this makes it a low-stakes change. If
> there's any port actually applying both, then this may be a nontrivial
> change, and may not be worth doing until we figure out our longer-term plan.
>
>
>
>
>> I do agree that distinguishing "test not applicable to this port" from
>> "this test is temporarily failing for unknown reasons" is a good thing to
>> do. It is unfortunate that we don't make the distinction very well right
>> now.
>>
>
> test_expectations.txt has a WONTFIX modifier for this purpose. Chromium
> used to have two files test_expectations.txt and
> test_expectations_wontfix.txt instead of having the modifier. I would kind
> of like us to move back to that model because then test_expectations.txt is
> a file that you hope to keep completely empty
> and test_expectations_wontfix.txt is a file that your rarely touch.
>
>
> It's good that there is a state for this, but WONTFIX doesn't seem like a
> great name to me, at least for tests that are inapplicable due to missing
> features. It implies both that the missing feature is by definition a bug,
> and also that the decision will not be reconsidered. Granted, this may be
> bikeshedding, but if I were, say, disabling tests for Apple's port that
> relate to the new WebSockets protocol because we don't have it on yet, I
> would be very reluctant to mark them WONTFIX.
>
> For tests that are inapplicable for reasons other than missing features,
> it may be simply that there is more than one acceptable behavior, in which
> case WONTFIX seems wrong as well.
>

The intention of WONTFIX is exactly that the decision likely won't need to
be reconsidered (e.g. because the test is platform specific). For the other
purposes (e.g. websockets), we use SKIP. I actually believe we should
rename WONTFIX to NEVERFIX to make it even more clear.

Ojan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120213/5ed21134/attachment-0001.html>