[webkit-dev] test_expectations.txt for non-chromium ports

[Maciej Stachowiak]

On Feb 13, 2012, at 3:22 PM, Ojan Vafai wrote:

> On Mon, Feb 13, 2012 at 3:09 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> 
>>  
>> I do agree that distinguishing "test not applicable to this port" from "this test is temporarily failing for unknown reasons" is a good thing to do. It is unfortunate that we don't make the distinction very well right now.
>> 
>> test_expectations.txt has a WONTFIX modifier for this purpose. Chromium used to have two files test_expectations.txt and test_expectations_wontfix.txt instead of having the modifier. I would kind of like us to move back to that model because then test_expectations.txt is a file that you hope to keep completely empty and test_expectations_wontfix.txt is a file that your rarely touch.
> 
> It's good that there is a state for this, but WONTFIX doesn't seem like a great name to me, at least for tests that are inapplicable due to missing features. It implies both that the missing feature is by definition a bug, and also that the decision will not be reconsidered. Granted, this may be bikeshedding, but if I were, say, disabling tests for Apple's port that relate to the new WebSockets protocol because we don't have it on yet, I would be very reluctant to mark them WONTFIX.
> 
> For tests that are inapplicable for reasons other than missing features, it may be simply that there is more than one acceptable behavior, in which case WONTFIX seems wrong as well.
> 
> The intention of WONTFIX is exactly that the decision likely won't need to be reconsidered (e.g. because the test is platform specific). For the other purposes (e.g. websockets), we use SKIP. I actually believe we should rename WONTFIX to NEVERFIX to make it even more clear.

I don't know about other organizations, but from Apple's point of view, it's rare that we'd want to publicly promise that we'll never implement something. We'd just want to document that we haven't implemented the feature yet, and thus some tests are inapplicable. So NEVERFIX would be something we'd be even more reluctant to apply. We would not even want to mark the difference between "we haven't enabled this feature yet, but probably will very soon" and "we have no plans to ever implement the feature unless something changes", as that would be communicating information about future releases.

I don't know of the intent of SKIP, but maybe it is ok for this purpose. I would expect it to be used for tests that are temporarily skipped due to bugs, based on the name, which seems different to me from "this functionality is not implemented in this port, rendering the test inapplicable".

Regards,
Maciej

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120213/fc90aa73/attachment.html>