[webkit-dev] test_expectations.txt for non-chromium ports

[Maciej Stachowiak]

On Feb 13, 2012, at 1:40 PM, Ojan Vafai wrote:

> On Mon, Feb 13, 2012 at 1:06 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> On Feb 10, 2012, at 4:20 PM, Dirk Pranke wrote:
> 
> >
> > For example, we might want to use only Skipped files for tests that
> > are always planned to be skipped, and test_expectations for things
> > that are supposed to be temporary workarounds to keep the tree green
> > until bugs can be filed and new baselines generated. Or, we might want
> > to do something else ...
> 
> I'd much prefer a single file, but perhaps with different status and uses of states than the current test_expectations.txt format.
> 
> I think agreeing on the list of states and whether to have a single file or cascading files are the two controversial points. While (if?) we figure this out, it seems better to me if we do the simple work of only allowing either of a test_expectations.txt file or Skipped lists. Until someone does the work of unifying the two systems, it will continue to be a source of confusion to allow both for a given port.

This seems like mainly an issue for ports that have both files present. If only one is actually used, then this makes it a low-stakes change. If there's any port actually applying both, then this may be a nontrivial change, and may not be worth doing until we figure out our longer-term plan.


>  
> I do agree that distinguishing "test not applicable to this port" from "this test is temporarily failing for unknown reasons" is a good thing to do. It is unfortunate that we don't make the distinction very well right now.
> 
> test_expectations.txt has a WONTFIX modifier for this purpose. Chromium used to have two files test_expectations.txt and test_expectations_wontfix.txt instead of having the modifier. I would kind of like us to move back to that model because then test_expectations.txt is a file that you hope to keep completely empty and test_expectations_wontfix.txt is a file that your rarely touch.

It's good that there is a state for this, but WONTFIX doesn't seem like a great name to me, at least for tests that are inapplicable due to missing features. It implies both that the missing feature is by definition a bug, and also that the decision will not be reconsidered. Granted, this may be bikeshedding, but if I were, say, disabling tests for Apple's port that relate to the new WebSockets protocol because we don't have it on yet, I would be very reluctant to mark them WONTFIX.

For tests that are inapplicable for reasons other than missing features, it may be simply that there is more than one acceptable behavior, in which case WONTFIX seems wrong as well.

Regards,
Maciej




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120213/1bcd4c94/attachment-0001.html>