On Wed, Sep 12, 2018 at 8:49 PM youenn fablet <youennf@gmail.com> wrote:

...

Since an experimental feature flag should not remain off for a long time,

...

a per-experimental-feature file listing the tests to which activate the feature could be considered.

That clashes with Ryosuke's point that moving tests shouldn't change behaviour.

I think it is better to discover the issue at the moment tests are moved. When moving/creating such tests in WPT, the potential issue will only arise after the export/import cycle. Chances are that the issue will get unnnoticed.

No, the move here is like moving it for debugging purposes. The test failing to function just because it got moved out of a test directory is bad.

We already have some hard-coded list of paths under which WTR/DRT behave differently, and it bites me every time I write tests in those directories, or write a test outside of that directory expecting the test to work.

I would strongly object to directory/file-based feature enabling/disabling.

The larger problems are:

...

...

- We don't want experimental features to be on in shipping browsers. It might be ok for them to be always on in tests, but then MiniBrowser would be different from WKTR. At the moment they are always on in tests, and use their default value in MiniBrowser (and shipping).

When implementing a new feature, the first thing we usually do is having an experimental flag turned off by default. At some point, we turn on the experimental flag for tests. When ready, the flag is turned on by default.

It is nice to keep the cost of implementing this workflow as low as possible. Having to tweak layout tests does not go in that direction. If this workflow was fully implementable by just updating WebPreferences.yaml, that would be great. That could also allow running mini browser in a mode where the same flags would be turned on as WKTR.

I would share the same concern Dean raised. The fact WTR/DRT behave differently from trunk is bad. We end up accidentally enabling or disabling features on trunk because WTR/DRT don't test what's enabled/disabled on trunk. The cost of that risk, to me, is a lot higher than the marginal cost of having to add a single line directive in each test as one develops a feature.

- R. Niwa