[webkit-efl] Changing the test expectations lookup sequence

[Christophe Dumez - SISA]

Hi,

I have several concerns:
1. What happens to the "efl" TestExpectations?
2. Is there any other port doing this?
3. A lot of tests fail because a feature is disabled at compile time for EFL port and therefore they fail for both efl-wk1 and efl-wk2. With the change you suggest, we will have to add those tests to both efl-wk1 and efl-wk2 TestExpectations, which means duplication.

Why don't we simply ignore the (generic) wk2 TestExpectations and have:
- "efl-wk2 -> efl -> generic"
- "efl-wk1 -> efl -> generic"

I do believe that the "efl" TestExpectation is useful not only for us but also for people from other ports that want to easily skip a test for our port (for e.g. because it requires a rebaseline).

Of course, there should be no duplication between efl-wk2 and efl TestExpectations (or efl-wk1 and efl TestExpectations).

Kr,
Christophe DUMEZ.
________________________________________
From: webkit-efl-bounces at lists.webkit.org [webkit-efl-bounces at lists.webkit.org] on behalf of Thiago Marcos P. Santos [tmpsantos at gmail.com]
Sent: Thursday, March 07, 2013 12:10
To: webkit-efl at lists.webkit.org
Subject: [webkit-efl] Changing the test expectations lookup sequence

Hi EWebkitters,

Currently on the EFL WK2 port, the lookup sequence for test
expectations is "efl-wk2 -> wk2 -> efl -> generic".

The "efl" part of this sequence is shared between EFL WebKit1 and EFL WebKit2.

I'm currently facing some challenges when gardening because of:

1. Currently we have some tests skipped (~90) on the "wk2"
expectations that have different expectations on "efl". It gives you
the impression that the test is running (because is marked as [
Failure ] when in fact is being skipped because "wk2" has precedence.
The flakiness dashboard also gets confused about it and gives a false
positive.

2. Removing or adding a file to the shared expectation requires
testing on both ports, which IMO is a heavy burden.

3. I'm trying to remove from the flaky list some tests that are now
considered stable (no failures on the past 100 runs). Because of the
shared expectations, I will have to analyse the results of one extra
bot, even though I have no intentions of doing gardening for WebKit1.

I would like to suggest changing the lookup sequence to "efl-wk2 ->
wk2 -> generic", so we can make EFL WK2 port completely independent.

Cheers,
_______________________________________________
webkit-efl mailing list
webkit-efl at lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-efl