[webkit-efl] Changing the test expectations lookup sequence
Christophe Dumez - SISA
ch.dumez at sisa.samsung.com
Thu Mar 7 03:03:34 PST 2013
I have several concerns:
1. What happens to the "efl" TestExpectations?
2. Is there any other port doing this?
3. A lot of tests fail because a feature is disabled at compile time for EFL port and therefore they fail for both efl-wk1 and efl-wk2. With the change you suggest, we will have to add those tests to both efl-wk1 and efl-wk2 TestExpectations, which means duplication.
Why don't we simply ignore the (generic) wk2 TestExpectations and have:
- "efl-wk2 -> efl -> generic"
- "efl-wk1 -> efl -> generic"
I do believe that the "efl" TestExpectation is useful not only for us but also for people from other ports that want to easily skip a test for our port (for e.g. because it requires a rebaseline).
Of course, there should be no duplication between efl-wk2 and efl TestExpectations (or efl-wk1 and efl TestExpectations).
From: webkit-efl-bounces at lists.webkit.org [webkit-efl-bounces at lists.webkit.org] on behalf of Thiago Marcos P. Santos [tmpsantos at gmail.com]
Sent: Thursday, March 07, 2013 12:10
To: webkit-efl at lists.webkit.org
Subject: [webkit-efl] Changing the test expectations lookup sequence
Currently on the EFL WK2 port, the lookup sequence for test
expectations is "efl-wk2 -> wk2 -> efl -> generic".
The "efl" part of this sequence is shared between EFL WebKit1 and EFL WebKit2.
I'm currently facing some challenges when gardening because of:
1. Currently we have some tests skipped (~90) on the "wk2"
expectations that have different expectations on "efl". It gives you
the impression that the test is running (because is marked as [
Failure ] when in fact is being skipped because "wk2" has precedence.
The flakiness dashboard also gets confused about it and gives a false
2. Removing or adding a file to the shared expectation requires
testing on both ports, which IMO is a heavy burden.
3. I'm trying to remove from the flaky list some tests that are now
considered stable (no failures on the past 100 runs). Because of the
shared expectations, I will have to analyse the results of one extra
bot, even though I have no intentions of doing gardening for WebKit1.
I would like to suggest changing the lookup sequence to "efl-wk2 ->
wk2 -> generic", so we can make EFL WK2 port completely independent.
webkit-efl mailing list
webkit-efl at lists.webkit.org
More information about the webkit-efl