[webkit-efl] test_expectations.txt vs Skipped for EFL - Was Re: [webkit-dev] handling failing tests ...

[Dominik Röttsches]

Hi EFL folks,

On 04/10/2012 12:42 AM, Dirk Pranke wrote:
> Recently I've noticed more people making changes and adding test
> failure suppressions to various ports' test_expectations.txt files.
> This is great!
>
> However, I don't think we have an agreement over what the "best
> practices" are here, so I thought I'd list out what I thought they
> were, and others can comment / correct me as necessary:
>
> 1) Don't mix test_expectations.txt files and Skipped files. This is
> really confusing to everyone involved ... your port should use one or
> the other where possible. (*)
>
> (*) I have an outstanding to-do to modify new-run-webkit-tests to a
> better way of tracking expectations to merge the inheritance/cascade
> aspects of Skipped files with the flexibility in types of failures
> that you get from expectations. Eventually we should have a mechanism
> that replaces both, but for now, we don't. See
> https://bugs.webkit.org/show_bug.cgi?id=83508 .

1) On webkit-dev Dirk recommends using only one approach per port. I'd 
like to discuss which approach of the two we're going to use.

EFL port currently has both, test_expectations.txt and Skipped.
Skipped is more frequently updated and works as our reference. But 
recently, I admit I moved some cases to test_expectations.txt after 
rebaselining.

In EFL, test_expectations.txt has ~20 commits, Skipped has ~108. So 
Skipped seems to be more frequently used.

I see two commits from Gyuyoung indicating that he'd like to move 
forward using test_expectations.txt, and there has been a discussion in
https://bugs.webkit.org/show_bug.cgi?id=75940

Which approach should we take?

Personally, I like the expressiveness of test_expectations.txt for 
differentiating between DEBUG and RELEASE and also associating tests 
with bugs very clearly, so I would vote in favour of moving everything 
from Skipped to test_expectations.txt.

2) As a next step, I'd like to mark all currently failing cases on the 
buildbots to be skipped. The failure retries are single-process and are 
taking too much time without any additional value. No doubt that we need 
to analyse and fix them, but the buildbot doesn't need to waste time and 
cycles on them for each build.

Your feedback is welcome.

Regards,

Dominik

-- 
Dominik Röttsches<dominik.rottsches at linux.intel.com>
Linux Software Engineer
Intel Finland Oy - BIC 0357606-4 - PL 281, 00181 Helsinki

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-efl/attachments/20120413/577ab160/attachment.html>