[webkit-efl] test_expectations.txt vs Skipped for EFL - Was Re: [webkit-dev] handling failing tests ...
Dominik Röttsches
dominik.rottsches at linux.intel.com
Fri Apr 13 06:46:56 PDT 2012
Hi EFL folks,
On 04/10/2012 12:42 AM, Dirk Pranke wrote:
> Recently I've noticed more people making changes and adding test
> failure suppressions to various ports' test_expectations.txt files.
> This is great!
>
> However, I don't think we have an agreement over what the "best
> practices" are here, so I thought I'd list out what I thought they
> were, and others can comment / correct me as necessary:
>
> 1) Don't mix test_expectations.txt files and Skipped files. This is
> really confusing to everyone involved ... your port should use one or
> the other where possible. (*)
>
> (*) I have an outstanding to-do to modify new-run-webkit-tests to a
> better way of tracking expectations to merge the inheritance/cascade
> aspects of Skipped files with the flexibility in types of failures
> that you get from expectations. Eventually we should have a mechanism
> that replaces both, but for now, we don't. See
> https://bugs.webkit.org/show_bug.cgi?id=83508 .
1) On webkit-dev Dirk recommends using only one approach per port. I'd
like to discuss which approach of the two we're going to use.
EFL port currently has both, test_expectations.txt and Skipped.
Skipped is more frequently updated and works as our reference. But
recently, I admit I moved some cases to test_expectations.txt after
rebaselining.
In EFL, test_expectations.txt has ~20 commits, Skipped has ~108. So
Skipped seems to be more frequently used.
I see two commits from Gyuyoung indicating that he'd like to move
forward using test_expectations.txt, and there has been a discussion in
https://bugs.webkit.org/show_bug.cgi?id=75940
Which approach should we take?
Personally, I like the expressiveness of test_expectations.txt for
differentiating between DEBUG and RELEASE and also associating tests
with bugs very clearly, so I would vote in favour of moving everything
from Skipped to test_expectations.txt.
2) As a next step, I'd like to mark all currently failing cases on the
buildbots to be skipped. The failure retries are single-process and are
taking too much time without any additional value. No doubt that we need
to analyse and fix them, but the buildbot doesn't need to waste time and
cycles on them for each build.
Your feedback is welcome.
Regards,
Dominik
--
Dominik Röttsches<dominik.rottsches at linux.intel.com>
Linux Software Engineer
Intel Finland Oy - BIC 0357606-4 - PL 281, 00181 Helsinki
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-efl/attachments/20120413/577ab160/attachment.html>
More information about the webkit-efl
mailing list