[webkit-efl] test_expectations.txt vs Skipped for EFL - Was Re: [webkit-dev] handling failing tests ...

[Raphael Kubo da Costa]

Dominik Röttsches <dominik.rottsches at linux.intel.com> writes:

> 1) On webkit-dev Dirk recommends using only one approach per port. I'd
> like to discuss which approach of the two we're going to use.
>
> EFL port currently has both, test_expectations.txt and Skipped.
> Skipped is more frequently updated and works as our reference. But
> recently, I admit I moved some cases to test_expectations.txt after
> rebaselining.
>
> In EFL, test_expectations.txt has ~20 commits, Skipped has ~108. So
> Skipped seems to be more frequently used.
>
> I see two commits from Gyuyoung indicating that he'd like to move
> forward using test_expectations.txt, and there has been a discussion
> in
> https://bugs.webkit.org/show_bug.cgi?id=75940
>
> Which approach should we take?
>
> Personally, I like the expressiveness of test_expectations.txt for
> differentiating between DEBUG and RELEASE and also associating tests
> with bugs very clearly, so I would vote in favour of moving everything
> from Skipped to test_expectations.txt.

IIRC new-run-webkit-tests and the test_expectations.txt infrastructure
were still under development when we wrote EFL's DumpRenderTree code,
which is why we used a Skipped file at all.

I don't have problems with moving everything to test_expectations.txt,
provided the comments (such as "the following tests are disabled because
the features are still under development" etc) are kept and things are
properly categorized.

If my memory doesn't fail me, ideally in the future old-run-webkit-tests
will be deprecated and everyone should move to test_expectations.txt
anyway.

> 2) As a next step, I'd like to mark all currently failing cases on the
> buildbots to be skipped. The failure retries are single-process and
> are taking too much time without any additional value. No doubt that
> we need to analyse and fix them, but the buildbot doesn't need to
> waste time and cycles on them for each build.

Are you sure there isn't an option to rerun the failing entries in
parallel? Perhaps it makes sense to add it if it doesn't exist.

Personally, I don't know the best approach to take here, just like there
wasn't a clear consensus on webkit-dev about whether to land failing
results or not.