[webkit-dev] Test Expectations: A Pipe Dream

[Adam Barth]

Some thoughts on the below:

1) An incremental step towards the below is getting the EWS bots to
run the tests in the first place.  That would be useful even if they
didn't know how to add new expectations to attachments.

2) Architecturally, this will require some changes to how the EWS
works.  The EWS is decentralized, which basically means that the
different bots operate independently.  Do implement LaTER, we'd need
the bots to coordinate at the end to decide where in the fallback
chain to add the new expectations.

3) I suspect this system will be difficult to build with the current
state of redness in the tree.  You should think carefully about how
you'd like to handle the situation where not all the tests are passing
when LaTER processes a patch.

In general, I like the approach of being able to land patches that
change the expected results of test on platforms that I don't have
handy on my desk.

Adam


On Fri, Feb 25, 2011 at 3:14 PM, Dimitri Glazkov <dglazkov at chromium.org> wrote:
> (re-sending to webkit-dev, since this idea was just mentioned on #webkit)
>
> If wishes were fishes, checking layout tests expectations into WebKit
> would work like so:
>
> * Authors of layout tests never need to submit test expectations with
> their patches
>
> * All new patches are flagged by a Layout Test Expectations Resolver
> (LaTER from here on).
>
> * LaTER is a set of EWS bots (one for each platform). It takes the new
> patch, applies/builds/runs tests/generates expectations.
>
> * Newly generated expectations are compared against existing
> expectations. A minimal set of changes for each platform is created.
>
> * This set of generated expectations is merged with the changes in the
> original patch, and attached to bug as a new patch, obsoleting the
> original patch. LaTER knows not to chew on this patch again.
>
> * The rest of the process (review/cq/land) is the same as it was before.
>
> Thoughts? Comments? More crazy ideas?
>
> :DG<