[webkit-dev] Best practices for landing new/changed layout test expectations?

[Osztrogonác Csaba]

Hi,

Ryosuke Niwa írta:

(snip)

> I don't like the idea that now I have to either use this tool or 
> manually read through TestExpectations to find tests that need 
> rebaseline. It's also annoying if we had all contributors add new lines 
> to TestExpectations because those changes will almost always conflict 
> when applying patches. In general, I don't like any solution that 
> requires modifying TestExpectations files a lot. It's unneeded svn churn.
> 
> It should be fairly straight forward to create a tool that analyzes 
> files changed in each commit and deduce which tests' expected results 
> have been changed. The tool can then fetch results from each port' bot 
> for those tests and automatically land them. It can then comment on the 
> bug automatically about these rebaseline commits. There is no need to 
> add & remove entries from TestExpectation files.

I like your idea about the new tool with some little improvement.

In my mind I see an improved EWS which analyzes the uploaded patch,
determine which expected files are touched (assuming that the author
updated the expected files at least on one platform). And then the
EWS can run only these tests on its own platform and upload the result
to bugzilla. It is much more cheaper than run all tests on all patches.
That's why only Chromium and Apple Mac EWS run test ... And to tell the
truth the waiting time is huge sometimes. This run tests on demand only
can be a good compromise for ports don't have 8x8 cores EWS machine.

I'm a little bit sceptic about automatical rebase commits. I prefer
only uploading the new results to the bugzilla. And then the author
and/or the port maintainer/gardener can review if the new results
are correct or not. And then they can integrate the new baselines
with the original patch (with a tool, of course) and land as one
commit. (Or try one more round on EWS if the results are too old.)

But unfortunately sometimes/regularly folks review and commit
faster than style checker bot run. :) In this case we might
need one more tool to check the bots automatically and help
the gardeners work to be able rebase in one patch instead
of separated patches for each port.

br,
Ossy