[webkit-dev] Best practices for landing new/changed layout test expectations?

[Philip Rogers]

The steps described here differ from what I've been doing. In the interest
of closure and port happiness, I've updated the wiki to reflect what is
being proposed:
https://trac.webkit.org/wiki/CreatingLayoutTests

Please feel free to update it further.

Philip


On Tue, Feb 26, 2013 at 2:34 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> On Tue, Feb 26, 2013 at 1:03 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> > On Tue, Feb 26, 2013 at 12:47 PM, Dirk Pranke <dpranke at chromium.org>
> wrote:
> >>
> >> On Tue, Feb 26, 2013 at 2:11 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> >> > On Tue, Feb 26, 2013 at 1:55 AM, Tom Hudson <tomhudson at google.com>
> >> > wrote:
> >> >>
> >> >> On Mon, Feb 25, 2013 at 10:34 PM, Ryosuke Niwa <rniwa at webkit.org>
> >> >> wrote:
> >> >>>
> >> >>> It should be fairly straight forward to create a tool that analyzes
> >> >>> files
> >> >>> changed in each commit and deduce which tests' expected results have
> >> >>> been
> >> >>> changed. The tool can then fetch results from each port' bot for
> those
> >> >>> tests
> >> >>> and automatically land them. It can then comment on the bug
> >> >>> automatically
> >> >>> about these rebaseline commits. There is no need to add & remove
> >> >>> entries
> >> >>> from TestExpectation files.
> >>
> >> Wait, what?
> >>
> >> For some reason neither I nor the mailing list archives got your
> >> initial message, nor  Silvia or Tom's responses, nor your responses
> >> (at least as of the time of me writing this), so I feel like I've
> >> missed a radical shift in this thread, and maybe I missed some of the
> >> context.
> >
> >
> > https://lists.webkit.org/pipermail/webkit-dev/2013-February/023967.html
> >
>
> This link doesn't point to any of those messages, but perhaps it's not
> that important.
>
> >> You're proposing that we automatically land updated baselines without
> >> review and then somehow update bugs, have people go back and look at
> >> the updated bugs to see if the baseline changes represent actual
> >> regressions or just expected changes?
> >
> >
> > Right. Given that the commit already contains information as to which
> tests
> > have been rebaselined, a script should be able to fetch new baselines for
> > those affected tests on each platform and land them or upload as patches
> as
> > needed.
> >
>
> It's possible that we could fetch and cluster new baselines based on
> what changed in the initial commit. I would be concerned that there
> could be a fair amount of noise in either direction (tests that
> changed on the initial platform didn't on others, and others did), and
> you'd also have to figure out how to cluster changes since most builds
> on the bots contain multiple changes. But, you could probably use some
> of garden-o-matic's results to help here.
>
> That said, I'm not sure this workflow would actually improve things
> much over garden-o-matic.
>
> I am quite a bit more reluctant to automatically land any such
> changes; it seems like it would be hard if not impossible to tell
> (programmatically) whether a baseline changed as expected or if it
> represented a regression.
>
> If we were to work on new tooling, I would be much more in favor of
> pushing this up to an EWS-time step like Ossy suggests.
>
> -- Dirk
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130313/8f2cd4f0/attachment.html>