[webkit-dev] Best practices for landing new/changed layout test expectations?

[Ryosuke Niwa]

On Mon, Feb 25, 2013 at 1:39 PM, Dirk Pranke <dpranke at google.com> wrote:

> On Mon, Feb 25, 2013 at 1:33 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> > On Mon, Feb 25, 2013 at 1:29 PM, Dirk Pranke <dpranke at google.com> wrote:
> >>
> >> On Mon, Feb 25, 2013 at 1:22 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> >> > On Mon, Feb 25, 2013 at 12:52 PM, Dirk Pranke <dpranke at google.com>
> >> > wrote:
> >> >
> >> >> > I remember some discussion of a [rebaseline] keyword in
> >> >> > TestExpectations,
> >> >> > but I'm not sure that ever made it in?
> >> >>
> >> >> The [ NeedsRebaseline ] enhancement is, as of yet, unimplemented and
> >> >> unclaimed [2]. It shouldn't be too hard for someone to try it if
> >> >> they're looking for a reason to explore the NRWT code :)
> >> >
> >> >
> >> > I object to adding such a thing. People add and forget about these
> >> > entries
> >> > way too often:
> >> >
> >>
> >> The rationale for adding the keyword was precisely this ... it would
> >> make it easier to programmatically identify tests needing updated
> >> baselines and alert people :).
> >>
> >> Do you have an alternative you'd suggest for keeping people from
> >> forgetting (particularly in the case where we have an expected failure
> >> waiting on a bug fix)?
> >
> > Just turn bots red. People maintaining bots will notice and rebaseline
> them
> > as needed.
>
> If there's a significant time lapse between failing test and bug fix,
> it seems like a bad idea to leave the tree red.


I don't think that's necessarily bad. If the port maintainer is not
available on IRC/Bugzilla and can't assist you in rebaselining tests, I
don't think you should be accountable for keeping bots green for them in
practice.

On the other hand, if all it takes to keep bots green is to commit new
baselines, then why don't patch authors just do that?

Other than people who are not yet committers, it doesn't make much sense
for patch authors to be adding test expectations when they could simply
commit new baselines after the fact unless you're landing a patch that
requires a rebaseline of mass scale, in which case, adding test
expectations temporarily might make sense.

All too often, people add test expectations and forget about them because
bots are green. It's the issue with incentives. People know they shouldn't
leave bots red so they have a much higher inceptive to fix them on time. If
they have added test expectations, then bots are green and other people
won't be nagging them to fix it so they have much less inceptive to spend
their time fixing bots.

Are you suggesting we should land a "failling" baseline in the meantime?
>

No. I'm suggesting patch authors perform their due diligence and either ask
port maintainers to rebaseline or rebaseline tests themselves.

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130225/4c7b7bdf/attachment.html>