[webkit-dev] Best practices for landing new/changed layout test expectations?

[Ryosuke Niwa]

On Mon, Feb 25, 2013 at 12:52 PM, Dirk Pranke <dpranke at google.com> wrote:

> On Mon, Feb 25, 2013 at 12:38 PM, Eric Seidel <eseidel at google.com> wrote:
> > I've noticed as of late several different approaches being used when
> > adding/changing LayoutTests which need rebaselining on other platforms.
> >
> > Obviously we cannot expect developers to test/rebaseline on all platforms
> > before landing given our current infrastructure.
> >
> > But what should we expect them to do?
> >
> > Currently some folks add failing expectations to other ports
> > TestExpectations.  Some add [skip].  Some even add them to the (new)
> global
> > TestExpectations file.
> >
> > What's the proper course of action?
> >
> > I checked:
> > http://trac.webkit.org/wiki/Keeping%20the%20Tree%20Green
> >
> http://trac.webkit.org/wiki/Creating%20and%20Submitting%20Layout%20Tests%20and%20Patches
> > http://trac.webkit.org/wiki/TestExpectations
> > and didn't see the "I expect this to fail/need rebaseline on other ports"
> > case discussed.
> >
>
> Memory says that the last time this was raised [1], the consensus was
> to land the change, turn the tree red, notify as many gardeners / port
> maintainers as possible, and rebaseline as quickly as possible. I.e.,
> don't add entries to TestExpectations. Of course, such a policy
> wouldn't play nicely w/ the EWS bots, and I think this discussion
> predated everyone switching over to TestExpectations (and certainly
> predated the generic expectations file).
>
> I don't find the above entirely satisfactory, but I also don't have
> any great alternatives to endorse given the existing tooling.
>
> Also, if the test fails generically (all ports), it probably shouldn't
> be landed. I'm not sure why you couldn't at least update the port
> you're developing on; if you don't, how do you know your fix is
> working at all? (Of course, there could be a case where a
> high-priority fix might break some low-priority tests).
>

Adding a new feature behind a build flag disabled everywhere by default.

> I remember some discussion of a [rebaseline] keyword in TestExpectations,
> > but I'm not sure that ever made it in?
>
> The [ NeedsRebaseline ] enhancement is, as of yet, unimplemented and
> unclaimed [2]. It shouldn't be too hard for someone to try it if
> they're looking for a reason to explore the NRWT code :)
>

I object to adding such a thing. People add and forget about these entries
way too often:

# Needs rebaseline
fast/sub-pixel/inline-block-with-padding.html [ Failure ]
# Rebaseline required after https://bugs.webkit.org/show_bug.cgi?id=101177
webkit.org/b/101177svg/dynamic-updates/SVGUseElement-dom-requiredFeatures.html
[
ImageOnlyFailure Pass ]
webkit.org/b/101177 svg/repaint/inner-svg-change-viewBox.svg [
ImageOnlyFailure Pass ]

# These tests need to be rebaselined after
https://bugs.webkit.org/show_bug.cgi?id=99984
webkit.org/b/99984 svg/as-image/image-preserveAspectRatio-all.svg [
ImageOnlyFailure Pass ]
webkit.org/b/99984 svg/filters/feImage-preserveAspectRatio-all.svg [
ImageOnlyFailure Pass ]
webkit.org/b/99984svg/dynamic-updates/SVGFEImageElement-dom-preserveAspectRatio-attr.html
[
ImageOnlyFailure Pass ]
webkit.org/b/99984svg/dynamic-updates/SVGFEImageElement-svgdom-preserveAspectRatio-prop.html
[ ImageOnlyFailure Pass ]

# Needs rebaseline after bug https://bugs.webkit.org/show_bug.cgi?id=104462
webkit.org/b/98687fast/css-generated-content/table-row-group-to-inline.html
[ Failure ]
webkit.org/b/104595 fast/css/empty-generated-content.html [ Failure ]

# Needs a rebaseline after WK108429 lands.
webkit.org/b/108429 svg/custom/text-ctm.svg [ Pass Failure ]

# Transient. Needs rebaseline.
webkit.org/b/103955 fast/repaint/caret-with-transformation.html [ Missing ]

# Needs rebaseline.
webkit.org/b/105574 editing/input/caret-at-the-edge-of-contenteditable.html
[ Failure ]
webkit.org/b/105574editing/input/reveal-caret-of-multiline-contenteditable.html
[ Failure ]
webkit.org/b/105574 editing/input/reveal-caret-of-multiline-input.html [
Failure ]
webkit.org/b/105574fast/spatial-navigation/snav-div-overflow-scrol-hidden.html
[ Failure ]

# Rebaseline required after https://webkit.org/b/31397
webkit.org/b/107567 svg/batik/text/xmlSpace.svg [ Failure ]
webkit.org/b/107567 fast/text/capitalize-empty-generated-string.html [
Failure ]
webkit.org/b/107567 fast/text/whitespace/007.html [ Failure ]
webkit.org/b/107567 fast/text/whitespace/006.html [ Failure ]
webkit.org/b/107567 fast/inline/drawStyledEmptyInlinesWithWS.html [ Failure
]
webkit.org/b/107567 tables/mozilla/bugs/bug1318.html [ Failure ]
webkit.org/b/107567 fast/inline/drawStyledEmptyInlines.html [ Failure ]
webkit.org/b/107567 tables/mozilla/bugs/bug113235-3.html [ Failure ]
webkit.org/b/107567 css2.1/t0505-c16-descendant-01-e.html [ Failure ]
webkit.org/b/107567 tables/mozilla/bugs/bug1188.html [ Failure ]
webkit.org/b/107567 editing/selection/extend-by-sentence-001.html [ Failure
]
webkit.org/b/107567 svg/carto.net/combobox.svg [ Failure ]

# Needs rebaseline after https://bugs.webkit.org/show_bug.cgi?id=70634
webkit.org/b/109507compositing/contents-opaque/contents-opaque-background-color.html
[ Failure
]
webkit.org/b/109507compositing/contents-opaque/contents-opaque-layer-opacity.html
[ Failure ]
webkit.org/b/109507compositing/contents-opaque/contents-opaque-layer-transform.html
[ Failure ]
webkit.org/b/109507 compositing/rtl/rtl-fixed.html [ Failure ]
webkit.org/b/109507 compositing/rtl/rtl-fixed-overflow.html [ Failure ]

# Rebaseline required after https://webkit.org/b/95772
webkit.org/b/109209fast/text/international/bidi-ignored-for-first-child-inline.html
[ Failure ]

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130225/cf9fb993/attachment.html>