[webkit-dev] Best practices for landing new/changed layout test expectations?

[Ryosuke Niwa]

On Mon, Feb 25, 2013 at 2:21 PM, Silvia Pfeiffer <silviapf at chromium.org>wrote:

> On Tue, Feb 26, 2013 at 8:22 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>
>
>> I object to adding such a thing. People add and forget about these
>> entries way too often:
>>
>> # Needs rebaseline
>> fast/sub-pixel/inline-block-with-padding.html [ Failure ]
>> # Rebaseline required after
>> https://bugs.webkit.org/show_bug.cgi?id=101177
>> webkit.org/b/101177svg/dynamic-updates/SVGUseElement-dom-requiredFeatures.html [
>> ImageOnlyFailure Pass ]
>> webkit.org/b/101177 svg/repaint/inner-svg-change-viewBox.svg [
>> ImageOnlyFailure Pass ]
>>
>> # These tests need to be rebaselined after
>> https://bugs.webkit.org/show_bug.cgi?id=99984
>> webkit.org/b/99984 svg/as-image/image-preserveAspectRatio-all.svg [
>> ImageOnlyFailure Pass ]
>> webkit.org/b/99984 svg/filters/feImage-preserveAspectRatio-all.svg [
>> ImageOnlyFailure Pass ]
>> webkit.org/b/99984svg/dynamic-updates/SVGFEImageElement-dom-preserveAspectRatio-attr.html [
>> ImageOnlyFailure Pass ]
>> webkit.org/b/99984svg/dynamic-updates/SVGFEImageElement-svgdom-preserveAspectRatio-prop.html
>> [ ImageOnlyFailure Pass ]
>>
>> # Needs rebaseline after bug
>> https://bugs.webkit.org/show_bug.cgi?id=104462
>> webkit.org/b/98687fast/css-generated-content/table-row-group-to-inline.html [ Failure ]
>> webkit.org/b/104595 fast/css/empty-generated-content.html [ Failure ]
>>
>> # Needs a rebaseline after WK108429 lands.
>> webkit.org/b/108429 svg/custom/text-ctm.svg [ Pass Failure ]
>>
>> # Transient. Needs rebaseline.
>> webkit.org/b/103955 fast/repaint/caret-with-transformation.html [
>> Missing ]
>>
>> # Needs rebaseline.
>> webkit.org/b/105574editing/input/caret-at-the-edge-of-contenteditable.html [ Failure ]
>> webkit.org/b/105574editing/input/reveal-caret-of-multiline-contenteditable.html [ Failure ]
>> webkit.org/b/105574 editing/input/reveal-caret-of-multiline-input.html [
>> Failure ]
>> webkit.org/b/105574fast/spatial-navigation/snav-div-overflow-scrol-hidden.html [ Failure ]
>>
>> # Rebaseline required after https://webkit.org/b/31397
>> webkit.org/b/107567 svg/batik/text/xmlSpace.svg [ Failure ]
>> webkit.org/b/107567 fast/text/capitalize-empty-generated-string.html [
>> Failure ]
>> webkit.org/b/107567 fast/text/whitespace/007.html [ Failure ]
>> webkit.org/b/107567 fast/text/whitespace/006.html [ Failure ]
>> webkit.org/b/107567 fast/inline/drawStyledEmptyInlinesWithWS.html [
>> Failure ]
>> webkit.org/b/107567 tables/mozilla/bugs/bug1318.html [ Failure ]
>> webkit.org/b/107567 fast/inline/drawStyledEmptyInlines.html [ Failure ]
>> webkit.org/b/107567 tables/mozilla/bugs/bug113235-3.html [ Failure ]
>> webkit.org/b/107567 css2.1/t0505-c16-descendant-01-e.html [ Failure ]
>> webkit.org/b/107567 tables/mozilla/bugs/bug1188.html [ Failure ]
>> webkit.org/b/107567 editing/selection/extend-by-sentence-001.html [
>> Failure ]
>> webkit.org/b/107567 svg/carto.net/combobox.svg [ Failure ]
>>
>> # Needs rebaseline after https://bugs.webkit.org/show_bug.cgi?id=70634
>> webkit.org/b/109507compositing/contents-opaque/contents-opaque-background-color.html [ Failure
>> ]
>> webkit.org/b/109507compositing/contents-opaque/contents-opaque-layer-opacity.html [ Failure ]
>> webkit.org/b/109507compositing/contents-opaque/contents-opaque-layer-transform.html [ Failure ]
>> webkit.org/b/109507 compositing/rtl/rtl-fixed.html [ Failure ]
>> webkit.org/b/109507 compositing/rtl/rtl-fixed-overflow.html [ Failure ]
>>
>> # Rebaseline required after https://webkit.org/b/95772
>> webkit.org/b/109209fast/text/international/bidi-ignored-for-first-child-inline.html [ Failure ]
>>
>
> I think it's great to be able to create such a list! The bug at
> https://bugs.webkit.org/show_bug.cgi?id=100415 points to the idea of
> extending the garden-o-matic tool to create rebaselines based on the
> keyword. I like this idea - it would help everyone, including
> non-committers, deal with updating testexpectations and keep the tree green
> longer. I don't believe in making bots go read - that's a productivity
> stopper.
>

I don't like the idea that now I have to either use this tool or manually
read through TestExpectations to find tests that need rebaseline. It's also
annoying if we had all contributors add new lines to TestExpectations
because those changes will almost always conflict when applying patches. In
general, I don't like any solution that requires modifying TestExpectations
files a lot. It's unneeded svn churn.

It should be fairly straight forward to create a tool that analyzes files
changed in each commit and deduce which tests' expected results have been
changed. The tool can then fetch results from each port' bot for those
tests and automatically land them. It can then comment on the bug
automatically about these rebaseline commits. There is no need to add &
remove entries from TestExpectation files.

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130225/978bb28a/attachment.html>