On Mon, Feb 25, 2013 at 12:52 PM, Dirk Pranke <span dir="ltr">&lt;<a href="mailto:dpranke@google.com" target="_blank">dpranke@google.com</a>&gt;</span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

On Mon, Feb 25, 2013 at 12:38 PM, Eric Seidel &lt;<a href="mailto:eseidel@google.com">eseidel@google.com</a>&gt; wrote:<br>
&gt; I&#39;ve noticed as of late several different approaches being used when<br>
&gt; adding/changing LayoutTests which need rebaselining on other platforms.<br>
&gt;<br>
&gt; Obviously we cannot expect developers to test/rebaseline on all platforms<br>
&gt; before landing given our current infrastructure.<br>
&gt;<br>
&gt; But what should we expect them to do?<br>
&gt;<br>
&gt; Currently some folks add failing expectations to other ports<br>
&gt; TestExpectations.  Some add [skip].  Some even add them to the (new) global<br>
&gt; TestExpectations file.<br>
&gt;<br>
&gt; What&#39;s the proper course of action?<br>
&gt;<br>
&gt; I checked:<br>
&gt; <a href="http://trac.webkit.org/wiki/Keeping%20the%20Tree%20Green" target="_blank">http://trac.webkit.org/wiki/Keeping%20the%20Tree%20Green</a><br>
&gt; <a href="http://trac.webkit.org/wiki/Creating%20and%20Submitting%20Layout%20Tests%20and%20Patches" target="_blank">http://trac.webkit.org/wiki/Creating%20and%20Submitting%20Layout%20Tests%20and%20Patches</a><br>
&gt; <a href="http://trac.webkit.org/wiki/TestExpectations" target="_blank">http://trac.webkit.org/wiki/TestExpectations</a><br>
&gt; and didn&#39;t see the &quot;I expect this to fail/need rebaseline on other ports&quot;<br>
&gt; case discussed.<br>
&gt;<br>
<br>
Memory says that the last time this was raised [1], the consensus was<br>
to land the change, turn the tree red, notify as many gardeners / port<br>
maintainers as possible, and rebaseline as quickly as possible. I.e.,<br>
don&#39;t add entries to TestExpectations. Of course, such a policy<br>
wouldn&#39;t play nicely w/ the EWS bots, and I think this discussion<br>
predated everyone switching over to TestExpectations (and certainly<br>
predated the generic expectations file).<br>
<br>
I don&#39;t find the above entirely satisfactory, but I also don&#39;t have<br>
any great alternatives to endorse given the existing tooling.<br>
<br>
Also, if the test fails generically (all ports), it probably shouldn&#39;t<br>
be landed. I&#39;m not sure why you couldn&#39;t at least update the port<br>
you&#39;re developing on; if you don&#39;t, how do you know your fix is<br>
working at all? (Of course, there could be a case where a<br>
high-priority fix might break some low-priority tests).<br></blockquote><div><br></div><div>Adding a new feature behind a build flag disabled everywhere by default.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">


&gt; I remember some discussion of a [rebaseline] keyword in TestExpectations,<br>
&gt; but I&#39;m not sure that ever made it in?<br>
<br>
The [ NeedsRebaseline ] enhancement is, as of yet, unimplemented and<br>
unclaimed [2]. It shouldn&#39;t be too hard for someone to try it if<br>
they&#39;re looking for a reason to explore the NRWT code :)<br></blockquote><div><br></div><div>I object to adding such a thing. People add and forget about these entries way too often:</div><div><br></div><div><div># Needs rebaseline</div>

<div>fast/sub-pixel/inline-block-with-padding.html [ Failure ]</div></div><div><div># Rebaseline required after <a href="https://bugs.webkit.org/show_bug.cgi?id=101177">https://bugs.webkit.org/show_bug.cgi?id=101177</a></div>

<div><a href="http://webkit.org/b/101177">webkit.org/b/101177</a> svg/dynamic-updates/SVGUseElement-dom-requiredFeatures.html [ ImageOnlyFailure Pass ]</div><div><a href="http://webkit.org/b/101177">webkit.org/b/101177</a> svg/repaint/inner-svg-change-viewBox.svg [ ImageOnlyFailure Pass ]</div>

</div><div><div><br></div><div># These tests need to be rebaselined after <a href="https://bugs.webkit.org/show_bug.cgi?id=99984">https://bugs.webkit.org/show_bug.cgi?id=99984</a></div><div><a href="http://webkit.org/b/99984">webkit.org/b/99984</a> svg/as-image/image-preserveAspectRatio-all.svg [ ImageOnlyFailure Pass ]</div>

<div><a href="http://webkit.org/b/99984">webkit.org/b/99984</a> svg/filters/feImage-preserveAspectRatio-all.svg [ ImageOnlyFailure Pass ]</div><div><a href="http://webkit.org/b/99984">webkit.org/b/99984</a> svg/dynamic-updates/SVGFEImageElement-dom-preserveAspectRatio-attr.html [ ImageOnlyFailure Pass ]</div>

<div><a href="http://webkit.org/b/99984">webkit.org/b/99984</a> svg/dynamic-updates/SVGFEImageElement-svgdom-preserveAspectRatio-prop.html [ ImageOnlyFailure Pass ]</div><div><br></div><div># Needs rebaseline after bug <a href="https://bugs.webkit.org/show_bug.cgi?id=104462">https://bugs.webkit.org/show_bug.cgi?id=104462</a></div>

<div><a href="http://webkit.org/b/98687">webkit.org/b/98687</a> fast/css-generated-content/table-row-group-to-inline.html [ Failure ]</div><div><a href="http://webkit.org/b/104595">webkit.org/b/104595</a> fast/css/empty-generated-content.html [ Failure ]</div>

<div><br></div><div># Needs a rebaseline after WK108429 lands.</div><div><a href="http://webkit.org/b/108429">webkit.org/b/108429</a> svg/custom/text-ctm.svg [ Pass Failure ]</div></div><div><br></div><div><div># Transient. Needs rebaseline.</div>

<div><a href="http://webkit.org/b/103955">webkit.org/b/103955</a> fast/repaint/caret-with-transformation.html [ Missing ]</div></div><div><br></div><div><div># Needs rebaseline.</div><div><a href="http://webkit.org/b/105574">webkit.org/b/105574</a> editing/input/caret-at-the-edge-of-contenteditable.html [ Failure ]</div>

<div><a href="http://webkit.org/b/105574">webkit.org/b/105574</a> editing/input/reveal-caret-of-multiline-contenteditable.html [ Failure ]</div><div><a href="http://webkit.org/b/105574">webkit.org/b/105574</a> editing/input/reveal-caret-of-multiline-input.html [ Failure ]</div>

<div><a href="http://webkit.org/b/105574">webkit.org/b/105574</a> fast/spatial-navigation/snav-div-overflow-scrol-hidden.html [ Failure ]</div><div><br></div><div># Rebaseline required after <a href="https://webkit.org/b/31397">https://webkit.org/b/31397</a></div>

<div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> svg/batik/text/xmlSpace.svg [ Failure ] </div><div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> fast/text/capitalize-empty-generated-string.html [ Failure ] </div>

<div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> fast/text/whitespace/007.html [ Failure ] </div><div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> fast/text/whitespace/006.html [ Failure ] </div>

<div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> fast/inline/drawStyledEmptyInlinesWithWS.html [ Failure ] </div><div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> tables/mozilla/bugs/bug1318.html [ Failure ] </div>

<div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> fast/inline/drawStyledEmptyInlines.html [ Failure ] </div><div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> tables/mozilla/bugs/bug113235-3.html [ Failure ] </div>

<div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> css2.1/t0505-c16-descendant-01-e.html [ Failure ] </div><div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> tables/mozilla/bugs/bug1188.html [ Failure ] </div>

<div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> editing/selection/extend-by-sentence-001.html [ Failure ] </div><div><a href="http://webkit.org/b/107567">webkit.org/b/107567</a> svg/<a href="http://carto.net/combobox.svg">carto.net/combobox.svg</a> [ Failure ] </div>

</div><div><br></div><div><div># Needs rebaseline after <a href="https://bugs.webkit.org/show_bug.cgi?id=70634">https://bugs.webkit.org/show_bug.cgi?id=70634</a></div><div><a href="http://webkit.org/b/109507">webkit.org/b/109507</a> compositing/contents-opaque/contents-opaque-background-color.html [ Failure ]</div>

<div><a href="http://webkit.org/b/109507">webkit.org/b/109507</a> compositing/contents-opaque/contents-opaque-layer-opacity.html [ Failure ]</div><div><a href="http://webkit.org/b/109507">webkit.org/b/109507</a> compositing/contents-opaque/contents-opaque-layer-transform.html [ Failure ]</div>

<div><a href="http://webkit.org/b/109507">webkit.org/b/109507</a> compositing/rtl/rtl-fixed.html [ Failure ]</div><div><a href="http://webkit.org/b/109507">webkit.org/b/109507</a> compositing/rtl/rtl-fixed-overflow.html [ Failure ]</div>

</div><div><div><br></div><div># Rebaseline required after <a href="https://webkit.org/b/95772">https://webkit.org/b/95772</a></div><div><a href="http://webkit.org/b/109209">webkit.org/b/109209</a> fast/text/international/bidi-ignored-for-first-child-inline.html [ Failure ]</div>

</div><div><br></div><div>- R. Niwa</div><div><br></div></div>