[webkit-dev] Best practices for landing new/changed layout test expectations?

[Eric Seidel]

Thanks for following up Philip.  I don't feel like this thread met
much consensus, more just went off into the weeds.


Given the history of that page, I'm not sure it truly reflects the
consensus of the project:
https://trac.webkit.org/wiki/CreatingLayoutTests?action=history

> Regardless of whether you are making a platform independent change or dependent change, it is your responsibility to ensure that the change does not negatively affect any other ports / platforms. This includes updating platform specific results or contacting a port maintainer to do this for you (for contacts, see: http://trac.webkit.org/wiki/WebKit%20Team).

I'm not sure I see anyone following this these days.  The EWS system
and sherriffiing cultures seem to have largely replaced
bot-monitoring.  We also don't really have a system of "core ports",
thus "all ports" seems a bit broad here.

> Once your change is landed, there are some steps required to verify the change. Note: These steps apply for all changes. If you are making a platform dependent change, you should expect that additional results will be required for each platform but the actions are the same.

I'm not sure it makes any sense to ask contributors to update other
ports.  In many cases, it's not possible for contributors to even
build other ports.  We clearly need some sort of automated system for
this, but right now I believe the expectation is that maintainers of
the various ports will add the port-specific results, and that those
contributing new layout tests should just land their results as the
common ones?  But I could be mistaken.


I think the reality is that the project doesn't really have consensus
here, likely meaning this is a good topic for the contributors
meeting.

On Wed, Mar 13, 2013 at 3:12 PM, Philip Rogers <pdr at google.com> wrote:
> The steps described here differ from what I've been doing. In the interest
> of closure and port happiness, I've updated the wiki to reflect what is
> being proposed:
> https://trac.webkit.org/wiki/CreatingLayoutTests
>
> Please feel free to update it further.
>
> Philip
>
>
> On Tue, Feb 26, 2013 at 2:34 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> On Tue, Feb 26, 2013 at 1:03 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>> > On Tue, Feb 26, 2013 at 12:47 PM, Dirk Pranke <dpranke at chromium.org>
>> > wrote:
>> >>
>> >> On Tue, Feb 26, 2013 at 2:11 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>> >> > On Tue, Feb 26, 2013 at 1:55 AM, Tom Hudson <tomhudson at google.com>
>> >> > wrote:
>> >> >>
>> >> >> On Mon, Feb 25, 2013 at 10:34 PM, Ryosuke Niwa <rniwa at webkit.org>
>> >> >> wrote:
>> >> >>>
>> >> >>> It should be fairly straight forward to create a tool that analyzes
>> >> >>> files
>> >> >>> changed in each commit and deduce which tests' expected results
>> >> >>> have
>> >> >>> been
>> >> >>> changed. The tool can then fetch results from each port' bot for
>> >> >>> those
>> >> >>> tests
>> >> >>> and automatically land them. It can then comment on the bug
>> >> >>> automatically
>> >> >>> about these rebaseline commits. There is no need to add & remove
>> >> >>> entries
>> >> >>> from TestExpectation files.
>> >>
>> >> Wait, what?
>> >>
>> >> For some reason neither I nor the mailing list archives got your
>> >> initial message, nor  Silvia or Tom's responses, nor your responses
>> >> (at least as of the time of me writing this), so I feel like I've
>> >> missed a radical shift in this thread, and maybe I missed some of the
>> >> context.
>> >
>> >
>> > https://lists.webkit.org/pipermail/webkit-dev/2013-February/023967.html
>> >
>>
>> This link doesn't point to any of those messages, but perhaps it's not
>> that important.
>>
>> >> You're proposing that we automatically land updated baselines without
>> >> review and then somehow update bugs, have people go back and look at
>> >> the updated bugs to see if the baseline changes represent actual
>> >> regressions or just expected changes?
>> >
>> >
>> > Right. Given that the commit already contains information as to which
>> > tests
>> > have been rebaselined, a script should be able to fetch new baselines
>> > for
>> > those affected tests on each platform and land them or upload as patches
>> > as
>> > needed.
>> >
>>
>> It's possible that we could fetch and cluster new baselines based on
>> what changed in the initial commit. I would be concerned that there
>> could be a fair amount of noise in either direction (tests that
>> changed on the initial platform didn't on others, and others did), and
>> you'd also have to figure out how to cluster changes since most builds
>> on the bots contain multiple changes. But, you could probably use some
>> of garden-o-matic's results to help here.
>>
>> That said, I'm not sure this workflow would actually improve things
>> much over garden-o-matic.
>>
>> I am quite a bit more reluctant to automatically land any such
>> changes; it seems like it would be hard if not impossible to tell
>> (programmatically) whether a baseline changed as expected or if it
>> represented a regression.
>>
>> If we were to work on new tooling, I would be much more in favor of
>> pushing this up to an EWS-time step like Ossy suggests.
>>
>> -- Dirk
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>