[webkit-dev] Trouble with updating Chromium' slayout test expectations

[Drew Wilson]

That's an interesting thread. I had always assumed that either the testers
always synced to the same rev as the builders, or at least that they pulled
the layout test files/expectations from the builders along with the
executables. Not doing this just seems inherently broken, but I understand
why we do it.

BTW, I'm figuring that most of webkit-dev can't access that group - here's
the summary, pulled from the public chromium documentation:

*Breaks Canary bots.* Canary test bots pick the latest WebKit revision
available. Because test bots start later than build bots, test bots tend to
pick later revisions.  i.e. when a build bot picks a revision X, the
corresponding test bot will pick a revision X+delta where delta >= 0.  This
implies that any new tests added or rebaselines done in the interval (X,
X+delta] will result in test failures because the build didn't contain
necessary changes.  To detect failure caused by this, compare gclient
update's stdout on build bots and test bots.

On that note, I suspect this topic is of little interest to webkit-dev at
large, so we may want to move this to chrome-webkit-gardening for future
discussion.

-atw

On Tue, Feb 22, 2011 at 10:43 AM, Mikhail Naganov <mnaganov at chromium.org>wrote:

> Ah, true. The code changes were long ago.
>
> On Tue, Feb 22, 2011 at 21:40, Mihai Parparita <mihaip at chromium.org>
> wrote:
> > But these were rebaselines, which don't depend on any code changes.
> >
> > Mihai
> >
> > On Tue, Feb 22, 2011 at 10:35 AM, Mikhail Naganov <mnaganov at chromium.org>
> wrote:
> >> I have a suspicion that the cause of canaries failures is described
> >> here:
> https://groups.google.com/a/google.com/group/chrome-webkit-gardening/browse_thread/thread/d3b3a4fc647e804d
> >> In short, bots can run tests prematurely, emitting false failures. I
> >> observed this during my gardening, and I think you should check which
> >> revisions of WebKit and Chromium bots used while running tests.
> >>
> >> On Tue, Feb 22, 2011 at 20:16, Drew Wilson <atwilson at google.com> wrote:
> >>> Yeah, it's weird. I rebaselined those tests, but the chromium canaries
> still
> >>> showed them as red. Now, however, the canaries are showing many of them
> >>> passing. I was planning to remove the lines from test_expectations
> today to
> >>> see if they stay green.
> >>> -atw
> >>>
> >>> On Tue, Feb 22, 2011 at 6:46 AM, Mihai Parparita <mihaip at chromium.org>
> >>> wrote:
> >>>>
> >>>> It looks like Drew checked in baselines with
> >>>> http://trac.webkit.org/changeset/79034 (which may be why
> >>>> rebaseline-chromium-webkit-tests isn't doing anything, since it
> >>>> already has "correct" pixel baselines), but given
> >>>> http://trac.webkit.org/changeset/79088 it didn't seem to work. Drew,
> >>>> any ideas why?
> >>>>
> >>>> Mihai
> >>>>
> >>>> On Tue, Feb 22, 2011 at 2:07 AM, Hajime Morita <morrita at google.com>
> wrote:
> >>>> > Hi Chromium WebKit folks,
> >>>> >
> >>>> > I'm looking for a help to retrieve the latest expectation files for
> >>>> > Chromium Mac LayoutTest.
> >>>> > At the weekend there was a change that triggers massive amount of
> >>>> > pixel test failures that requires rebaselining.
> >>>> > (https://bugs.webkit.org/b/54736)
> >>>> > But the buildbot doesn't have the latest result for part of them,
> that
> >>>> > means
> >>>> >
> >>>> >  * rebaseline-chromium-webkit-tests doesn't update pixel results
> >>>> >  * [layout test results] links on buildbot dashboard are broken.
> >>>> >
> >>>> > Then now around 100 tests for Mac Chromium are
> >>>> > left marked as failed (BUGWK54736 lines), waiting for their fresh
> >>>> > expectations.
> >>>> >
> >>>> > Unfortunately I have no idea what's happening
> >>>> > but missing 100 tests will possibly hurt us.
> >>>> > How could I update these expectations?
> >>>> > Any suggestions and/or helps are appreciated.
> >>>> >
> >>>> > Regards.
> >>>> > --
> >>>> > morrita
> >>>> > _______________________________________________
> >>>> > webkit-dev mailing list
> >>>> > webkit-dev at lists.webkit.org
> >>>> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >>>> >
> >>>
> >>>
> >>> _______________________________________________
> >>> webkit-dev mailing list
> >>> webkit-dev at lists.webkit.org
> >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >>>
> >>>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110222/eaab1b1d/attachment.html>