[webkit-dev] A not-quite baked proposal for moving pixel baselines out of trunk

[Adam Barth]

Leandro's recent message reminded me that there was some discussion on
this list about the burden caused by storing and maintaining pixel
baselines for an ever-growing list of configurations.  I've been
working on a proposal for moving pixel baselines to a "branch" so that
they don't bother most folks.

Unfortunately, I haven't been able to work out all the details yet.
Here's a somewhat rough, work-in-progress proposal.  If you like this
approach, I can spend more time trying to solve the remaining
problems.

== Motivation ==

Maintaining pixel test results in svn.webkit.org for the various
Chromium configuration imposes a number of costs on the WebKit
community:

All members of the community need to store these results locally when
they create a checkout of WebKit.
Updating the pixel results creates churn in the project that increases
the time required to update a working copy and makes it more difficult
to understand what is actually changing in the project.

As the Chromium port grows in complexity, there is pressure from the
Chromium project to expand the number of test configurations,
increasing these costs on the WebKit community.

== Proposal (Work-in-progress) ==

One option is to stop running pixel test altogether, but that
decreases test coverage and costs correctness.  Another possibility is
to store the pixel results in a location where they are largely
invisible to the rest of the WebKit community.  For example, we could
store the pixel results for chromium-linux the following location:

http://svn.webkit.org/repository/webkit/PixelResults/chromium-linux

Chromium contributors could then use gclient and DEPS to map these
results into their working copies and non-chromium contributors could
safely ignore any changes in the PixelResults “branch”.

FIXME: What about non-platform specific results?

== Problems ==

The main reason this proposal is half-baked is I haven't quite been
able to work out how folks can do some common operations:

1) Land patch with expected image failures (+ revert)
2) Land patch with new image baselines (+ revert)
3) Land new baselines

Because the pixel results and trunk are in the same SVN repo, we
should be able to do these things atomically, but dealing with the
sparse checkout is kind of tricky.

Another big question mark is how this would work for contributors who
use git.  When git mirrors an SVN repro, it only mirrors "trunk", so
the PixelResults directory wouldn't exist in the git version of the
repository.

Thoughts?
Adam