[webkit-dev] Chromium "GPU" LayoutTests

[Adam Barth]

On Mon, Aug 22, 2011 at 3:52 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> On Mon, Aug 22, 2011 at 3:45 PM, Adam Barth <abarth at webkit.org> wrote:
>> On Mon, Aug 22, 2011 at 3:35 PM, James Robinson <jamesr at google.com> wrote:
>>> On Mon, Aug 22, 2011 at 3:24 PM, Adam Barth <abarth at webkit.org> wrote:
>>>> On Mon, Aug 22, 2011 at 3:18 PM, James Robinson <jamesr at google.com> wrote:
>>>> > On Mon, Aug 22, 2011 at 3:07 PM, Adam Barth <abarth at webkit.org> wrote:
>>>> >> On Mon, Aug 22, 2011 at 3:02 PM, James Robinson <jamesr at google.com>
>>>> >> wrote:
>>>> >> > On Mon, Aug 22, 2011 at 2:55 PM, Adam Barth <abarth at webkit.org>
>>>> >> > wrote:
>>>> >> >> I've been trying to wrap my mind around the "GPU" LayoutTests that
>>>> >> >> Chromium runs.  In
>>>> >> >> <http://trac.webkit.org/browser/trunk/LayoutTests/platform>, there
>>>> >> >> are
>>>> >> >> the following directories:
>>>> >> >>
>>>> >> >> chromium-gpu
>>>> >> >> chromium-gpu-cg-mac
>>>> >> >> chromium-gpu-linux
>>>> >> >> chromium-gpu-win
>>>> >> >>
>>>> >> >> These seem to be related to the webkit_gpu_tests step on these bots:
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29
>>>> >> >> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Linux
>>>> >> >> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Vista
>>>> >> >> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Win7
>>>> >> >>
>>>> >> >> This file
>>>> >> >>
>>>> >> >>
>>>> >> >> <http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/builders.py>
>>>> >> >> also lists the following bots:
>>>> >> >>
>>>> >> >> Webkit Win - GPU
>>>> >> >> Webkit Win7 - GPU
>>>> >> >> Webkit Linux - GPU
>>>> >> >> Webkit Linux 32 - GPU
>>>> >> >> Webkit Mac10.5 (CG) - GPU
>>>> >> >> Webkit Mac10.6 (CG) - GPU
>>>> >> >>
>>>> >> >> Questions:
>>>> >> >>
>>>> >> >> 1) Do these "- GPU" bots exist anywhere?  (I can't find them, and a
>>>> >> >> recent bug comment indicates that they might be fictional.)
>>>> >> >
>>>> >> > The " - GPU" bots are the same bots as the normal layout test bots,
>>>> >> > but
>>>> >> > run
>>>> >> > as a separate step.
>>>> >> >
>>>> >> >
>>>> >> >  See http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29/builds/185,
>>>> >> > for example.  The "bot name" for the GPU tests is the normal bot name
>>>> >> > with "
>>>> >> > - GPU" appended.
>>>> >> >
>>>> >> >>
>>>> >> >> 2) Are there any other bots related to the GPU configuration other
>>>> >> >> than those listed above?  (For example, is there any coverage of
>>>> >> >> this
>>>> >> >> configuration on build.webkit.org?)
>>>> >> >
>>>> >> > build.webkit.org seems to not be running these tests, for reasons
>>>> >> > that
>>>> >> > are
>>>> >> > unclear to me, but they run on the other chromium bots that run
>>>> >> > layout
>>>> >> > tests.
>>>> >> >
>>>> >> >> 3) Why is webkit_gpu_tests a separate step from webkit_tests?
>>>> >> >
>>>> >> > Different flags are passed to DumpRenderTree, but it runs some of the
>>>> >> > same
>>>> >> > tests.  This is so we can get coverage for (for instance) the 2d
>>>> >> > canvas
>>>> >> > API
>>>> >> > in our hardware and software paths.
>>>> >>
>>>> >> Could we instead use the layoutTestController to enable or disable
>>>> >> whatever flags at test-time rather than on the command line?  For
>>>> >> example, we could put the guts of the 2d canvas tests into a
>>>> >> JavaScript file and include it in two HTML files, one that sets the
>>>> >> flag and one that does not.  That's how we handle strict vs quirks
>>>> >> mode, for example.
>>>> >
>>>> > There are 802 tests in canvas/philip/tests/ (which is an imported
>>>> > suite),
>>>> > currently all implemented as separate .html files, and another 166 in
>>>> > fast/canvas.  It seems a bit tedious to manually wrap each of these
>>>> > tests in
>>>> > two different wrappers.  That doesn't seem like a very practical option,
>>>> > although it has some nice aspects.
>>>>
>>>> I'd be happy to edit these thousand files if it meant we could remove
>>>> the complexity of the GPU configuration.
>>>
>>> You would have to modify all currently existing tests, and all such tests
>>> that will be added in the future.  I'm not sure how to patch tests that will
>>> only exist in the future.
>>
>> Presumably new tests will just follow the pattern of the existing
>> tests.  Given that we only have 166 of these tests so far, that's not
>> likely to be too much work.
>>
>>>>  The ongoing maintenance cost
>>>> of the GPU configuration strikes me as a lot greater than the one-time
>>>> cost of changing these tests.  (There's also the side-benefit of
>>>> letting all the other ports increase their test coverage by wiring up
>>>> the LayoutTestController method.)
>>>>
>>>> I suspect (although I'd have to look more carefully), that we could
>>>> make this work for the canvas/philip/tests tests without actually
>>>> modifying the files (e.g., using iframes), if we wanted the ability to
>>>> easily import new versions of that test suite.
>>>
>>> That would seem to imply either running all of the relevant twice for all
>>> ports, unless this iframe solution was chromium-specific.  Would that really
>>> be simpler?
>>
>> Other ports could skip some of these tests, if they desire.  We could
>> put them in separate directories to facilitate skipping half of them,
>> for example.
>>
>
> You could do one pass in a generic directory and another pass from a
> platform directory, perhaps, and then you wouldn't need to do the
> skipping (but that's just because they're skipped by default).
>
>>> What concrete costs are you attempting to minimize?  It is difficult to
>>> modify our tools due to this configuration, but the complexity is restricted
>>> nearly fully to chromium-specific scripts.  The tests themselves and other
>>> ports are unaffected.
>>
>> The complexity is not restricted to chromium-specific scripts.  The
>> Chromium GPU configuration causes a moderately large amount of
>> complexity in the common infrastructure shared by the entire project
>> because it's a special case in many places.  That's why it keeps
>> breaking when we make changes to the tooling for the project as a
>> whole.
>>
>
> I'm not aware of any cases where the GPU abstraction is known outside
> of webkitpy/layout_tests/port/* (which is a statement about my
> undoubtedly faulty knowledge, not the truthfulness of your sentence).
> What other tools are becoming aware of it?
>
> At any rate, it was always my intention that this should be abstracted
> and hidden properly. Perhaps if the abstraction is leaking, we need to
> fix the abstraction?

For example, I would need to do extra work to support the GPU
configuration in garden-o-matic under the current scheme but I would
need to do zero work to support it if it were not a special case.

>> If we made the changes above, then we'd get the same test coverage
>> without this added complexity, which looks like a win.
>
> Modifying LayoutTestController to make this a runtime setting,
> modifying the existing tests, and having to modify new tests in the
> future also sound like costs themselves. I'm not yet convinced that
> your suggestion is less work either now or in the long term, although
> it is certainly an interesting idea.

It's a one-time cost that I'm happy to pay to remove this ongoing
source of complexity.  If you assume that the future is large (which
our usual operating hypothesis in WebKit), then almost any ongoing
cost out-weighs a one-time cost.

Adam