[webkit-dev] Chromium "GPU" LayoutTests

[Adam Barth]

On Mon, Aug 22, 2011 at 4:41 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> On Mon, Aug 22, 2011 at 3:24 PM, Adam Barth <abarth at webkit.org> wrote:
>> On Mon, Aug 22, 2011 at 3:18 PM, James Robinson <jamesr at google.com> wrote:
>>> On Mon, Aug 22, 2011 at 3:07 PM, Adam Barth <abarth at webkit.org> wrote:
>>>> On Mon, Aug 22, 2011 at 3:02 PM, James Robinson <jamesr at google.com> wrote:
>>>> > On Mon, Aug 22, 2011 at 2:55 PM, Adam Barth <abarth at webkit.org> wrote:
>>>> >> I've been trying to wrap my mind around the "GPU" LayoutTests that
>>>> >> Chromium runs.  In
>>>> >> <http://trac.webkit.org/browser/trunk/LayoutTests/platform>, there are
>>>> >> the following directories:
>>>> >>
>>>> >> chromium-gpu
>>>> >> chromium-gpu-cg-mac
>>>> >> chromium-gpu-linux
>>>> >> chromium-gpu-win
>>>> >>
>>>> >> These seem to be related to the webkit_gpu_tests step on these bots:
>>>> >>
>>>> >>
>>>> >> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29
>>>> >>
>>>> >>
>>>> >> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29
>>>> >> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Linux
>>>> >> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Vista
>>>> >> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Win7
>>>> >>
>>>> >> This file
>>>> >>
>>>> >> <http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/builders.py>
>>>> >> also lists the following bots:
>>>> >>
>>>> >> Webkit Win - GPU
>>>> >> Webkit Win7 - GPU
>>>> >> Webkit Linux - GPU
>>>> >> Webkit Linux 32 - GPU
>>>> >> Webkit Mac10.5 (CG) - GPU
>>>> >> Webkit Mac10.6 (CG) - GPU
>>>> >>
>>>> >> Questions:
>>>> >>
>>>> >> 1) Do these "- GPU" bots exist anywhere?  (I can't find them, and a
>>>> >> recent bug comment indicates that they might be fictional.)
>>>> >
>>>> > The " - GPU" bots are the same bots as the normal layout test bots, but
>>>> > run
>>>> > as a separate step.
>>>> >
>>>> >  See http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29/builds/185,
>>>> > for example.  The "bot name" for the GPU tests is the normal bot name
>>>> > with "
>>>> > - GPU" appended.
>>>> >
>>>> >>
>>>> >> 2) Are there any other bots related to the GPU configuration other
>>>> >> than those listed above?  (For example, is there any coverage of this
>>>> >> configuration on build.webkit.org?)
>>>> >
>>>> > build.webkit.org seems to not be running these tests, for reasons that
>>>> > are
>>>> > unclear to me, but they run on the other chromium bots that run layout
>>>> > tests.
>>>> >
>>>> >> 3) Why is webkit_gpu_tests a separate step from webkit_tests?
>>>> >
>>>> > Different flags are passed to DumpRenderTree, but it runs some of the
>>>> > same
>>>> > tests.  This is so we can get coverage for (for instance) the 2d canvas
>>>> > API
>>>> > in our hardware and software paths.
>>>>
>>>> Could we instead use the layoutTestController to enable or disable
>>>> whatever flags at test-time rather than on the command line?  For
>>>> example, we could put the guts of the 2d canvas tests into a
>>>> JavaScript file and include it in two HTML files, one that sets the
>>>> flag and one that does not.  That's how we handle strict vs quirks
>>>> mode, for example.
>>>
>>> There are 802 tests in canvas/philip/tests/ (which is an imported suite),
>>> currently all implemented as separate .html files, and another 166 in
>>> fast/canvas.  It seems a bit tedious to manually wrap each of these tests in
>>> two different wrappers.  That doesn't seem like a very practical option,
>>> although it has some nice aspects.
>>
>> I'd be happy to edit these thousand files if it meant we could remove
>> the complexity of the GPU configuration.  The ongoing maintenance cost
>> of the GPU configuration strikes me as a lot greater than the one-time
>> cost of changing these tests.  (There's also the side-benefit of
>> letting all the other ports increase their test coverage by wiring up
>> the LayoutTestController method.)
>>
>> I suspect (although I'd have to look more carefully), that we could
>> make this work for the canvas/philip/tests tests without actually
>> modifying the files (e.g., using iframes), if we wanted the ability to
>> easily import new versions of that test suite.
>
> Off-list, I wrote to you that the original motivation for the GPU
> tests was to be able to run *the exact same set of tests* through two
> different runtime configurations, and you suggested that maybe there
> was a common testing strategy that we use elsewhere in the test suites
> to accomplish this
> (perhaps using iframes as you suggest above).
>
> It may be that this common strategy is not that well-known or
> understood (I certainly wasn't aware of it). Can you expand on this
> more? Maybe we should document some examples on webkit.org or
> something?

The strategy is just what I wrote above.  For example, we run a number
of CSS tests both in standards and quirks mode.  We do this by having
most of the test logic in a JavaScript file that is included by two
HTML files, one in strict mode and one in quirks mode.  This approach
imposes no ongoing costs on the rest of the project.

Adam


>>>> >> As far as I can tell, other ports, such as Apple-Mac, make use of the
>>>> >> GPU but don't impose as large a complexity tax on the project.  Is
>>>> >> there something different about Chromium's GPU support that requires
>>>> >> this additional complexity?
>>>> >
>>>> > Other ports have less test coverage, and they don't run the pixel tests
>>>> > at
>>>> > all on the bots.
>>>>
>>>> Adam
>>>
>>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>