[webkit-dev] LayoutTests results fallback graph

[Adam Barth]

On Wed, Jul 13, 2011 at 1:56 PM, Adam Barth <abarth at webkit.org> wrote:
> On Wed, Jul 13, 2011 at 1:36 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>> On Wed, Jul 13, 2011 at 1:28 PM, Adam Barth <abarth at webkit.org> wrote:
>>> On Wed, Jul 13, 2011 at 1:21 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>>> On Tue, Jul 12, 2011 at 10:01 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>>>> On Tue, Jul 12, 2011 at 9:01 PM, Adam Barth <abarth at webkit.org> wrote:
>>>>>> On Tue, Jul 12, 2011 at 8:34 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>>>>>>> On Tue, Jul 12, 2011 at 8:19 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>>>>>>>
>>>>>>>> Hum. I take it back ... it still wouldn't be a tree, since
>>>>>>>> chromium-mac-leopard would fall back to chromium-mac-snowleopard, then
>>>>>>>> mac-leopard, but chromium-mac-snow-leopard would fall back to
>>>>>>>> mac-snowleopard (giving chromium-mac-snowleopard two parents). And it
>>>>>>>> looks like chromium-mac-leopard picks up 3,494 baselines from
>>>>>>>> mac-leopard :(.
>>>>>>>
>>>>>>> Can we create chromium-mac and move everything that's shared between
>>>>>>> chromium-mac-leopard and chromium-mac-snowleopard there?
>>>>>>> It seems wrong for chromium-mac-leopard to fallback to
>>>>>>> chromium-mac-snowleopard.
>>>>>>
>>>>>> This somewhat surprising fallback strategy is common across ports.
>>>>>> The "why" is explained on this wiki page:
>>>>>>
>>>>>> http://trac.webkit.org/wiki/LayoutTestsSearchPath
>>>>>>
>>>>>
>>>>> In addition, we do actually have a 'chromium-mac'; we don't have a
>>>>> 'chromium-mac-snowleopard'. I think I mixed that in my mind while
>>>>> typing this with the apple mac ports, where there are mac-leopard,
>>>>> mac-sl, and mac ports (the latter representing lion/future).
>>>>>
>>>>> Once Lion ships, chromium will undoubtedly add a chromium-mac-snowleopard dir.
>>>>>
>>>>> -- Dirk
>>>>>
>>>>
>>>> Okay, I pulled together a slightly more comprehensive report ... in
>>>> short, we pull things from everywhere. Maybe this is useful to someone
>>>> if they want to try and treeify the fallbacks :)
>>>>
>>>> The format should be fairly self-explanatory. It is a rollup report
>>>> for all of the baselines, grouped on the combination of ports,
>>>> platforms, and type of baselines. The first column is the
>>>> port/platform configuration. The second is the location of the test
>>>> ("generic" means not in a platform/* directory). The third is the type
>>>> of baseline for the test, the fourth is the location of the baseline
>>>> used, and the fifth is the total # of such baselines in that location.
>>>
>>> To confirm my understanding:
>>>
>>> This row means that the Chromium Mac port running on Snow Leopard gets
>>> at least 5567 -expected.png files from the LayoutTests/platform/mac
>>> directory?
>>>
>>> chromium-mac-snowleopard,generic,png,mac,5567
>>>
>>
>> That is correct.
>>
>>> This is great data!  If you're interested in crunching numbers, it
>>> might be interested to hack up the deduplicate-tests script to figure
>>> out how much of the possible sharing we're realizing with our current
>>> fallback graph.
>>
>> I'm not sure I follow what you have in mind here ...
>
> No worries.  I'll figure it out myself.

There are approximately 1500 redundant test results that we aren't
able to collapse using our current fallback strategy.

Adam