[webkit-qt] Reproducing buildbot layout test results

[Andras Becsi]

Hi all,

the metrics in Qt layout test results are adjusted to Debian Lenny 
(5.0.4). Debian was chosen because we are familiar with it, it contains 
less bloat than Ubuntu and a Debian setup seemed to produce more stable 
results across changing Qt versions (currently there are however issues 
eg.: http://webkit.sed.hu/buildbot/waterfall the x86-32 Linux Qt-4.7.0 
Release has 140 failing tests due to metric differences). Some time 
soon, we are going to rebaseline the tests to match Qt 4.7.0 results. 
The Release bot is currently using Qt 4.6.2.

Our recent experiments concluded that the issue is not only related to 
fontconfig (et al.), but we could not nail down a fixed set of packages 
which seem to cause the problem (X itself and other libraries might also 
be involved). Further more there are almost thousand tests failing on 
our ARMv5 Test bot, which has the same Lenny version but Debian-ARM (the 
package versions are AFAIK the same as for x86*). So there also seem to 
be Qt-related platform issues.

To make the unusable layout regression testing somehow usable, we need 
to pin down the metrics, say that these are the correct results and 
specify the exact environment.

We are trying to find a solution, either by providing VM and/or bootable 
live CD images with the same environment as our buildbot servers, or by 
making a poll on our blog where everyone can vote for the distro which 
would make him the least headaches to set up (and hopefully not empty 
into a distrowar).
Eventually we could adjust the results to match the most used distro by 
QtWebKit developers (I'm not sure whether Ubuntu or SuSE would win).

  BR:
Andras

2010-09-22 15:14 keltezéssel, Andreas Kling írta:
> On 09/22/2010 02:58 PM, ext Antonio Gomes wrote:
>> ps: It sounds like a good time to migrate to Ubuntu: Qt4.7,
>> QtWebKit2.0, and all fresh "new". It deserves a new life to the bots
>
> +1
>
> Bringing the bots closer to what developers are (likely) using sounds
> like a great idea.
>
> Szeged folks: is this within the realm of possibility?
>
> -Kling
> _______________________________________________
> webkit-qt mailing list
> webkit-qt at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-qt