[webkit-dev] Top 100 sites browsing tests proprosal

[Aaron Boodman]

On Sun, Jan 22, 2012 at 12:00 PM, Peter Kasting <pkasting at google.com> wrote:
> On Sun, Jan 22, 2012 at 11:55 AM, Balazs Kelemen <kbalazs at webkit.org> wrote:
>>
>> As the goal is to test "real world" use case I think it can be even better
>> to simply load the sites from network.
>> I see only two disadvantage of that but neither of them are blocker:
>>    1. The sites changing over time. However, as this would be a smoke test
>> (and not a regtest or a perf test) I don't think it's a big problem
>>    2. Cannot test offline. Well, I don't think taking part in the
>> development of WebKit is possible without being online anyway :)
>> Do you see other disadvantages?
>
>
> Loading over a network connection is slower.  Especially if you expect
> developers to run these tests locally, this could be a serious time sink.
>
> Loading real-world sites over a real network connection adds numerous
> possible flakiness/failure sources.  I would never use something like this
> as part of an automated system.
>
> Arguably, it's not kind to site authors to constantly reload their real
> sites as part of an automated testing program.
>
> From experience using offline copies of popular sites for testing in both
> Firefox and Chrome, I strongly suggest you go the offline route.

How would you capture and replay websites offline? This is non-trivial.

In Chromium, we have --record-mode and --playback-mode flags for this
use case. They put the cache, cookie jar, and some other components
into a special mode where everything is aggressively captured and
replayed without going to the net. In the case of Firefox, I believe
someone at one time implemented a Firefox extension that rewrote pages
to capture resources and use local references.

Since most websites are not redistributable, it seems like you need
some code like Chromium's or Firefox's in WebKit in order to make this
proposal work with offline data.

- a