[webkit-dev] Top 100 sites browsing tests proprosal

[Tony Gentilcore]

On Mon, Jan 23, 2012 at 2:14 AM, Aaron Boodman <aa at google.com> wrote:
> On Sun, Jan 22, 2012 at 12:00 PM, Peter Kasting <pkasting at google.com> wrote:
>> On Sun, Jan 22, 2012 at 11:55 AM, Balazs Kelemen <kbalazs at webkit.org> wrote:
>>>
>>> As the goal is to test "real world" use case I think it can be even better
>>> to simply load the sites from network.
>>> I see only two disadvantage of that but neither of them are blocker:
>>>    1. The sites changing over time. However, as this would be a smoke test
>>> (and not a regtest or a perf test) I don't think it's a big problem
>>>    2. Cannot test offline. Well, I don't think taking part in the
>>> development of WebKit is possible without being online anyway :)
>>> Do you see other disadvantages?
>>
>>
>> Loading over a network connection is slower.  Especially if you expect
>> developers to run these tests locally, this could be a serious time sink.
>>
>> Loading real-world sites over a real network connection adds numerous
>> possible flakiness/failure sources.  I would never use something like this
>> as part of an automated system.
>>
>> Arguably, it's not kind to site authors to constantly reload their real
>> sites as part of an automated testing program.
>>
>> From experience using offline copies of popular sites for testing in both
>> Firefox and Chrome, I strongly suggest you go the offline route.
>
> How would you capture and replay websites offline? This is non-trivial.

Very much non-trivial ;)

Depending on which route you end up deciding to take, you might also
want to consider http://code.google.com/p/web-page-replay/ which
scrapes and replays web pages without being tied to a specific
browser. It does some fuzzy request matching and injects a bit of JS
to ensure deterministic playback w.r.t the issues Aaron mentions here.

Without something like that, it is surprisingly difficult to replay
real-world web pages.

>
> In Chromium, we have --record-mode and --playback-mode flags for this
> use case. They put the cache, cookie jar, and some other components
> into a special mode where everything is aggressively captured and
> replayed without going to the net. In the case of Firefox, I believe
> someone at one time implemented a Firefox extension that rewrote pages
> to capture resources and use local references.
>
> Since most websites are not redistributable, it seems like you need
> some code like Chromium's or Firefox's in WebKit in order to make this
> proposal work with offline data.
>
> - a
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev