[webkit-dev] Top 100 sites browsing tests proprosal

[Joe Mason]

I don't think this needs to be done by every developer on every checkin.  It could be done by the buildbots daily, and whenever a crash is found, somebody would need to find the cause and turn it into a real regression test for later.  The fact that the web sites change over time isn't important because the cause of the failure would get a separate test.

(It would still be a good idea to save the site locally so that the exact version can be replayed after a crash, but this would be a temporary copy which is thrown away as soon as the test succeeds.)

From: webkit-dev-bounces at lists.webkit.org [mailto:webkit-dev-bounces at lists.webkit.org] On Behalf Of Peter Kasting
Sent: Sunday, January 22, 2012 3:01 PM
To: Balazs Kelemen
Cc: webkit-dev at lists.webkit.org
Subject: Re: [webkit-dev] Top 100 sites browsing tests proprosal

On Sun, Jan 22, 2012 at 11:55 AM, Balazs Kelemen <kbalazs at webkit.org<mailto:kbalazs at webkit.org>> wrote:
As the goal is to test "real world" use case I think it can be even better to simply load the sites from network.
I see only two disadvantage of that but neither of them are blocker:
   1. The sites changing over time. However, as this would be a smoke test (and not a regtest or a perf test) I don't think it's a big problem
   2. Cannot test offline. Well, I don't think taking part in the development of WebKit is possible without being online anyway :)
Do you see other disadvantages?

Loading over a network connection is slower.  Especially if you expect developers to run these tests locally, this could be a serious time sink.

Loading real-world sites over a real network connection adds numerous possible flakiness/failure sources.  I would never use something like this as part of an automated system.

Arguably, it's not kind to site authors to constantly reload their real sites as part of an automated testing program.

>From experience using offline copies of popular sites for testing in both Firefox and Chrome, I strongly suggest you go the offline route.

PK

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120123/9486c6bf/attachment.html>