[webkit-dev] Pages from the wiki vanished from google search results

[William Siegrist]

On Jun 5, 2011, at 7:20 PM, Matt Falkenhagen wrote:

>> 
>> http://www.google.com/search?hl=en&q=webkit+team+%22Here+is+the+key+for+each%22
>> http://www.google.com/search?hl=en&q=Qt+Port+of+WebKit+%22List+of+all+QtWebKit+wiki+pages%22
> 
> I had noticed this too and asked about it internally. It turns out
> webkit.org was (is?) returning 403 errors to googlebot. This caused
> the pages to be lost. However, it is still returning content on
> https:// addresses, so those are now starting to be picked up as you
> can now see in the Google searches above.


Googlebot was apparently blacklisted at some point for hitting the servers too hard. I removed them, they should not get 403s anymore. 

-Bill