[webkit-dev] Spam and indexing
annulen at yandex.ru
Thu Mar 28 14:10:32 PDT 2019
28.03.2019, 23:58, "Alexey Proskuryakov" <ap at webkit.org>:
> The robots.txt file that we have on bugs.webkit.org currently allows search engines access to individual bug pages, but not to any bug lists. As a result, search engines and the Internet Archive only index bugs that were filed before robots.txt changes a few years ago, and bugs that are directly linked from webpages elsewhere. These bugs are where most spam content naturally ends up on.
> This is quite wrong, as indexing just a subset of bugs is not beneficial to anyone other than spammers. So we can go in either direction:
> 1. Allow indexers to enumerate bugs, thus indexing all of them.
> Seems reasonable that people should be able to find bugs using search engines.
Yes, and it may give better result even than searching bugzilla directly
>On the other hand, we'll need to do something to ensure that indexers don't destroy Bugzilla performance,
This can be solved by caching
>and of course spammers will love having more flexibility.
rel="nofollow" on all links in comments should be enough to make spamming useless
> 2. Block indexing completely.
> Seems like no one was bothered by lack of indexing on new bugs so far.
That's survival bias - if nobody can find relevant bugs, nobody will ever complain
> For reference, here is the current robots.txt content:
> $ curl https://bugs.webkit.org/robots.txt
> User-agent: *
> Allow: /index.cgi
> Allow: /show_bug.cgi
> Disallow: /
> Crawl-delay: 20
> - Alexey
> - Alexey
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
More information about the webkit-dev