Page MenuHomePhabricator

'Randompage' in robots.txt should be 'Random'?
Closed, ResolvedPublic

Description

Author: fvulto

Description:
The file robots.txt on wikipedia.org contains:

Disallow: /wiki/Special:Randompage
Disallow: /wiki/Special%3ARandompage

but the actual link is Special:Random - not Special:Randompage, so
shouldn't robots.txt contain:

Disallow: /wiki/Special:Random
Disallow: /wiki/Special%3ARandom

The current setting lets Google index random pages. Now when you select
a search result, Google will serve you another (random) page - NOT the
page as suggested in the search results. For example, search Google for:

allinanchor:"Special:Random" site:wikipedia.org

and click on link '../wiki/Special:Random'.

Freddy Vulto
http://www.fvue.nl/wiki/Google_indexes_MediaWiki_page_with_url_of_Random_page


Version: unspecified
Severity: trivial
URL: http://en.wikipedia.org/robots.txt

Details

Reference
bz7775

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 9:26 PM
bzimport set Reference to bz7775.
bzimport added a subscriber: Unknown Object (MLST).

Some time ago Special:Randompage was moved to Special:Random; the old name remains as an alias.
Conveniently the new name is a prefix of the old one, so we only need one entry. ;)

Set. Note that due to caching you may see old items for a while.