Bug 1411186 - Bugzilla 5 Beta site has no robots.txt
Summary: Bugzilla 5 Beta site has no robots.txt
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Bugzilla
Classification: Community
Component: Administration
Version: 5.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: 5.0
Assignee: Jeff Fearn 🐞
QA Contact: tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-09 01:15 UTC by Jason McDonald
Modified: 2018-12-09 06:29 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-16 00:34:11 UTC


Attachments (Terms of Use)

Description Jason McDonald 2017-01-09 01:15:27 UTC
Description of problem:
The beta site is missing a robots.txt.  The lack of this file will allow search engines to index the bugs there, polluting the search engines with references to non-production bugs.

The beta site should have a copy of the robots.txt from Partner-Bugzilla, which will tell search engines to ignore the entire site.

When Bugzilla 5 goes to production, it should have the same robots.txt that the current Bugzilla 4 production instance has.

Version-Release number of selected component (if applicable):
5.0.3 (beta.bugzilla.redhat.com)

How reproducible:
Always

Steps to Reproduce:
1. wget https://beta.bugzilla.redhat.com/robots.txt

Actual results:
404 Not Found

Expected results:
The robots.txt from partner-bugzilla is:

User-agent: *
Disallow: /

Additional info:
The robots.txt from production is:

User-agent: *
Crawl-delay: 5
Disallow: /
Disallow: /show_bug.cgi*ctype=xml*
Allow: /index.cgi
Allow: /show_bug.cgi
Allow: /sitemap_index.xml
Allow: /sitemap*xml.gz
Sitemap: https://bugzilla.redhat.com/sitemap_index.xml

Comment 1 Jeff Fearn 🐞 2017-01-09 02:16:20 UTC
Looks like another victim of the path change.

https://beta.bugzilla.redhat.com/bugzilla/robots.txt


Note You need to log in before you can comment on or make changes to this bug.