Bug 1411186

Summary:	Bugzilla 5 Beta site has no robots.txt
Product:	[Community] Bugzilla	Reporter:	Jason McDonald <jmcdonal>
Component:	Administration	Assignee:	Jeff Fearn 🐞 <jfearn>
Status:	CLOSED CURRENTRELEASE	QA Contact:	tools-bugs <tools-bugs>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	5.0	CC:	huiwang, qgong, yijli
Target Milestone:	5.0
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-01-16 00:34:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jason McDonald 2017-01-09 01:15:27 UTC

Description of problem:
The beta site is missing a robots.txt.  The lack of this file will allow search engines to index the bugs there, polluting the search engines with references to non-production bugs.

The beta site should have a copy of the robots.txt from Partner-Bugzilla, which will tell search engines to ignore the entire site.

When Bugzilla 5 goes to production, it should have the same robots.txt that the current Bugzilla 4 production instance has.

Version-Release number of selected component (if applicable):
5.0.3 (beta.bugzilla.redhat.com)

How reproducible:
Always

Steps to Reproduce:
1. wget https://beta.bugzilla.redhat.com/robots.txt

Actual results:
404 Not Found

Expected results:
The robots.txt from partner-bugzilla is:

User-agent: *
Disallow: /

Additional info:
The robots.txt from production is:

User-agent: *
Crawl-delay: 5
Disallow: /
Disallow: /show_bug.cgi*ctype=xml*
Allow: /index.cgi
Allow: /show_bug.cgi
Allow: /sitemap_index.xml
Allow: /sitemap*xml.gz
Sitemap: https://bugzilla.redhat.com/sitemap_index.xml

Comment 1 Jeff Fearn 🐞 2017-01-09 02:16:20 UTC

Looks like another victim of the path change.

https://beta.bugzilla.redhat.com/bugzilla/robots.txt