Red Hat Bugzilla – Bug 126482
htdig does not understand robots.txt file
Last modified: 2015-03-04 20:14:09 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040619
Description of problem:
hidig-3.2.0b5 does not work if the website has _any_ robots.txt file.
It is a known bug of this version of htdig. A patch is available. See
The patch is a very simple one:
--- htdig/Server.cc.orig 2003-10-27 17:28:52.000000000 -0600
+++ htdig/Server.cc 2003-11-13 11:31:24.000000000 -0600
@@ -338,6 +338,8 @@
String fullpatt = "^[^:]*://[^/]*(";
fullpatt << pattern << ')';
+ if (pattern.length() == 0)
+ fullpatt = "";
I have the same symptoms with FC2 (i386). Removing the (correct!)
robot.txt file makes htdig index my site again.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Set up a website
2. Create a robot.txt file (with at least "User Agent: *" line)
3. Update to FC2
4. See that the word database is empty
Actual Results: Searching the website gives an arror about
db.words.db not found
Expected Results: The website is searchable
OK, looks sane and logical.
Included in htdig-3.2.0b6-1 and later.
Read ya, Phil