Bug 462451 – wget -r only gets index.html and robots.txt

Bug 462451 - wget -r only gets index.html and robots.txt

Summary:	wget -r only gets index.html and robots.txt

Status:	CLOSED NOTABUG

Aliases:	None

Product:	Fedora
Classification:	Fedora
Component:	wget (Show other bugs)
Sub Component:
Version:	rawhide
Hardware:	All Linux

Priority	medium Severity medium
Target Milestone:	---
Target Release:	---
Assigned To:	Karsten Hopp
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2008-09-16 09:21 EDT by Patrice Dumas
Modified:	2008-09-25 11:35 EDT (History)
CC List:	1 user (show)

See Also:
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2008-09-25 11:35:56 EDT
Type:	---
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Groups: None (edit)

Description Patrice Dumas 2008-09-16 09:21:14 EDT

Description of problem:

This may be the same than Bug 286161, but the patch that was
problematic shouldn't be there.

For example (beware that this is a scratch build, so will 
disappear in 1 week):
wget -r -np http://koji.fedoraproject.org/scratch/pertusus/task_827728/



$ wget -r -np http://koji.fedoraproject.org/scratch/pertusus/task_827728/
--2008-09-16 15:20:42--  http://koji.fedoraproject.org/scratch/pertusus/task_827728/
Resolving koji.fedoraproject.org... 209.132.176.125
Connecting to koji.fedoraproject.org|209.132.176.125|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3607 (3.5K) [text/html]
Saving to: `koji.fedoraproject.org/scratch/pertusus/task_827728/index.html'

100%[============================================================================================>] 3,607       --.-K/s   in 0s      

2008-09-16 15:20:42 (45.8 MB/s) - `koji.fedoraproject.org/scratch/pertusus/task_827728/index.html' saved [3607/3607]

Loading robots.txt; please ignore errors.
--2008-09-16 15:20:42--  http://koji.fedoraproject.org/robots.txt
Connecting to koji.fedoraproject.org|209.132.176.125|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26 [text/plain]
Saving to: `koji.fedoraproject.org/robots.txt'

100%[============================================================================================>] 26          --.-K/s   in 0s      

2008-09-16 15:20:42 (727 KB/s) - `koji.fedoraproject.org/robots.txt' saved [26/26]

FINISHED --2008-09-16 15:20:42--
Downloaded: 2 files, 3.5K in 0s (31.5 MB/s)


Version-Release number of selected component (if applicable):

wget-1.11.4-1.fc10.i386

How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Micah Cowan 2008-09-16 12:14:32 EDT

Not a bug. Contents of robots.txt file:

User-agent: *
Disallow: /

Please see http://wget.addictivecode.org/FrequentlyAskedQuestions#robots

Comment 2 Patrice Dumas 2008-09-25 11:35:56 EDT

Sorry for the RTFM bug report...

Note You need to log in before you can comment on or make changes to this bug.