Bug 462451 - wget -r only gets index.html and robots.txt
Summary: wget -r only gets index.html and robots.txt
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: wget
Version: rawhide
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Karsten Hopp
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2008-09-16 13:21 UTC by Patrice Dumas
Modified: 2008-09-25 15:35 UTC (History)
1 user (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2008-09-25 15:35:56 UTC
Type: ---
Embargoed:

Attachments (Terms of Use)

Description Patrice Dumas 2008-09-16 13:21:14 UTC 
Description of problem:

This may be the same than Bug 286161, but the patch that was
problematic shouldn't be there.

For example (beware that this is a scratch build, so will 
disappear in 1 week):
wget -r -np http://koji.fedoraproject.org/scratch/pertusus/task_827728/



$ wget -r -np http://koji.fedoraproject.org/scratch/pertusus/task_827728/
--2008-09-16 15:20:42--  http://koji.fedoraproject.org/scratch/pertusus/task_827728/
Resolving koji.fedoraproject.org... 209.132.176.125
Connecting to koji.fedoraproject.org|209.132.176.125|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3607 (3.5K) [text/html]
Saving to: `koji.fedoraproject.org/scratch/pertusus/task_827728/index.html'

100%[============================================================================================>] 3,607       --.-K/s   in 0s      

2008-09-16 15:20:42 (45.8 MB/s) - `koji.fedoraproject.org/scratch/pertusus/task_827728/index.html' saved [3607/3607]

Loading robots.txt; please ignore errors.
--2008-09-16 15:20:42--  http://koji.fedoraproject.org/robots.txt
Connecting to koji.fedoraproject.org|209.132.176.125|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26 [text/plain]
Saving to: `koji.fedoraproject.org/robots.txt'

100%[============================================================================================>] 26          --.-K/s   in 0s      

2008-09-16 15:20:42 (727 KB/s) - `koji.fedoraproject.org/robots.txt' saved [26/26]

FINISHED --2008-09-16 15:20:42--
Downloaded: 2 files, 3.5K in 0s (31.5 MB/s)


Version-Release number of selected component (if applicable):

wget-1.11.4-1.fc10.i386

How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Micah Cowan 2008-09-16 16:14:32 UTC 
Not a bug. Contents of robots.txt file:

User-agent: *
Disallow: /

Please see http://wget.addictivecode.org/FrequentlyAskedQuestions#robots
Comment 2 Patrice Dumas 2008-09-25 15:35:56 UTC 
Sorry for the RTFM bug report...
Note You need to log in before you can comment on or make changes to this bug.