Bug 462451 - wget -r only gets index.html and robots.txt
wget -r only gets index.html and robots.txt
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: wget (Show other bugs)
rawhide
All Linux
medium Severity medium
: ---
: ---
Assigned To: Karsten Hopp
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-09-16 09:21 EDT by Patrice Dumas
Modified: 2008-09-25 11:35 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-09-25 11:35:56 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Patrice Dumas 2008-09-16 09:21:14 EDT
Description of problem:

This may be the same than Bug 286161, but the patch that was
problematic shouldn't be there.

For example (beware that this is a scratch build, so will 
disappear in 1 week):
wget -r -np http://koji.fedoraproject.org/scratch/pertusus/task_827728/



$ wget -r -np http://koji.fedoraproject.org/scratch/pertusus/task_827728/
--2008-09-16 15:20:42--  http://koji.fedoraproject.org/scratch/pertusus/task_827728/
Resolving koji.fedoraproject.org... 209.132.176.125
Connecting to koji.fedoraproject.org|209.132.176.125|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3607 (3.5K) [text/html]
Saving to: `koji.fedoraproject.org/scratch/pertusus/task_827728/index.html'

100%[============================================================================================>] 3,607       --.-K/s   in 0s      

2008-09-16 15:20:42 (45.8 MB/s) - `koji.fedoraproject.org/scratch/pertusus/task_827728/index.html' saved [3607/3607]

Loading robots.txt; please ignore errors.
--2008-09-16 15:20:42--  http://koji.fedoraproject.org/robots.txt
Connecting to koji.fedoraproject.org|209.132.176.125|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26 [text/plain]
Saving to: `koji.fedoraproject.org/robots.txt'

100%[============================================================================================>] 26          --.-K/s   in 0s      

2008-09-16 15:20:42 (727 KB/s) - `koji.fedoraproject.org/robots.txt' saved [26/26]

FINISHED --2008-09-16 15:20:42--
Downloaded: 2 files, 3.5K in 0s (31.5 MB/s)


Version-Release number of selected component (if applicable):

wget-1.11.4-1.fc10.i386

How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Micah Cowan 2008-09-16 12:14:32 EDT
Not a bug. Contents of robots.txt file:

User-agent: *
Disallow: /

Please see http://wget.addictivecode.org/FrequentlyAskedQuestions#robots
Comment 2 Patrice Dumas 2008-09-25 11:35:56 EDT
Sorry for the RTFM bug report...

Note You need to log in before you can comment on or make changes to this bug.