Bug 912358 - wget syntax to login to protected site using http://username:password@server/ stopped working
wget syntax to login to protected site using http://username:password@server/...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: wget (Show other bugs)
18
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Tomáš Hozza
Fedora Extras Quality Assurance
: Patch
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-02-18 09:05 EST by Adam Pribyl
Modified: 2013-07-24 14:43 EDT (History)
1 user (show)

See Also:
Fixed In Version: wget-1.14-5.fc18
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-21 20:37:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
wget capture boa http (3.53 KB, application/x-zip-compressed)
2013-02-25 04:10 EST, Adam Pribyl
no flags Details
Patch for this issue. (1.06 KB, patch)
2013-03-20 07:59 EDT, Tomáš Hozza
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
GNU Savannah 38554 None None None Never

  None (edit)
Description Adam Pribyl 2013-02-18 09:05:36 EST
Description of problem:
For years I used http://username:password@server/ to download a files from password protected pages. After update from F16 to F17 (and this applies to F18 wget too) I can not login this way anymore.

Version-Release number of selected component (if applicable):
wget-1.14-3.fc18.x86_64
wget-1.13.4-7.fc17.i686


How reproducible:
Always

Steps to Reproduce:
1. wget http://username:password@server/jpg/image.jpg
  
Actual results:

It is repeating many times:

wget http://username:password@server/jpg/image.jpg
--2013-02-18 14:41:25--  http://username:*password*@server/jpg/image.jpg
Resolving server (server)... 192.168.1.1
Connecting to server (server)|192.168.1.1|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Failed writing HTTP request: Bad file descriptor.
Retrying.

--2013-02-18 14:41:26--  (try: 2)  http://username:*password*@server/jpg/image.jpg
Connecting to server (server)|192.168.1.1|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Failed writing HTTP request: Bad file descriptor.
Retrying.

--2013-02-18 14:41:28--  (try: 3)  http://username:*password*@server/jpg/image.jpg
Connecting to server (server)|192.168.1.1|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Failed writing HTTP request: Bad file descriptor.
Retrying.

With no success.

Expected results:
Login in on try 2.

Additional info:
With explicit --user and --password the login works OK.
Comment 1 Tomáš Hozza 2013-02-19 07:38:39 EST
I'm not able to reproduce your issue.

I set up a httpd server with php. I used php site with Digest HTTP
Authentication [1].

$ rpm -qi wget
Name        : wget
Version     : 1.14
Release     : 3.fc18
Architecture: x86_64
Install Date: Fri 25 Jan 2013 01:12:35 PM CET
Group       : Applications/Internet
Size        : 2046302
License     : GPLv3+
Signature   : RSA/SHA256, Wed 17 Oct 2012 05:38:13 PM CEST, Key ID ff01125cde7f38bd
Source RPM  : wget-1.14-3.fc18.src.rpm
Build Date  : Wed 17 Oct 2012 01:13:08 PM CEST
Build Host  : buildvm-14.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.gnu.org/software/wget/
Summary     : A utility for retrieving files using the HTTP or FTP protocols
Description :
GNU Wget is a file retrieval utility which can use either the HTTP or
FTP protocols. Wget features include the ability to work in the
background while you are logged out, recursive retrieval of
directories, file name wildcard matching, remote file timestamp
storage and comparison, use of Rest with FTP servers and Range with
HTTP servers to retrieve files over slow or unstable connections,
support for Proxy servers, and configurability.

$ wget http://localhost/auth/
--2013-02-19 10:48:33--  http://localhost/auth/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authorization failed.

$ wget http://admin:mypass@localhost/auth/
--2013-02-19 10:48:24--  http://admin:*password*@localhost/auth/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Reusing existing connection to localhost:80.
HTTP request sent, awaiting response... 200 OK
Length: 29 [text/html]
Saving to: ‘index.html.1’

100%[===========================================>] 29          --.-K/s   in 0s      

2013-02-19 10:48:25 (1.85 MB/s) - ‘index.html.1’ saved [29/29]


Also note that:
wget-1.13.4-7.fc17 was built on	2012-10-10
wget-1.14-3.fc18 was built on 2012-10-17
and nothing changed in wget since then. 

[1] http://php.net/manual/en/features.http-auth.php (Example #2)
Comment 2 Adam Pribyl 2013-02-19 14:27:03 EST
This is HTTP kind of password protection in Boa/0.94.14rc21.

I tried it with apache and there wget works too. Against boa it repeatadly prints
"Failed writing HTTP request: Bad file descriptor."

The last version that is working is the one from F16 i.e. wget-1.12-6.fc16.i686.rpm also build at 2012-10-10. I never tried version from F17 or F18 up to now.

This is wget 1.12 on F17 (from F16 rpm) - working OK.

./wget http://username:password@server/
--2013-02-19 20:23:09--  http://username:*password*@server/
Resolving server... 192.168.1.1
Connecting to server|192.168.1.1|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to server|192.168.1.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 699 [text/html]
Saving to: `index.html'

100%[==============================================================>] 699         --.-K/s   in 0s      

2013-02-19 20:23:09 (25.4 MB/s) - `index.html' saved [699/699]
Comment 3 Tomáš Hozza 2013-02-20 03:58:06 EST
(In reply to comment #2)
> This is HTTP kind of password protection in Boa/0.94.14rc21.

Can you please describe how can I set up your testing environment? Also if you
use Boa, please describe how to set up authentication (also attaching http server
configuration would be great). I have no experiences with Boa and was not able to
get php nor other authentication working in a short time. I need to reproduce your
issue first, so I can find out what is wrong with wget.

Thanks!
Comment 4 Tomáš Hozza 2013-02-20 04:58:11 EST
The following upstream bug looks the same as your issue:
savannah.gnu.org/bugs/?func=detailitem&item_id=34141
Comment 5 Adam Pribyl 2013-02-20 14:11:36 EST
It is an embedded system - this means it could be met in embedded devices that use uclinux.

It looks like the boa in Fedora does not have the Auth patch/directive included, therefore I can not create a test case with Fedoras' boa version.

At the moment I can not provide any other way to reproduce it, but I may try to create an image for KVM from a different device - but at the moment I am not sure it exhibits the same behaviour.
Comment 6 Tomáš Hozza 2013-02-21 02:41:01 EST
(In reply to comment #5)
> It is an embedded system - this means it could be met in embedded devices
> that use uclinux.
> 
> It looks like the boa in Fedora does not have the Auth patch/directive
> included, therefore I can not create a test case with Fedoras' boa version.
> 
> At the moment I can not provide any other way to reproduce it, but I may try
> to create an image for KVM from a different device - but at the moment I am
> not sure it exhibits the same behaviour.

I'm looking forward to your reproducer. I will try to come up with something
by myself.
Comment 7 Adam Pribyl 2013-02-24 13:33:38 EST
As the patch for authentication is unsupported by Boa, I am not able to provide anything to reproduce this issue, and there is a known workaround I would consider this a WONTFIX, as it may not be a wget fault.

If you agree then please close the bug.
Comment 8 Tomáš Hozza 2013-02-25 03:07:28 EST
(In reply to comment #7)
> As the patch for authentication is unsupported by Boa, I am not able to
> provide anything to reproduce this issue, and there is a known workaround I
> would consider this a WONTFIX, as it may not be a wget fault.
> 
> If you agree then please close the bug.

I would like to try one more thing before closing this bug. Would it be possible
to set up a sample user account on your embedded system and then capture packets
(using tcpdump or wireshark) exchanged between wget and your embedded device?

Please create two .pcap files with the communication:
1. When using wget http://username:password@server/ syntax.
2. When using --user and --password options. 

Thank you.
Comment 9 Adam Pribyl 2013-02-25 04:10:39 EST
Created attachment 702250 [details]
wget capture boa http

Here is a password protected zip file with a tcpdump of both cases. Password is Boa2Http

command: tcpdump -tttnn -w boa_http_wget.pcap -i eth0 host 192.168.2.133

The bad one:
$ wget http://test:testt@192.168.2.133/
--2013-02-25 09:42:52--  http://test:*password*@192.168.2.133/
Connecting to 192.168.2.133:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Failed writing HTTP request: Bad file descriptor.
Retrying.

--2013-02-25 09:42:53--  (try: 2)  http://test:*password*@192.168.2.133/
Connecting to 192.168.2.133:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Failed writing HTTP request: Bad file descriptor.
Retrying.

--2013-02-25 09:42:55--  (try: 3)  http://test:*password*@192.168.2.133/
Connecting to 192.168.2.133:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Failed writing HTTP request: Bad file descriptor.
Retrying.

--2013-02-25 09:42:58--  (try: 4)  http://test:*password*@192.168.2.133/
Connecting to 192.168.2.133:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Failed writing HTTP request: Bad file descriptor.
Retrying.

--2013-02-25 09:43:02--  (try: 5)  http://test:*password*@192.168.2.133/
Connecting to 192.168.2.133:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Failed writing HTTP request: Bad file descriptor.
Retrying.

OK:
$ wget --user test --password testt http://192.168.2.133/
--2013-02-25 09:45:29--  http://192.168.2.133/
Connecting to 192.168.2.133:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Failed writing HTTP request: Bad file descriptor.
Retrying.

--2013-02-25 09:45:30--  (try: 2)  http://192.168.2.133/
Connecting to 192.168.2.133:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 699 [text/html]
Saving to: ‘index.html.2’

100%[==============================================================>] 699         --.-K/s   in 0s      

2013-02-25 09:45:30 (27.6 MB/s) - ‘index.html.2’ saved [699/699]
Comment 10 Tomáš Hozza 2013-03-20 05:44:55 EDT
I think I got to the bottom of this issue. I succeeded with setting up boa server
and I was able to find the Auth patch [1] for boa. It is not working 100%
correctly I think, but it at least behave similarly to what you described and
what was in your communication dumps.

The problem is that boa server answers to the initial GET packet with two 401
Unauthorised packets (at least in my case). From your dumps I see that boa is
answering with only one packet, but it is somehow broken. Although wget seems
to interpret it correctly it will close the connection without setting opened
socket variable to -1. Then it fails trying to write to that already closed
socket. This repeats for default maximum retries times.

Reason why wget-1.12 works is how it handle connections. In wget-1.12 the
connection established for GET request is closed before authentication and
the socket variable is set to -1. So when wget tries to authenticate to the
server, it establishes new connection and everything works.

Reason why --user and --password options works is that those credentials are
used as default for site-wide authentication. What wget does in this case is
pretty much the same as when you pass credentials in the URL with one difference.
When wget receives 401 Unauthorised from server and have those "default"
credentials set AND have NOT credentials in URL, then it adds the host address
to a list of hosts which request authentication. After that it also closes the
initial connection, then fails to write new GET request with authentication
credentials. BUT when it tries for the second time it will find out that the
host address is in list of hosts which request authentication and therefore
it will send GET request including authentication credentials. So that's why
it works.

I think the proper fix for this would be to set the socket variable to -1
in case of a problem when authenticating and closing the socket. This will
make sure that wget will not try to write data to already closed socket
but rather it will create new connection.

I'll prepare a testing build with the fix and add it here. It would be really
great if you could test it.

[1] http://sourceforge.net/tracker/?func=detail&aid=935995&group_id=78&atid=300078
Comment 11 Tomáš Hozza 2013-03-20 07:59:55 EDT
Created attachment 713205 [details]
Patch for this issue.
Comment 12 Tomáš Hozza 2013-03-20 08:24:05 EDT
Can you please test the following build if it works with your boa server and
wget when using http://username:password@server/ syntax?

http://koji.fedoraproject.org/koji/taskinfo?taskID=5147051

Thank you!
Comment 13 Adam Pribyl 2013-03-20 15:25:18 EDT
Yes, this build works OK. Thanks a lot.

wget  http://username:pass@server/jpg/image.jpg
--2013-03-20 20:20:18--  http://username:*password*@server/jpg/image.jpg
Resolving server (server)... 192.168.2.133
Connecting to server (server)|192.168.2.133|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to server (server)|192.168.2.133|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [image/jpeg]
Saving to: ‘image.jpg’

    [ <=>                                                           ] 9,833       --.-K/s   in 0.007s  

2013-03-20 20:20:18 (1.40 MB/s) - ‘image.jpg’ saved [9833]
Comment 14 Tomáš Hozza 2013-03-21 04:01:41 EDT
Thank you for testing.

I will wait for the upstream response. If there will be none in some significant
time period (1/2 a year or so) I'll include this patch in Fedora wget.
Comment 15 Tomáš Hozza 2013-07-12 02:07:55 EDT
Good news everyone :) Upstream accepted my patch so I will fix this issue
in Fedora and push an update soon.
Comment 16 Fedora Update System 2013-07-15 03:55:29 EDT
wget-1.14-8.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/wget-1.14-8.fc19
Comment 17 Fedora Update System 2013-07-15 04:00:14 EDT
wget-1.14-5.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/wget-1.14-5.fc18
Comment 18 Fedora Update System 2013-07-15 21:33:50 EDT
Package wget-1.14-5.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing wget-1.14-5.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-13011/wget-1.14-5.fc18
then log in and leave karma (feedback).
Comment 19 Fedora Update System 2013-07-21 20:37:56 EDT
wget-1.14-8.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 20 Fedora Update System 2013-07-23 23:44:55 EDT
wget-1.14-5.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 21 Adam Pribyl 2013-07-24 03:38:46 EDT
Confirming fixed in f18. Thank you.

Note You need to log in before you can comment on or make changes to this bug.