Description of problem: For years I used http://username:password@server/ to download a files from password protected pages. After update from F16 to F17 (and this applies to F18 wget too) I can not login this way anymore. Version-Release number of selected component (if applicable): wget-1.14-3.fc18.x86_64 wget-1.13.4-7.fc17.i686 How reproducible: Always Steps to Reproduce: 1. wget http://username:password@server/jpg/image.jpg Actual results: It is repeating many times: wget http://username:password@server/jpg/image.jpg --2013-02-18 14:41:25-- http://username:*password*@server/jpg/image.jpg Resolving server (server)... 192.168.1.1 Connecting to server (server)|192.168.1.1|:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Failed writing HTTP request: Bad file descriptor. Retrying. --2013-02-18 14:41:26-- (try: 2) http://username:*password*@server/jpg/image.jpg Connecting to server (server)|192.168.1.1|:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Failed writing HTTP request: Bad file descriptor. Retrying. --2013-02-18 14:41:28-- (try: 3) http://username:*password*@server/jpg/image.jpg Connecting to server (server)|192.168.1.1|:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Failed writing HTTP request: Bad file descriptor. Retrying. With no success. Expected results: Login in on try 2. Additional info: With explicit --user and --password the login works OK.
I'm not able to reproduce your issue. I set up a httpd server with php. I used php site with Digest HTTP Authentication [1]. $ rpm -qi wget Name : wget Version : 1.14 Release : 3.fc18 Architecture: x86_64 Install Date: Fri 25 Jan 2013 01:12:35 PM CET Group : Applications/Internet Size : 2046302 License : GPLv3+ Signature : RSA/SHA256, Wed 17 Oct 2012 05:38:13 PM CEST, Key ID ff01125cde7f38bd Source RPM : wget-1.14-3.fc18.src.rpm Build Date : Wed 17 Oct 2012 01:13:08 PM CEST Build Host : buildvm-14.phx2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : http://www.gnu.org/software/wget/ Summary : A utility for retrieving files using the HTTP or FTP protocols Description : GNU Wget is a file retrieval utility which can use either the HTTP or FTP protocols. Wget features include the ability to work in the background while you are logged out, recursive retrieval of directories, file name wildcard matching, remote file timestamp storage and comparison, use of Rest with FTP servers and Range with HTTP servers to retrieve files over slow or unstable connections, support for Proxy servers, and configurability. $ wget http://localhost/auth/ --2013-02-19 10:48:33-- http://localhost/auth/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Authorization failed. $ wget http://admin:mypass@localhost/auth/ --2013-02-19 10:48:24-- http://admin:*password*@localhost/auth/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Reusing existing connection to localhost:80. HTTP request sent, awaiting response... 200 OK Length: 29 [text/html] Saving to: ‘index.html.1’ 100%[===========================================>] 29 --.-K/s in 0s 2013-02-19 10:48:25 (1.85 MB/s) - ‘index.html.1’ saved [29/29] Also note that: wget-1.13.4-7.fc17 was built on 2012-10-10 wget-1.14-3.fc18 was built on 2012-10-17 and nothing changed in wget since then. [1] http://php.net/manual/en/features.http-auth.php (Example #2)
This is HTTP kind of password protection in Boa/0.94.14rc21. I tried it with apache and there wget works too. Against boa it repeatadly prints "Failed writing HTTP request: Bad file descriptor." The last version that is working is the one from F16 i.e. wget-1.12-6.fc16.i686.rpm also build at 2012-10-10. I never tried version from F17 or F18 up to now. This is wget 1.12 on F17 (from F16 rpm) - working OK. ./wget http://username:password@server/ --2013-02-19 20:23:09-- http://username:*password*@server/ Resolving server... 192.168.1.1 Connecting to server|192.168.1.1|:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Connecting to server|192.168.1.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 699 [text/html] Saving to: `index.html' 100%[==============================================================>] 699 --.-K/s in 0s 2013-02-19 20:23:09 (25.4 MB/s) - `index.html' saved [699/699]
(In reply to comment #2) > This is HTTP kind of password protection in Boa/0.94.14rc21. Can you please describe how can I set up your testing environment? Also if you use Boa, please describe how to set up authentication (also attaching http server configuration would be great). I have no experiences with Boa and was not able to get php nor other authentication working in a short time. I need to reproduce your issue first, so I can find out what is wrong with wget. Thanks!
The following upstream bug looks the same as your issue: savannah.gnu.org/bugs/?func=detailitem&item_id=34141
It is an embedded system - this means it could be met in embedded devices that use uclinux. It looks like the boa in Fedora does not have the Auth patch/directive included, therefore I can not create a test case with Fedoras' boa version. At the moment I can not provide any other way to reproduce it, but I may try to create an image for KVM from a different device - but at the moment I am not sure it exhibits the same behaviour.
(In reply to comment #5) > It is an embedded system - this means it could be met in embedded devices > that use uclinux. > > It looks like the boa in Fedora does not have the Auth patch/directive > included, therefore I can not create a test case with Fedoras' boa version. > > At the moment I can not provide any other way to reproduce it, but I may try > to create an image for KVM from a different device - but at the moment I am > not sure it exhibits the same behaviour. I'm looking forward to your reproducer. I will try to come up with something by myself.
As the patch for authentication is unsupported by Boa, I am not able to provide anything to reproduce this issue, and there is a known workaround I would consider this a WONTFIX, as it may not be a wget fault. If you agree then please close the bug.
(In reply to comment #7) > As the patch for authentication is unsupported by Boa, I am not able to > provide anything to reproduce this issue, and there is a known workaround I > would consider this a WONTFIX, as it may not be a wget fault. > > If you agree then please close the bug. I would like to try one more thing before closing this bug. Would it be possible to set up a sample user account on your embedded system and then capture packets (using tcpdump or wireshark) exchanged between wget and your embedded device? Please create two .pcap files with the communication: 1. When using wget http://username:password@server/ syntax. 2. When using --user and --password options. Thank you.
Created attachment 702250 [details] wget capture boa http Here is a password protected zip file with a tcpdump of both cases. Password is Boa2Http command: tcpdump -tttnn -w boa_http_wget.pcap -i eth0 host 192.168.2.133 The bad one: $ wget http://test:testt@192.168.2.133/ --2013-02-25 09:42:52-- http://test:*password*@192.168.2.133/ Connecting to 192.168.2.133:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Failed writing HTTP request: Bad file descriptor. Retrying. --2013-02-25 09:42:53-- (try: 2) http://test:*password*@192.168.2.133/ Connecting to 192.168.2.133:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Failed writing HTTP request: Bad file descriptor. Retrying. --2013-02-25 09:42:55-- (try: 3) http://test:*password*@192.168.2.133/ Connecting to 192.168.2.133:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Failed writing HTTP request: Bad file descriptor. Retrying. --2013-02-25 09:42:58-- (try: 4) http://test:*password*@192.168.2.133/ Connecting to 192.168.2.133:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Failed writing HTTP request: Bad file descriptor. Retrying. --2013-02-25 09:43:02-- (try: 5) http://test:*password*@192.168.2.133/ Connecting to 192.168.2.133:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Failed writing HTTP request: Bad file descriptor. Retrying. OK: $ wget --user test --password testt http://192.168.2.133/ --2013-02-25 09:45:29-- http://192.168.2.133/ Connecting to 192.168.2.133:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Failed writing HTTP request: Bad file descriptor. Retrying. --2013-02-25 09:45:30-- (try: 2) http://192.168.2.133/ Connecting to 192.168.2.133:80... connected. HTTP request sent, awaiting response... 200 OK Length: 699 [text/html] Saving to: ‘index.html.2’ 100%[==============================================================>] 699 --.-K/s in 0s 2013-02-25 09:45:30 (27.6 MB/s) - ‘index.html.2’ saved [699/699]
I think I got to the bottom of this issue. I succeeded with setting up boa server and I was able to find the Auth patch [1] for boa. It is not working 100% correctly I think, but it at least behave similarly to what you described and what was in your communication dumps. The problem is that boa server answers to the initial GET packet with two 401 Unauthorised packets (at least in my case). From your dumps I see that boa is answering with only one packet, but it is somehow broken. Although wget seems to interpret it correctly it will close the connection without setting opened socket variable to -1. Then it fails trying to write to that already closed socket. This repeats for default maximum retries times. Reason why wget-1.12 works is how it handle connections. In wget-1.12 the connection established for GET request is closed before authentication and the socket variable is set to -1. So when wget tries to authenticate to the server, it establishes new connection and everything works. Reason why --user and --password options works is that those credentials are used as default for site-wide authentication. What wget does in this case is pretty much the same as when you pass credentials in the URL with one difference. When wget receives 401 Unauthorised from server and have those "default" credentials set AND have NOT credentials in URL, then it adds the host address to a list of hosts which request authentication. After that it also closes the initial connection, then fails to write new GET request with authentication credentials. BUT when it tries for the second time it will find out that the host address is in list of hosts which request authentication and therefore it will send GET request including authentication credentials. So that's why it works. I think the proper fix for this would be to set the socket variable to -1 in case of a problem when authenticating and closing the socket. This will make sure that wget will not try to write data to already closed socket but rather it will create new connection. I'll prepare a testing build with the fix and add it here. It would be really great if you could test it. [1] http://sourceforge.net/tracker/?func=detail&aid=935995&group_id=78&atid=300078
Created attachment 713205 [details] Patch for this issue.
Can you please test the following build if it works with your boa server and wget when using http://username:password@server/ syntax? http://koji.fedoraproject.org/koji/taskinfo?taskID=5147051 Thank you!
Yes, this build works OK. Thanks a lot. wget http://username:pass@server/jpg/image.jpg --2013-03-20 20:20:18-- http://username:*password*@server/jpg/image.jpg Resolving server (server)... 192.168.2.133 Connecting to server (server)|192.168.2.133|:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Connecting to server (server)|192.168.2.133|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [image/jpeg] Saving to: ‘image.jpg’ [ <=> ] 9,833 --.-K/s in 0.007s 2013-03-20 20:20:18 (1.40 MB/s) - ‘image.jpg’ saved [9833]
Thank you for testing. I will wait for the upstream response. If there will be none in some significant time period (1/2 a year or so) I'll include this patch in Fedora wget.
Good news everyone :) Upstream accepted my patch so I will fix this issue in Fedora and push an update soon.
wget-1.14-8.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/wget-1.14-8.fc19
wget-1.14-5.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/wget-1.14-5.fc18
Package wget-1.14-5.fc18: * should fix your issue, * was pushed to the Fedora 18 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing wget-1.14-5.fc18' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-13011/wget-1.14-5.fc18 then log in and leave karma (feedback).
wget-1.14-8.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
wget-1.14-5.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.
Confirming fixed in f18. Thank you.