Bug 484491 - Yum cannot download repomd.xml when metalink has https url & http(s) proxy is specified
Yum cannot download repomd.xml when metalink has https url & http(s) proxy is...
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
11
All Linux
low Severity low
: ---
: ---
Assigned To: Seth Vidal
Fedora Extras Quality Assurance
:
: 501851 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-02-07 06:55 EST by Zaid Sheikh
Modified: 2014-01-21 18:08 EST (History)
20 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-08-03 15:14:26 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
File: /etc/yum.repos.d/fedora-rawhide.repo (1.46 KB, text/plain)
2009-02-07 06:55 EST, Zaid Sheikh
no flags Details
yum.conf (864 bytes, text/plain)
2009-02-07 09:48 EST, Zaid Sheikh
no flags Details

  None (edit)
Description Zaid Sheikh 2009-02-07 06:55:45 EST
Created attachment 331192 [details]
File: /etc/yum.repos.d/fedora-rawhide.repo

Description of problem:
Fedora 11 Alpha by default specifies an https url for metalink in fedora-rawhide.repo. However, yum seems not to support https links when an http proxy is specified (Proxy server is https too, ie allows CONNECT method). As a result the following error message is shown:
Error: Cannot retrieve repository metadata (repomd.xml) for repository: rawhide. Please verify its path and try again

Version-Release number of selected component (if applicable):
Tested on yum-3.2.20-5.fc8 and yum-3.2.21-8.fc11.noarch

How reproducible:
always

Steps to Reproduce:
1.Specify http/https proxy in yum in fc11 Alpha default install (I used LiveCD).
2.Do "yum search xyz" etc.
3.Tried same in Fedora 10 after changing http:// to https://. Same error for metalink url. Did the same for baseurl: this time an additional error was generated along with the previous one: [Errno 14] HTTP Error 501: Not Implemented. 

  
Actual results:
Yum error:
Error: Cannot retrieve repository metadata (repomd.xml) for repository: rawhide. Please verify its path and try again

Expected results:
yum shuold have worked.

Additional info:
Problem can be resolved just by changing all https links to http.
Comment 1 James Antill 2009-02-07 09:14:58 EST
What is in your yum.conf and/or set for the *_proxy env. variables?
Comment 2 Zaid Sheikh 2009-02-07 09:48:29 EST
Created attachment 331197 [details]
yum.conf
Comment 3 Zaid Sheikh 2009-02-07 09:49:45 EST
Attached yum.conf. I haven't set any enviroment variables for proxy.
Comment 4 seth vidal 2009-03-03 16:06:28 EST
I'm pretty sure this is related to bug #487101
Comment 5 Fritz Elfert 2009-05-14 09:16:47 EDT
Just my 2 cents:

Regardless of configuring a proxy in either yum.conf or the environment, stracing yum on my current FC11 (preview) reveals, that it opens a connection to the proxy and then issues a

GET https://mirrors.fedoraproject.org/metalink?repo=fedora-11&arch=i386 HTTP/1.1

which is totally wrong for https URLs (needs to issue a CONNECT isntead of a GET, then switch to SSL after CONNECTED response ...). Therefore, the proxy (squid in this case) responds with

HTTP/1.0 400 Bad Request

Reason is:
yum (in yumRepo.py) uses urlgrabber which uses urllib2 which simply doesn't support https over proxies out of the box. See bzrlib's transport/http/_urllib2_wrappers.py for an example how that's done the right way.

Cheers
 -Fritz
Comment 6 Walter Francis 2009-05-28 12:08:10 EDT
Same problem here..  At work we're behind a forced proxy, and I was having trouble figuring out what was going on until I found this bug..  Changing the mirror metalink URI from https to http allows me to update here.

Any more info to provide to get this fixed?  This is on an up-to-date F11 image as of this date.
Comment 7 Walter Francis 2009-05-28 12:41:43 EDT
BTW, even per bug #487101 I made sure that https_proxy (and HTTPS_PROXY) were set, no joy...  Yum will not update unless I change mirrorlist=https:// to http://
Comment 8 seth vidal 2009-06-04 15:18:47 EDT
At the moment I don't have a good answer for you. There are a couple of places where urlgrabber and python in general cannot handle certain kinds of proxies.
Comment 9 Bug Zapper 2009-06-09 07:06:38 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 10 Adam Gibson 2009-06-12 16:55:24 EDT
I was hit by this issue too.

Fritz has the best description of the problem.  When you specify a proxy server in the yum.conf configuration,  It should use the CONNECT method to establish an https connection through a proxy.

Changing the https to http worked around the problem.  https would be better to use from a security standpoint though.

We use the yum.conf settings:
proxy=http://10.x.x.x/3128

(where 10.x.x.x is replaced with the real squid proxy server IP of course in our config)
Comment 11 Simon Andrews 2009-06-17 04:50:56 EDT
I was also hit by this.

As a quick fix would it not be possible to change all https URLs to http when a proxy variable is set?

As a side issue I also found it odd that yum -d 10 didn't spit out any information about the URL it was trying to retrieve, or the proxy it was using before dying with the error about failing to retrieve repomd.xml.  Seeing exactly what it was trying and at what stage it was failing would have made it easier to track down the problem.

This bug title should also be changed to reflect the fact that this is now a bug in the released F11, not just the alpha.
Comment 12 James Antill 2009-06-17 09:48:20 EDT
 This is the bug, and I assume the fix. Looks like 2.6.2 will not be broken. I guess we could backport it. Anyone who is having the problem want to try it?

http://bugs.python.org/issue1424152
Comment 13 Tim Taiwanese Liim 2009-06-25 17:03:15 EDT
Re: Comment #12
Jemes,
I can try it out.  Please show me how.

Seth,
Since the python we have does not handle https proxy well. how about
change *.repo back to http instead of https?  We can move to https
when https proxy in python of F11 is fixed properly.
Comment 14 seth vidal 2009-07-09 11:40:34 EDT
*** Bug 501851 has been marked as a duplicate of this bug. ***
Comment 15 Jochen Wiedmann 2009-07-17 04:36:33 EDT
(In reply to comment #13)

> Since the python we have does not handle https proxy well. how about
> change *.repo back to http instead of https?  We can move to https
> when https proxy in python of F11 is fixed properly.  

Sounds like a good solution. In fact, this worked fairly well for me.
Comment 16 seth vidal 2009-08-03 11:07:07 EDT
If anyone here is running on rawhide-ish and has a proxy that can test something new - please comment on this bug and I'll send you a link to something to test.
Comment 17 Fritz Elfert 2009-08-03 13:13:11 EDT
I can give it a try. (in the meantime i'm on official F11, but setting up a rawhide-ish VM, shouldn't take long)

-Fritz
Comment 18 seth vidal 2009-08-03 13:40:58 EDT
Grab the pkg from:

http://lists.baseurl.org/pipermail/yum/2009-July/022868.html

it is built for rawhide. Test it out. Let me know if anything changes for this problem.

thanks
Comment 19 Fritz Elfert 2009-08-03 14:15:18 EDT
Works like a charm :-)

I even got bold and rebuilt the src.rpm on a regular F11, installed it manually, changed the URLs in fedora.repo back to their original https://.., ran yum clean all and after that a yum update without any problems. Logs at at our proxy show, that yum now correctly uses CONNECT when requesting https URLs.

Nice Job!
 -Fritz
Comment 20 seth vidal 2009-08-03 14:32:51 EDT
Fritz,
 This is excellent news. Thanks,
-sv
Comment 21 seth vidal 2009-08-03 15:14:26 EDT
This package in koji (soon to be in rawhide, I hope)

http://koji.fedoraproject.org/koji/buildinfo?buildID=125399

should solve it - I'm going to close this rawhide. Please reopen if this is still broken.
Comment 22 Simon Andrews 2009-08-04 03:44:24 EDT
Can this fix not be released as an update for F11?  There's a long way to go to EOL and it would be good to not have to manually change the repo definitions every time we install a new machine.
Comment 23 seth vidal 2009-08-04 08:04:39 EDT
it might be able to be backported - but it needs more testing in rawhide first, I think.
It brings in a new dependency (pycurl) so it does make things a bit harder for backporting.
Comment 24 Fritz Elfert 2009-09-16 18:27:55 EDT
Oh well,
while working nicely for a few updates, meanwhile a strange bug has shown up:

When fetching *big* files (> 20Mb), it fails with the following message and yum goes into a retry loop, fetching the same file over and over again from different mirrors ...:

http://ftp.uni-kl.de/pub/linux/fedora/linux/updates/11/i386/kdegames-4.3.1-4.fc11.i586.rpm: [Errno 14] HTTP Error 200 : http://ftp.uni-kl.de/pub/linux/fedora/linux/updates/11/i386/kdegames-4.3.1-4.fc11.i586.rpm
Trying other mirror.

I tried disabling the proxy, but that does not seem to help. Currently, when this happens, i have to interrupt yum and download the relevant file manually with wget into /var/cache/yum/<reponame>/packages, then restart yum update ....

Errno 14 is an EFAULT, so this seems to be some buffer overflow?! ....

-Fritz
PS: Can't reopen this bug, 'cause i'm not the owner ...
Comment 25 seth vidal 2009-09-17 10:01:57 EDT
This is on python-urlgrabber of what version?
Comment 26 Fritz Elfert 2009-09-17 10:53:34 EDT
I'm using python-urlgrabber-3.9.0-3.fc11 (rebuild on F11 of your previously posted src.rpm).

Actually, i just looked into grabber.py and realized, that Errno isn't meant to be a "standard" unix error code (as defined in errno.h), but some generic code of your grabber lib.

So:
Is there any way to convince yum to produce more verbose stack traces so i can find out the real reason?

-Fritz
Comment 27 seth vidal 2009-09-17 12:16:06 EDT
That particular bug is fixed by the newer python-urlgrabber in rawhide.

errno is coming from pycurl and it is about a timeout that is a fixed length, rather than being based on inactivity.

update to urlgrabber 3.9.0-8 and it goes away.
Comment 28 Fritz Elfert 2009-09-17 21:53:35 EDT
Thanks for the hint. Works well now.

Note You need to log in before you can comment on or make changes to this bug.