This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 542224 - libproxy support for yum/urlgrabber (could add PAC / socks / desktop support)
libproxy support for yum/urlgrabber (could add PAC / socks / desktop support)
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: python-urlgrabber (Show other bugs)
19
All Linux
low Severity medium
: ---
: ---
Assigned To: packaging-team-maint
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks: 857263
  Show dependency treegraph
 
Reported: 2009-11-28 17:56 EST by Dirk Hohndel
Modified: 2015-02-18 10:33 EST (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 857263 (view as bug list)
Environment:
Last Closed: 2015-02-18 07:00:19 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
libproxy patch (1.22 KB, patch)
2009-12-24 17:19 EST, David Woodhouse
no flags Details | Diff
urlgrabber patch (1.57 KB, patch)
2009-12-24 18:41 EST, David Woodhouse
no flags Details | Diff
yum patch (2.72 KB, patch)
2009-12-24 18:45 EST, David Woodhouse
no flags Details | Diff
urlgrabber patch to use libproxy by default (2.20 KB, patch)
2009-12-26 05:12 EST, David Woodhouse
no flags Details | Diff
Updated patch to use only http proxies (2.38 KB, patch)
2010-01-12 07:11 EST, David Woodhouse
no flags Details | Diff
libproxy support patch for urlgrabber 3.9.1 (3.16 KB, patch)
2011-03-14 08:13 EDT, Yan Li
no flags Details | Diff

  None (edit)
Description Dirk Hohndel 2009-11-28 17:56:13 EST
Description of problem:
When setting up system wide proxy with Network Proxy Settings tool, that proxy is not honored by yum (so yum requires a manual entry in /etc/yum.conf

Version-Release number of selected component (if applicable):
12

How reproducible:
Change proxy settings from Gnome session (System->Preferences->Network Proxy) and try to run yum while behind a firewall that blocks http accesses

Steps to Reproduce:
1. change proxy settings
2. start yum
3. watch it hang
  
Actual results:
hangs

Expected results:
works

Additional info:
Comment 1 seth vidal 2009-11-30 15:35:25 EST
yum honors the systemwide setting in the environment variables - but sense I do not where gnome writes this data I have no idea where to get that.

However, the systemwide setting needs to be written to the http_proxy environment variable. If it is not then the bug is in the gnome configuration app.

Another possibility is that the setting app is setting the environment variable but you need to restart X for that to take effect.

closing this notabug. alternatively it could be reassigned to whatever app is setting this variable.

thanks
Comment 2 David Woodhouse 2009-12-24 17:19:36 EST
Created attachment 380247 [details]
libproxy patch

(In reply to comment #1)
> yum honors the systemwide setting in the environment variables - but sense I do
> not where gnome writes this data I have no idea where to get that.

And you don't need to know. Just use libproxy, which will do it all for you -- including giving the correct results when there is a proxy autoconfig file which returns different results according to the URL being fetched. The naïve $http_proxy approach can't cope with that.

This patch works, but isn't ideal -- we should actually be instantiating a _single_ global ProxyFactory, and using it anew for _each_ URL rather than assuming that all URLs in the same repo will use the same proxy result. But that's not _such_ an unreasonable assumption, and this is at least a step in the right direction...
Comment 3 David Woodhouse 2009-12-24 18:41:32 EST
Created attachment 380262 [details]
urlgrabber patch

This adds optional libproxy support to urlgrabber. In fact you can pass it _anything_ with a getProxies() method which takes an URL and returns a list of proxies; it doesn't have to be libproxy at all. Urlgrabber will use the first proxy and ignore any others.
Comment 4 David Woodhouse 2009-12-24 18:45:01 EST
Created attachment 380263 [details]
yum patch

And this is a nicer yum patch which uses the newly-added urlgrabber support, instantiating a single ProxyFactory and using it individually for each URL.
Comment 5 seth vidal 2009-12-25 00:38:17 EST
why not get this as a patch into urlgrabber or better yet curl?
Comment 6 David Woodhouse 2009-12-25 18:24:52 EST
I already put some of it into urlgrabber -- the patches in comment #3 and comment #4 go together. There are two things we get from not doing it _entirely_ behind yum's back:

Firstly, we get a single global ProxyFactory rather than instantiating lots of them. Someone who actually knows some python could probably fix that.

Secondly, yum gets to override proxy behaviour according to its own per-repo configuration. I was half expecting that you'd want to make the libproxy thing only happen if 'proxy=auto' is set or something like that (even though the $http_proxy behaviour wasn't configurable).
Comment 7 David Woodhouse 2009-12-26 05:12:40 EST
Created attachment 380415 [details]
urlgrabber patch to use libproxy by default

OK, here's a version which puts it entirely into urlgrabber. I've made some attempt to ensure that we get a global ProxyFactory which is used for all connections (so the PAC file is only fetched once, etc.), and it seems to be working as expected. Someone with a bit more clue should probably look at that though.

No changes to yum are needed for this, and the existing proxy configuration should still work -- urlgrabber will still use the 'proxies' dictionary by preference, and will fall back to using libproxy.

The user can still disable libproxy by setting the 'proxy_factory' to None instead of its default value. Or by setting it to anything else with a getProxies() method, in fact.
Comment 8 Gerhardus Geldenhuis 2009-12-30 12:20:51 EST
I believe gnome uses http://projects.gnome.org/gconf/ for storage of config settings. Whether gconf sets env variables is a different matter.
Comment 9 David Woodhouse 2010-01-11 20:23:50 EST
ping
Comment 10 James Antill 2010-01-11 23:56:38 EST
+            proxies = opts.proxy_factory.getProxies(self.url);
+            if len(proxies) > 0 and proxies[0] != 'direct://':
+                self.curl_obj.setopt(pycurl.PROXY, proxies[0])

 Do we know the return value here will always be valid to pass to curl?

 Also I'm 95% sure the above doesn't work for a mirrorlist (which is to say it'll only call .getProxies() for the first URL).
Comment 11 David Woodhouse 2010-01-12 07:08:58 EST
The return value will be a socks:// or http:// URL. I thought that cURL would accept the former, but it seems not. I'll make it loop through the proxies[] array looking for http.

As for mirrorlists, I think it's working as expected. If I add some debugging, I see this:

proxies for http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-updates-released-12&arch=x86_64: 
http://twosheds.infradead.org:3128
proxies for http://ftp.upjs.sk/pub/mirrors/rpmfusion.org/free/fedora/updates/12/x86_64/repodata/repomd.xml: 
http://twosheds.infradead.org:3128
rpmfusion-free-updates                                   | 3.3 kB     00:00
Comment 12 David Woodhouse 2010-01-12 07:11:40 EST
Created attachment 383228 [details]
Updated patch to use only http proxies

$ interdiff -p1 ~/urlgrabber-libproxy-default.patch ~/urlgrabber-libproxy-httponly.patch 
diff -u b/urlgrabber/grabber.py b/urlgrabber/grabber.py
--- b/urlgrabber/grabber.py
+++ b/urlgrabber/grabber.py
@@ -1243,8 +1243,12 @@
                         self.curl_obj.setopt(pycurl.PROXY, proxy)
         elif opts.proxy_factory:
             proxies = opts.proxy_factory.getProxies(self.url);
-            if len(proxies) > 0 and proxies[0] != 'direct://':
-                self.curl_obj.setopt(pycurl.PROXY, proxies[0])
+            for proxy in proxies:
+                if proxy.startswith('http://'):
+                    if DEBUG: DEBUG.info('using proxy "%s" for url %s' % \
+                                 (proxy, self.url))
+                    self.curl_obj.setopt(pycurl.PROXY, proxy)
+                    break
             
         # FIXME username/password/auth settings
Comment 13 David Woodhouse 2010-05-19 17:49:53 EDT
Did this fix get into F-13?
Comment 14 James Antill 2010-05-20 09:20:47 EDT
No the urlgrabber in F-13 is basically the one from before this bug was opened. We can probably get it in rawhide soon though, and maybe do an update at some point.
Comment 15 Felipe Contreras 2010-06-16 20:21:51 EDT
Ah, this is much needed.
Comment 16 Andreas Thienemann 2010-09-03 07:26:03 EDT
I have been testing this on FC13 and while it works nicely for all PackageKit related tasks and makes them work behind proxies, it seems to break mock:

Exception AttributeError: "'NoneType' object has no attribute 'px_proxy_factory_free'" in <bound method ProxyFactory.__del__ of <libproxy.ProxyFactory object at 0x20c7850>> ignored

All the yum calls mock make do break.
Comment 17 Andreas Thienemann 2010-09-03 07:45:04 EDT
Interesting. I fiddled around a bit. Seems that after wiping some caches yum does work but the AttributeError is still there...
Comment 18 Bug Zapper 2010-11-04 01:03:29 EDT
This message is a reminder that Fedora 12 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 12.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '12'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 12's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 12 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 19 Felipe Contreras 2010-11-04 06:56:59 EDT
Same on F13.
Comment 20 David Woodhouse 2011-01-06 20:55:37 EST
(In reply to comment #16)
> Exception AttributeError: "'NoneType' object has no attribute
> 'px_proxy_factory_free'" in <bound method ProxyFactory.__del__ of
> <libproxy.ProxyFactory object at 0x20c7850>> ignored

That's going to be a bug in the libproxy python bindings, I think. The px_proxy_factory_free() function got dropped from the library, I think, and the python bindings should now just be calling free(). Have you managed to get mismatching versions of libproxy and libproxy-python installed in the chroot?

Is it still happening?
Comment 21 Yan Li 2011-03-14 08:13:59 EDT
Created attachment 484157 [details]
libproxy support patch for urlgrabber 3.9.1

Improved patch (based on David Woodhouse's patch): better handling of exceptions  thrown from libproxy. When exceptions were caught, we'll just fall back to next proxy setting mechanism (environment variables here).
Comment 22 Felipe Contreras 2012-02-02 14:34:41 EST
Right. This is never going to happen.
Comment 23 Stephen John Smoogen 2012-09-13 19:48:21 EDT
Problem is still in F17. Need to find a solution and get the correct packager assigned to it.
Comment 24 Fedora End Of Life 2013-04-03 16:16:04 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
Comment 25 Alexander Kahl 2013-06-01 07:04:42 EDT
Having to modify Yum's proxy settings each time a laptop changes between proxy requiring and non-requiring network environments is extremely annoying, so what stands in the path of accepting the patches above, submitted long ago? I really think Fedora's usability would benefit from libproxy support in Yum.
Comment 26 Gonzalo Odiard 2014-06-10 10:22:10 EDT
This issue is affecting us too. Any chance to get the patches integrated?
Comment 27 David Woodhouse 2014-06-10 10:40:00 EDT
I think that setting 'proxy=libproxy' in yum.conf should get the desired behaviour.

It's not enabled by default, and eventually it *should* be — that default should become part of the packaging guidelines for Fedora. But we have some real work to do before then; getting PacRunner integrated with NetworkManager so that it's always giving the right answers, ensuring that libproxy-pacrunner is present or pacrunner's own replacement for libproxy, and generally making sure that libproxy *is* the right thing to do in all cases.

In the meantime, having it present but disabled by default isn't such a bad thing.
Comment 28 Fedora End Of Life 2015-01-09 16:41:04 EST
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 29 Fedora End Of Life 2015-02-18 07:00:19 EST
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 30 Fedora End Of Life 2015-02-18 10:33:35 EST
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.