Bug 138535

[bpm]$ diff -wruN /tmp/urllib2.py urllib2.py
--- /tmp/urllib2.py     2004-11-09 18:48:54.263275984 -0600
+++ urllib2.py  2004-11-09 18:45:54.555595704 -0600
@@ -465,7 +465,7 @@
     handler_order = 1000  # after all other processing

     def http_response(self, request, response):
-        code, msg, hdrs = response.code, response.msg, response.info()
+        code, msg, hdrs = response.status, response.msg, response.info()

         if code not in (200, 206):
             response = self.parent.error(
--END--
works for me.

works nice
thanks!

That's not the right fix for this problem. Fixing it in python 2.4
isn't likely to get into fedora core.

we need to fix it in urlgrabber for python 2.4 - but I think this
backward compat break better be noted as a python bug.

If this is an incompatibility between Python 2.3 and Python 2.4,
please produce a small test-case that doesn't involve yum and submit
it to the Python bug tracker on sourceforge.  If this can't be fixed
for 2.4.0, it may be fixed for 2.4.1.
    http://sourceforge.net/tracker/?group_id=5470&atid=105470

me too - yum crashes:

# yum -y update
Setting up Update Process
Setting up Repo:  development
Traceback (most recent call last):
  File "/usr/bin/yum", line 8, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 68, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 381, in doCommands
    return self.updatePkgs()
  File "/usr/share/yum-cli/cli.py", line 766, in updatePkgs
    self.doRepoSetup()
  File "/usr/share/yum-cli/cli.py", line 70, in doRepoSetup
    repo.getRepoXML()
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 465, in
getRepoXML
    result = self.get(relative=remote, local=local, copy_local=1)
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 443, in get
    checkfunc=checkfunc)
  File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line
414, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line
400, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
595, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
527, in _retry
    return apply(func, (opts,) + args, {})
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
581, in retryfunc
    fo = URLGrabberFileObject(url, filename, opts)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
703, in __init__
    self._do_open()
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
747, in _do_open
    fo, hdr = self._make_request(req, opener)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
823, in _make_request
    fo = opener.open(req)
  File "/usr/lib/python2.4/urllib2.py", line 364, in open
    response = meth(req, response)
  File "/usr/lib/python2.4/urllib2.py", line 468, in http_response
    code, msg, hdrs = response.code, response.msg, response.info()
AttributeError: HTTPResponse instance has no attribute 'code'

Seth, sorry to add to this thread.  I've had to add yet another patch
to a python file to get yum to download and install todays rawhide
updates.  This is given the first patch to urllib2.pl.  This is to
urllib.pl.  I know that this is not a YUM error, but a python bug. 
All I care is that I can get it to work.  It probally is a very bad
HACK.  It might cause cancer in lab rats for all I know.  It does
enable my system to be be bandaged (without to much bleeding) till a
good fix is available.

[bpm]$ diff -wruN /tmp/urllib.py ./urllib.py
--- /tmp/urllib.py      2004-11-11 08:36:35.785508280 -0600
+++ ./urllib.py 2004-11-11 08:27:34.000000000 -0600
@@ -836,6 +836,8 @@
         addbase.__init__(self, fp)
         self.headers = headers
         self.url = url
+        self.status = 200  # Status-Code
+       self.msg = None

     def info(self):
         return self.headers

Was this filed upstream yet? If so, what is the bug number?

Thanks, both changes fixed yum for now.

Thanks, both changes fixed yum for now.

*** Bug 138987 has been marked as a duplicate of this bug. ***

Tested both patches from bryan. This seemed to fix this issue.

This is definitely a bug in Python 2.4. It seems that jhylton applied
a patch (SF number 852995) between revisions 1.56 and 1.57 of
urllib2.py but never bothered to finish the processor implementation.
Give my patch a try and see if it resolves it.

Created attachment 106866 [details]
Patch to Python 2.4b2's urllib2.py

well....
Traceback (most recent call last):
  File "/usr/bin/yum", line 8, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 51, in main
    base.getOptionsConfig(args)
  File "/usr/share/yum-cli/cli.py", line 133, in getOptionsConfig
    self.conf = yumconf(configfile = yumconffile, root=root)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 227, in
__init__
    self._doFileRepo(fn)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 299, in
_doFileRepo
    doRepoSection(self, repoconf, section)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 313, in
doRepoSection
    mirrorurls = getMirrorList(mirrorlist)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 390, in
getMirrorList
    fo = urlresolver.urlopen(url)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
427, in urlopen
    return default_grabber.urlopen(url, **kwargs)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
555, in urlopen
    return self._retry(opts, retryfunc, url)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
527, in _retry
    return apply(func, (opts,) + args, {})
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
554, in retryfunc
    return URLGrabberFileObject(url, filename=None, opts=opts)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
703, in __init__
    self._do_open()
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
747, in _do_open
    fo, hdr = self._make_request(req, opener)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
823, in _make_request
    fo = opener.open(req)
  File "/usr/lib/python2.4/urllib2.py", line 364, in open
    response = meth(req, response)
  File "/usr/lib/python2.4/urllib2.py", line 468, in http_response
    code, msg, hdrs = response.__dict__.get(status, None),
response.msg, response.info()
NameError: global name 'status' is not defined

Crap, my bad...

Created attachment 106867 [details]
FIXED Python 2.4b2 patch

Good.  Works fine.  Does this obsolete the hack to urllib.pl?

Yes, it does. And could someone please tell the Python people that
they need to test not only the success paths, but the failure paths as
well? ;)

Humm..
I still have a traceback (but a new one this time):

<snip>

livna-test: ################################################## 8/8
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for kdepim-devel to pack into transaction set.
Traceback (most recent call last):
  File "/usr/bin/yum", line 8, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 104, in main
    (result, resultmsgs) = base.buildTransaction()
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 174,
in buildTransaction
    (rescode, restring) = self.resolveDeps()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 148,
in resolveDeps
    self.populateTs(test=1)
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 106,
in populateTs
    self.downloadHeader(po)
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 454,
in downloadHeader
    end=end, checkfunc=checkfunc, copy_local=1)
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 443, in get
    checkfunc=checkfunc)
  File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line
414, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line
400, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
595, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
527, in _retry
    return apply(func, (opts,) + args, {})
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
581, in retryfunc
    fo = URLGrabberFileObject(url, filename, opts)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
703, in __init__
    self._do_open()
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
747, in _do_open
    fo, hdr = self._make_request(req, opener)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
823, in _make_request
    fo = opener.open(req)
  File "/usr/lib/python2.4/urllib2.py", line 364, in open
    response = meth(req, response)
  File "/usr/lib/python2.4/urllib2.py", line 469, in http_response
    code, msg, hdrs = response.__dict__.get('status', None),
response.msg, response.info()
AttributeError: addinfourl instance has no attribute 'msg'

Ignacio, if you have a correct patch, submit it along with a test case
that fails before the patch and succeeds after to the Python patch
tracker on sourceforge.net,
http://sourceforge.net/tracker/?group_id=5470&atid=305470

After looking deeper into Python 2.4b2 it seems that my patch isn't
enough to fix all issues. It seems that Python is very, very broken in
its current beta. A downgrade might be a good idea at this point, but
of course it's not my call.

*** Bug 139891 has been marked as a duplicate of this bug. ***

but how to apply this patch? what file is the .diff or patch file? 
please elaborate. don't know about patching, much. 
thanks 

Given the size of the changes, I'd recommend just using vi and
changing the lines manually

code, msg, hdrs = response.__dict__.get('status', None),
response.__dict__.get('msg',None), response.info()

made things "work" for me (before I got the traceback in #20), but
I've not looked at the code enough to know whether this is enough to
fix the problem properly or not. Suppose response.info() could be
undefined too ;)

Doing the patch as suggested fixes things for a yum check-update point
of view but yum update produces the following output :-

[root@xxxxxxxx ~]# yum update
Setting up Update Process
Setting up Repo:  development
repomd.xml                100% |=========================| 1.1 kB    00:00
Reading repository metadata in from local files
developmen: ################################################## 3559/3559
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for esound-devel to pack into transaction set.
Traceback (most recent call last):
  File "/usr/bin/yum", line 8, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 104, in main
    (result, resultmsgs) = base.buildTransaction()
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 174,
in buildTransaction
    (rescode, restring) = self.resolveDeps()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 148,
in resolveDeps
    self.populateTs(test=1)
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 106,
in populateTs
    self.downloadHeader(po)
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 454,
in downloadHeader
    end=end, checkfunc=checkfunc, copy_local=1)
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 443, in get
    checkfunc=checkfunc)
  File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line
414, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line
400, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
595, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
527, in _retry
    return apply(func, (opts,) + args, {})
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
581, in retryfunc
    fo = URLGrabberFileObject(url, filename, opts)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
703, in __init__
    self._do_open()
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
747, in _do_open
    fo, hdr = self._make_request(req, opener)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line
823, in _make_request
    fo = opener.open(req)
  File "/usr/lib/python2.4/urllib2.py", line 364, in open
    response = meth(req, response)
  File "/usr/lib/python2.4/urllib2.py", line 468, in http_response
    code, msg, hdrs = response.__dict__.get('status', None),
response.msg, response.info()
AttributeError: addinfourl instance has no attribute 'msg'


Also after the patch up2date in GUI mode nolonger shows any channel
information and doesn't get any updates it just reports that the
system is up2date when updates are actually available so for now I
have removed the patch and am relying on up2date to get any updates.

David: do the patch in Additional Comment #6 as well to make the
"AttributeError: addinfourl instance has no attribute 'msg'" error go away

The patches at comment #6 and #17 fixed yum for me! :-)

*** Bug 140835 has been marked as a duplicate of this bug. ***

The common problem between urlgrabber.py and python-2.4
is how to attach a readline object to read a response.

urlgrabber has chosen to steal various socket methods and
add them to a sub-classed HTTPResponse.

python-2.4 has chosen a twistier (and apparently Windoze
friendly path instead).

Here's comments from the do_open method in urllib2.py:
        # Pick apart the HTTPResponse object to get the addinfourl
        # object initialized properly.
                                                                     
          
        # Wrap the HTTPResponse object in socket's file object adapter
        # for Windows.  That adapter calls recv(), so delegate recv()
        # to read().  This weird wrapping allows the returned object to
        # have readline() and readlines() methods.
                                                                     
          
        # XXX It might be better to extract the read buffering code
        # out of socket._fileobject() and into a base class.
                                                                     
          
        r.recv = r.read
        fp = socket._fileobject(r)
                                                                     
          
        resp = addinfourl(fp, r.msg, req.get_full_url())
        resp.code = r.status
        resp.msg = r.reason
        return resp

Note that the returned object has changed in python-2.4, urlgrabber
do_open is still returning r, and not returning a addinfourl object.

Patch to do so in a moment. Disclaimer: I have no bloody clue
whether python or urlgrabber is right or wrong, only what I see
when I stare at
    print foo
inserted at various points, and even then I'm not sure what I'm
looking at. But the patch WORKSFORME, and applies to urlgrabber,
not python.

It's a backward compat issue for python. If they're breaking API for
these modules then it shouldn't be happening a in a minor version change.

Patching it in yum is the WRONG PLACE.

Created attachment 107469 [details]
Patch to fox urlgrabber in yum.

Hmmm ... Either "an object that has readline" or "readline method"
is a more accurate term than "readline object". Sorry for the
imprecision, I don't need no steenkin' objects ;-)

And argue with the python guys, not me. All I see is bustage,
urlgrabber is doing one thing and python is doing another.

The patch WORKSFORME, and is a far smaller context with less
risk than hacking python release candidates. jmho, ymmv.

And none of the other hacks here are anywhere near close
to solving the problem. Sure you can variables to objects
until joy, I tried that for a couple hours too.

And thanks for the help. Next time you might want to look
at the patch before screaming WRONG PLACE.

Well done to the folks who made the patches :-) 
 

The new patch for urlgrabber is fixing the issue (on three
fedora-devel boxes) and seems better than the two other patches
(comment #6 and #17) regarding isolation.

Just another workaround that was mentioned on the ml was to use an ftp
repositry instead of http. It doesn't fix the actual problem, but it
sounds to me like the easiest temporary workaround for now, and
requires only a minor configuration change.

here is my output after applying the patch mentioned here:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=138535#c33

patch -p1 keepalive.py < grabber.patch 
missing header for unified diff at line 3 of patch
patching file keepalive.py
and now yum works again for me 
thanks a lot

This patch isn't fixing in the right place but I'm juggling too many
other things at the moment to delve deeper into the python changes. 
Adding jbj's hack to our yum package but leaving this open so that we
can still get python fixed.

I agree with Seth (comment #32) that the bug is in python, but we have
to live with it, so yum has to work around that bug.

But the patch Jeff posted and was added in rawhide (comment #33)
breaks on python <= 2.2 (e.g. Fedora Core 1).

Granted, FC1 is EOLed (and used yum 2.0.x), but for the sake of
upstream yum and Fedora Legacy projects, could that patch be revised
to work for python 2.x and Seth, could that be applied to upstream
yum? Thanks!

the urlgrabber maintainer is working on a fix for this.

You guys keep repeating that this is a Python bug.  Has anyone created
a small standalone script that reproduces the problem (works in 2.3.x,
breaks in 2.4.0), and submitted that bug to the Python bug database at
http://sf.net/projects/python ?

One report refers to the patch http://python.org/sf/852995 but nobody
has added a comment to that bug before now.  I've added one, but since
there's no testcase I didn't have much to say.  At least there is a
link from that SF tracker item to this bugzilla item now.

I'm the author of the Python urllib2 patch.

Thankyou Jeff: if it weren't for your comment, it seems nobody who
helps maintain Python would have learned about this...

If this is an Python standard library API change, I'd like to know
where you found this urllib2 "API" documented! (the details of the
HTTP response API that happens to be used in the implementation of
HTTPHandler & its new friend HTTPErrorProcessor, that is: whoever
wrote keepalive.py made a guess that these implementation details
would not change -- that guess turned out to be incorrect)  It's not
documented, unless I missed it (always possible...), and is an
implementation detail of those classes.

If you want to be immune from such changes, you must instantiate your
own OpenerDirector and .add_handler() for each of an explicit set of
handlers (which is trivial), instead of using the rather annoying
build_opener() utility function; you should also not derive from
HTTPHandler (from a quick glance at keepalive.py, I don't believe you
have any reason to derive from it, and implementation inheritance is
well-known to be fragile like this).  These changes would prevent your
code getting clobbered by base-class or (as in this case) friend class
changes.

Nevertheless. of course, we certainly intended to avoid gratuitous (or
even non-gratuitous!-) breakage of classes reimplementing HTTPHandler,
and .do_open() in particular.  So, I agree that in that sense this is
a Python bug, though I'm not immediately sure what the correct fix is,
if any.  Will submit a Python patch when I think about it again at a
more civilised hour, and post a reference to the Python SF tracker
when I do so.

(Hmm, by "I'm the author of the Python urllib2 patch.", I mean the one
everyone here is complaining about, http://python.org/sf/852995 ,
rather than the fixes for yum that have been posted here)

*** Bug 147931 has been marked as a duplicate of this bug. ***

I'm pretty confident this is fixed and closed upstream.

latest urlgrabber in yum 2.2.0 for FC3 and CVS for rawhide

I've still got this problem in FC4 final version.  

Python 2.4.1.

Traceback is:

Gathering header information file(s) from server(s)
Server: Fedora Core 4 - i386 - Base
retrygrab() failed for:
  http://fedora.redhat.com/releases/fedora-core-4/headers/header.info
  Executing failover method
Traceback (most recent call last):
  File "/usr/bin/yum", line 30, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum/yummain.py", line 233, in main
    clientStuff.get_package_info_from_servers(serverlist, HeaderInfo)
  File "/usr/share/yum/clientStuff.py", line 846, in get_package_info_from_servers
    progress_obj=None)
  File "/usr/share/yum/clientStuff.py", line 1327, in grab
    bandwidth, conf.retries, retrycodes, checkfunc)
  File "/usr/share/yum/urlgrabber.py", line 237, in retrygrab
    progress_obj, throttle, bandwidth)
  File "/usr/share/yum/urlgrabber.py", line 314, in urlgrab
    fo = urllib2.urlopen(url)
  File "/usr/lib/python2.4/urllib2.py", line 130, in urlopen
    return _opener.open(url, data)
  File "/usr/lib/python2.4/urllib2.py", line 364, in open
    response = meth(req, response)
  File "/usr/lib/python2.4/urllib2.py", line 468, in http_response
    code, msg, hdrs = response.code, response.msg, response.info()
AttributeError: HTTPResponse instance has no attribute 'code'

I've still got this problem in FC4 final version.  

Python 2.4.1.

Traceback is:

Gathering header information file(s) from server(s)
Server: Fedora Core 4 - i386 - Base
retrygrab() failed for:
  http://fedora.redhat.com/releases/fedora-core-4/headers/header.info
  Executing failover method
Traceback (most recent call last):
  File "/usr/bin/yum", line 30, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum/yummain.py", line 233, in main
    clientStuff.get_package_info_from_servers(serverlist, HeaderInfo)
  File "/usr/share/yum/clientStuff.py", line 846, in get_package_info_from_servers
    progress_obj=None)
  File "/usr/share/yum/clientStuff.py", line 1327, in grab
    bandwidth, conf.retries, retrycodes, checkfunc)
  File "/usr/share/yum/urlgrabber.py", line 237, in retrygrab
    progress_obj, throttle, bandwidth)
  File "/usr/share/yum/urlgrabber.py", line 314, in urlgrab
    fo = urllib2.urlopen(url)
  File "/usr/lib/python2.4/urllib2.py", line 130, in urlopen
    return _opener.open(url, data)
  File "/usr/lib/python2.4/urllib2.py", line 364, in open
    response = meth(req, response)
  File "/usr/lib/python2.4/urllib2.py", line 468, in http_response
    code, msg, hdrs = response.code, response.msg, response.info()
AttributeError: HTTPResponse instance has no attribute 'code'

What ver of yum are you using?
I ask b/c there's no reason for anything to be attempting to get 
http://fedora.redhat.com/releases/fedora-core-4/headers/header.info

I think you've got an old version of yum for some reason.
yum --version

bash-3.00$ yum --version
2.0.7


I just tried reinstalling yum-2.3.2-7.noarch.rpm and it failed horribly.  This
is probably the cause of the problem.

I did an upgrade from FC2 rather than a clean install.  I'll do a reinstall from
scratch.