Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: If there's a curl error during the server connection / file download, urlgrabber prints an error message as: $ urlgrabber http://no.such.host [Errno 14] PYCURL ERROR 6 - "" 6 is CURLE_COULDNT_RESOLVE_HOST libcurl error code, but there's no matching error message to give human-readable failure reason. It seems that pycurl makes that error text available with error code in the exception. After changing basicfirst.py example from python-pycurl to match urlgrabber code: c.setopt(c.URL, 'http://no.such.host') c.setopt(c.WRITEFUNCTION, t.body_callback) try: c.perform() except pycurl.error, e: print repr(e) I get an error message too: error(6, 'Could not resolve host: no.such.host; Cannot allocate memory') Note: 'Cannot allocate memory' here is confusing, but that's what curl itself reports too: $ curl http://no.such.host curl: (6) Could not resolve host: no.such.host; Cannot allocate memory In case of urlgrabber, the message seems to get lost somewhere though. Version-Release number of selected component (if applicable): python-urlgrabber-3.9.1-7.fc13 python-pycurl-7.19.0-7.fc13 libcurl-7.20.1-4.fc13
This is mostly fixed upstream in urlgrabber. However, the real bug is that pycurl doesn't DO anything useful here at all.
The upstream fix is to use hard-coded list of error messages? Has anyone managed to figure out why urlgrabber does not see the messages it should? Fixed messages seems to give less info than what curl can provide, e.g.: $ PYTHONPATH=. urlgrabber https://redhat.com [Errno 14] curl#51 - "Peer certificate failed verification" $ python ../basicfirst.py Testing libcurl/7.20.1 NSS/3.12.6.2 zlib/1.2.3 libidn/1.16 libssh2/1.2.4 error(51, "SSL: certificate subject name 'www.redhat.com' does not match target host name 'redhat.com'")
that's interesting. Did something change here in pycurl in f13? B/c in the past the second arg of of the error messages has been: '' which is not-so-helpful to print out.
It must still be '' by the time urlgrabber sees it as we do: errstr = str(e.args[1]) if not errstr: errstr = pyerr2str.get(errcode, '<Unknown>') msg = 'curl#%s - "%s"' % (errcode, errstr) ...so we only use our internal messages if the errstr is ''.
looking into this more - urlgrabber is STILL only getting the error code and not the error message. So there is _something else_ happening here. re-opening it.
If I do: diff --git a/urlgrabber/grabber.py b/urlgrabber/grabber.py index 65e0720..a56aa5d 100644 --- a/urlgrabber/grabber.py +++ b/urlgrabber/grabber.py @@ -1447,8 +1447,8 @@ class PyCurlFileObject(object): raise err def _do_open(self): - self.curl_obj = _curl_cache - self.curl_obj.reset() # reset all old settings away, just in case + self.curl_obj = pycurl.Curl() + # self.curl_obj.reset() # reset all old settings away, just in case # setup any ranges self._set_opts() self._do_grab() ...then I get: % urlgrabber https://redhat.com [Errno 14] curl#51 - "SSL: certificate subject name 'www.redhat.com' does not match target host name 'redhat.com'"
The problem appears to be that calling .reset() on a curl object destroys all error information (wtf!) ... so seems like a pycurl bug.
and before anyone asks - if we don't do reset() we end up using whatever settings got left over from the last use. Which is not-so-good.
Created attachment 472056 [details] Proposed patch A curl object (handle) loses all options when it's passed to curl_easy_reset() -- including the pointer to error buffer, which was set in do_curl_new(): curl_easy_setopt(self->handle, CURLOPT_ERRORBUFFER, self->error); It seems that the right thing is to reconfigure the curl object after reset like it is done during its initialization in do_curl_new(). urlgrabber displays the error message with this patch applied: $ urlgrabber http://no.such.host [Errno 14] PYCURL ERROR 6 - "Could not resolve host: no.such.host; Cannot allocate memory" The patch applies cleanly on Rawhide version of pycurl, I haven't tested F-13.
This message is a reminder that Fedora 13 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '13'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 13's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 13 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
$ rpm -q python-urlgrabber python-urlgrabber-3.9.1-7.fc14.noarch $ urlgrabber http://no.such.host [Errno 14] PYCURL ERROR 6 - "" $ urlgrabber https://redhat.com [Errno 14] PYCURL ERROR 51 - ""
The attached patch solves the issue. Jeffrey, could you consider applying it, please?
This message is a notice that Fedora 14 is now at end of life. Fedora has stopped maintaining and issuing updates for Fedora 14. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At this time, all open bugs with a Fedora 'version' of '14' have been closed as WONTFIX. (Please note: Our normal process is to give advanced warning of this occurring, but we forgot to do that. A thousand apologies.) Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, feel free to reopen this bug and simply change the 'version' to a later Fedora version. Bug Reporter: Thank you for reporting this issue and we are sorry that we were unable to fix it before Fedora 14 reached end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged to click on "Clone This Bug" (top right of this page) and open it against that version of Fedora. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping