Bug 758446

Using yum on command line:

rocessing delta metadata
Download delta size: 31 M
http://mirror.uv.es/mirror/fedora/linux/updates/16/i386/drpms/firefox-7.0.1-3.fc16_8.0-3.fc16.i686.drpm: [Errno 14] Downloaded more than max size for http://mirror.uv.es/mirror/fedora/linux/updates/16/i386/drpms/firefox-7.0.1-3.fc16_8.0-3.fc16.i686.drpm: 2597452 > 4320605
Trying other mirror.
Traceback (most recent call last):=========-  ] 233 kB/s | 3.3 MB     00:03 ETA 
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1689, in _progress_update
    if self._over_max_size(cur=self._amount_read-self._reget_length):
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1709, in _over_max_size
    if cur > int(float(max_size) * 1.10):
ValueError: cannot convert float NaN to integer

 Current download cancelled, interrupt (ctrl-c) again within two seconds
to exit.

[ 9370.004413] yum[4844]: segfault at 5f5f6f72 ip 4423223f sp bfd79cc0 error 4 in libpython2.7.so.1.0[441c5000+15e000]

Hello!

Is this reproducible?  Can you check that there was no digit skipped at the beginning of '2597452 > 4320605'?  There should have been no newline between the URL and the 'cur' and 'max_size' numbers, as the format string is "Downloaded more than max size for %s: %s > %s".  

I wonder if there was a copy-paste error.  Unless there's something broken in Python, this could never have been printed:

>>> cur=2597452
>>> max_size=4320605
>>> cur > int(float(max_size) * 1.10)
False

Total download size: 15 M
Is this ok [y/N]: y
Downloading Packages:
http://ftp.cica.es/fedora/linux/releases/16/Everything/i386/os/Packages/glibc-2.14.90-14.i686.rpm: [Errno 14] Downloaded more than max size for http://ftp.cica.es/fedora/linux/releases/16/Everything/i386/os/Packages/glibc-2.14.90-14.i686.rpm: 1166867 > 4272653
Trying other mirror.
Traceback (most recent call last):%) 61% [============        ] 935 kB/s | 2.5 MB     00:01 ETA 
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1689, in _progress_update
    if self._over_max_size(cur=self._amount_read-self._reget_length):
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1709, in _over_max_size
    if cur > int(float(max_size) * 1.10):
ValueError: cannot convert float NaN to integer

 Current download cancelled, interrupt (ctrl-c) again within two seconds
to exit.

(1/2): glibc-2.14.90-14.i686.rpm                                         | 4.1 MB     00:01     
http://ftp.udl.es/pub/fedora/linux/releases/16/Everything/i386/os/Packages/glibc-2.14.90-14.i686.rpm: [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=fedora clean metadata
Trying other mirror.
Traceback (most recent call last):%) 80% [================    ] 1.3 MB/s | 3.3 MB     00:00 ETA 
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1689, in _progress_update
    if self._over_max_size(cur=self._amount_read-self._reget_length):
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1709, in _over_max_size
    if cur > int(float(max_size) * 1.10):
ValueError: cannot convert float NaN to integer
(1/2): glibc-2.14.90-14.i686.rpm                                         | 4.1 MB     00:01     
http://mirror2.hs-esslingen.de/fedora/linux/releases/16/Everything/i386/os/Packages/glibc-2.14.90-14.i686.rpm: [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=fedora clean metadata
Trying other mirror.
Traceback (most recent call last):%) 22% [====                ] 1.2 MB/s | 926 kB     00:02 ETA 
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1689, in _progress_update
    if self._over_max_size(cur=self._amount_read-self._reget_length):
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1709, in _over_max_size
    if cur > int(float(max_size) * 1.10):
ValueError: cannot convert float NaN to integer
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1689, in _progress_update
    if self._over_max_size(cur=self._amount_read-self._reget_length):
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1709, in _over_max_size
    if cur > int(float(max_size) * 1.10):
ValueError: cannot convert float NaN to integer


Exiting on user cancel

I too am having this most bizarre problem. I have some additional information to add:
 1. Everything works fine when I am plugging in via the ethernet port.
 2. The exception is reported every single time that I am using the wireless
 3. The behavior can be reproduced without yum by using urlgrabber directly (/usr/bin/urlgrabber http://66.254.113.98/fedora/linux/releases/15/Everything/i386/os/repodata/fd30a38d6f21677ae397428c0b8088a1f00a77a5ac11859e8eb55ba3bc42430d-filelists.sqlite.bz2) and get the same error.
 4. I used pdb to step through the code and set breakpoints in grabber.py: PyCurlFileObject._retrieve, PyCurlFileObject._hdr_retrieve, PyCurlFileObject._progress_update and PyCurlFileObject._over_max_size and what I print out defies all reason: max_size in _over_max_size() always seems to be an integer that is less than self.size, but float(max_size)*1.10 produces the ValueError. Changing that to something like max_size+(max_size>>3) to approximate the 10% over limit bypasses the strange ValueError. (I tried to break the calculation down and found the exception being thrown when float(max_size) is done. But, print max_size works fine and print type(max_size) says int.) The file is downloaded properly and seems to not be corrupted. However, if I now run yum, the problem just moves elsewhere: 

  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1695, in _progress_update
    rc=self.___progress_update(download_total, downloaded, upload_total, uploaded)
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1706, in ___progress_update
    self.opts.progress_obj.update(downloaded)
  File "/usr/share/yum-cli/output.py", line 70, in update
    TextMeter.update(self, amount_read, now)
  File "/usr/lib/python2.7/site-packages/urlgrabber/progress.py", line 141, in update
    self._do_update(amount_read, now)
  File "/usr/lib/python2.7/site-packages/urlgrabber/progress.py", line 246, in _do_update
    frtime = format_time(rtime)
  File "/usr/lib/python2.7/site-packages/urlgrabber/progress.py", line 664, in format_time
    seconds = int(seconds)
ValueError: cannot convert float NaN to integer

  5. Is this somehow related to some other package -- numpy or something?

I am very willing to run any tests needed -- really curious on how this is happening...

A correction: 
 The ValueError is caused in converting float->int.
I unwrapped the conditional like so:
i1 = int(cur)
f1 = int(max_size)
f1 = f1*1.10
i2 = int(f1) #==>ValueError

This is very, very weird.  Not only am I unable to reproduce any of this but also can't understand what possibly could introduce NaNs at two unrelated places.  I've installed numpy but Python still works the same as before:

$ python
>>> 0.0/0.0
ZeroDivisionError: float division by zero
>>> float(None)
TypeError: float() argument must be a string or a number
>>> 1e400-1e400
nan
>>> 1e400/1e400
nan

Although pycurl calls the progress callback way too often, all urlgrabber does there is a simple rate estimation and a progress meter.  In a very specific cases it can try to divide by zero, but no way we could get 'inf' or 'nan' there..

It's very unlikely that your Python behaves differently, but just in case- could you try the above?  Also, printing the integer value of f1 before that float multiply (that produces nan) would be great!

I am not familiar with "urlgrabber" specifically, but I recently ran into a somewhat similar bug on an Ubuntu Precise system, and figured I'd post my findings here in case they helped someone track down the exact problem.

In short, it seems that somehow the python-pycurl module causes unepected/incorrect NaNs to appear:
=================================
$ python
Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pycurl
>>> def geturl(url):
...   curl=pycurl.Curl()
...   curl.setopt(pycurl.URL,url)
...   curl.perform()
...   print
...   print "before:", 1.0/2, 1.0/2
...   curl.close()
...   print "after:", 1.0/2, 1.0/2
... 
>>> 
>>> geturl("https://launchpad.net/api/")
Object: <lp.services.webapp.servers.WebServiceClientRequest instance URL=https://launchpad.net>, name: ''
before: 0.5 0.5
after: nan 0.5
=================================

Specifically, the first floating point operation after the call to curl.close() always seems to return a NaN, though after that first operation floating point math appears to work normally again. 

However, in my tests this NaN error only happens with https: URLs, not http: ones, so I'm not sure if it's actually the same bug as what's being reported above for urlgrabber.

So far, I've only noticed this on a few i686 systems.  Several other systems don't show the error at all, but on the systems that do show it it occurs consistently.

> However, in my tests this NaN error only happens with https: URLs, not http: ones

Might be a bug in crypto libraries...

http://en.wikipedia.org/wiki/SSE2

.. In such cases, the corrupt floating-point state caused by failure to emit emms may go undetected for millions of instructions before ultimately causing the floating-point routine to fail, returning NaN. Since the problem is not locally apparent in the MMX code, the bug can be very time consuming to find and correct. As SSE2 does not have this problem, usually provides much better throughput and provides more registers in 64-bit code, it should be preferred for nearly all vectorization work.

> So far, I've only noticed this on a few i686 systems.

Possibly ones with MMX but without SSE2?
$ grep flags /proc/cpuinfo

There's some assembly sse code nss library, but that's correct.  Might be this gcc bug: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44578

(In reply to comment #9)
> > So far, I've only noticed this on a few i686 systems.
> 
> Possibly ones with MMX but without SSE2?
> $ grep flags /proc/cpuinfo

I did a survey of our machines, and can confirm that all the systems that show this problem are in the with-MMX-but-without-SSE2 category.

However, I also found machines in that category but not showing the problem, so it would seem there are other factors at play.  Based on the machines I am able to look at, it appears the problem might be limited to Ubuntu Precise; I don't see it either Ubuntu Lucid or Debian Etch.  (None of our machines old enough to be missing SSE2 have anything newer than Precise installed on them.)appears

Please provide the output of curl --version on the machine that shows the problem.

(In reply to comment #12)
> Please provide the output of curl --version on the machine that shows the
> problem.

This might be getting off-topic for this Redhat bug, but for what it's worth 
here's what I get.... (Hopefully one of the original bug reporters can also 
provide this info.)

# curl --version
curl 7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtmp rtsp smtp smtps telnet tftp 
Features: GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP 


Additional information on the libraries involved:
# ldd /usr/lib/python2.7/dist-packages/pycurl.so | grep tls
        libcurl-gnutls.so.4 => /usr/lib/i386-linux-gnu/libcurl-gnutls.so.4 (0x001ec000)
        libgnutls.so.26 => /usr/lib/i386-linux-gnu/libgnutls.so.26 (0x0043b000)

Package versions:
python-pycurl 7.19.0-4ubuntu3
libcurl3-gnutils 7.22.0-3ubuntu4
libgnutls26 2.12.14-5ubuntu3.2

(In reply to comment #13)
Your libcurl does not use nss, so this is probably assigned to a wrong component (unless both nss and gnutls contain the same bug independently of each other).

> Package versions:
> python-pycurl 7.19.0-4ubuntu3
> libcurl3-gnutils 7.22.0-3ubuntu4
> libgnutls26 2.12.14-5ubuntu3.2

The reproducer from comment #8 does not expose the problem on my Ubuntu i686 VM with the above versions of packages and sse* flags explicitly disabled.

This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.