Red Hat Bugzilla – Bug 227772
yum-fastermirror breaks yum with "thread.error: can't start new thread"
Last modified: 2008-02-18 17:28:51 EST
Description of problem: after recent yum update, yum-fastestmirror break yum on my system yum dies with message thread.error: can't start new thread when yum-fastermirror is disabled, yum works well. Version-Release number of selected component (if applicable): yum-fastestmirror-1.0.2-2.fc6 yum-3.0.3-1.fc6 How reproducible: always (on my system) Steps to Reproduce: 1. enable fastestmirror in file /etc/yum/pluginconf.d/fastestmirror.conf 2. run "yum shell" in bash 3. enter "update" command in yum shell Actual results: yum dies if fastestmirror is enabled Cannot use fastestmirror Expected results: yum works with yum-fastestmirror enabled Additional info: I am including log of yum messages; ulimit and system info included I started having problems with fastestmirror after yum has been updated to version yum.noarch 3.0.3-1.fc6 (Jan 16) Previous version of yum (that is yum-3.0.1-2.fc6.noarch) worked with fastestmirror correctly. So I am not sure, whether the problem is yum or yum-utils...
Created attachment 147618 [details] yum messages with fastestmirror enabled, then with disabled
I dont think this error has anything tp do with yum or the the fastestmirror plugin. The error happens in the python threading code. The fastest mirror plugin works by starting a thread for each url in the mirrorlist, the single thread is timing how long time the connect to the URL takes. And add it to a list. When all threads are completted the list is sorted, so the the fastest Url are first in the list. The list is the written to a cachefile and then returning the sorted mirrorlist to yum. The error happens after a while when the plugins try to start a new thread, then the python threading code make a Traceback because it cant start any more threads. It looks like there is some kind of limit for the number of concurrent threads. Maybe a recent python update have introduced this limit / "Bug". There can be code some kind of exception handling in the plugin to catch there kind of errors, but i don't think it is the right thing to do.
Luke: I have added you to the CC, maybe you have some comments to this bug.
I am able to reproduce this on FC6 and rawhide. [...] 301 302 303 Traceback (most recent call last): File "foo.py", line 16, in <module> y.start() File "/usr/lib/python2.5/threading.py", line 434, in start _start_new_thread(self.__bootstrap, ()) thread.error: can't start new thread Are you seriously hitting that limit? Try setting 'verbose=1' in your /etc/yum/pluginconf.d/fastestmirror.conf and see if you're really pulling from that many mirrors.
Ah, disregard my suggestion; I didn't see that you already had verbose output attached to this bug.
The bug appeared after I installed yum version 3.0.3-1.fc6. Previous yum version I had installed before was 3.0.1-2.fc6, and it did not exhibit this problem. I have not changed python version in the meantime (I have python-2.4.4-1.fc6 installed, and I installed it even before yum version 3.0.1-2 ) While I have reported the problem for yum-fastestmirror-1.0.2-2.fc6, it was also present with yum-fastestmirror-1.0.1-1.fc6 **after** I updated yum to 3.0.3. Before yum update fastermirror plugin worked nicely for a few weeks. This might perhaps suggest the problem is in yum, rather than python. I had included ulimit -S -a output in the original bug report - I do not believe there is anything non-standard here.
Tried witn new version of yum and yum-utils - still did not work; So I run 'yum shell' through strace (this is what I should have done at the very start) and all became obvious; Thread creation hit my ulimit -v setting of 425000 - for each thread 10MiB+4KiB buffer is allocated with mmap, then clone syscall is performed; What failed was that mmap2 call; After increasing address space limit (ulimit -v) to 880000, yum run successfully once, then again failed. after ulimit -S -v unlimited is successfully run, yum works all the time. I was mislead to believe it is yum version problem because I changed /etc/security/limits.conf at the same time I upgraded yum. ** However **, with more then 170 mirrors tested for speed, there is no sane value for ulimit -v; I do not consider unlimited a sane value. Anyway, requiring user not to have ulimit on address space seems not to be a very good idea. Perhaps adding a limit on a number of concurrently started thread, e.g. 20, would be a good idea - it would allow to run fastestmirror with sane settings (300000 ?) for address space limit.
For some weeks I run yum with ulimit -S -v 999000 and so far the value has always been high enough. Still I believe that limiting the number of threads running at any given time would be the proper solution.
Pilorz: thanks for the work-around, it works with it. I've seen this problem long time ago, but just disabled fastestmirror, now I tried it again to see if it has been fixed but still doesn't work... btw, when using the workaround after fastestmirror goes through all mirrors the first time, next time the ulimit workaround is not needed as it skips all the mirrors saying that they have already been timed... So I agree the proper solution is to limit the number of threads run at the same time.
I added a 'maxthreads' configuration option to fastestmirror, which defaults to 15. http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=commitdiff;h=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e Please let me know if this solves your problem. You can get the latest version here: http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=blob_plain;f=plugins/fastestmirror/fastestmirror.py;hb=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e
yum-utils-1.1.9-1.fc8 has been pushed to the Fedora 8 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update yum-utils'
yum-utils-1.1.9-1.fc7 has been pushed to the Fedora 7 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update yum-utils'
(In reply to comment #10) > I added a 'maxthreads' configuration option to fastestmirror, which defaults to 15. > > http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=commitdiff;h=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e > > Please let me know if this solves your problem. You can get the latest version > here: > http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=blob_plain;f=plugins/fastestmirror/fastestmirror.py;hb=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e I have downloaded the version from link above and put in into FC6 system; If I set the maxthreads limit quite low (to 3) I can use yum shell with ulimit -S -v 322000 The funny thing is that yum with no shell (just yum update) seems to work with much higher value of maxthreads option. Thank you for the patch, seems to work for me. Wojtek PS. notes for other people willing to test the new version with maxthreads; 1. set ulimit -S -v to some value in range 250000 - 2000000 2. test yum list pattern or yum shell , then command like update or list 3. remove /var/cache/yum/timehosts.txt before testing 4. I feel that number of threads that yum wants to create (that is number of mirrors it wants to try( depends on whetherthe system is behind proxy/firewall with masquarade or not. Unfortunately I do not have enough number of configurations or knowledge to be reasonably sure it is that way.
yum-utils-1.1.9-1.fc8 has been pushed to the Fedora 8 stable repository. If problems still persist, please make note of it in this bug report.
yum-utils-1.1.9-1.fc7 has been pushed to the Fedora 7 stable repository. If problems still persist, please make note of it in this bug report.
Closing, this has been fixed for a while.