Bug 227772 - yum-fastermirror breaks yum with "thread.error: can't start new thread"
yum-fastermirror breaks yum with "thread.error: can't start new thread"
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: yum-utils (Show other bugs)
6
i386 Linux
medium Severity high
: ---
: ---
Assigned To: Seth Vidal
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-02-07 19:13 EST by Wojciech Pilorz
Modified: 2008-02-18 17:28 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-02-18 17:28:51 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
yum messages with fastestmirror enabled, then with disabled (7.00 KB, text/plain)
2007-02-07 19:13 EST, Wojciech Pilorz
no flags Details

  None (edit)
Description Wojciech Pilorz 2007-02-07 19:13:22 EST
Description of problem:

after recent yum update, yum-fastestmirror break yum on my system
yum dies with message
thread.error: can't start new thread

when yum-fastermirror is disabled, yum works well.

Version-Release number of selected component (if applicable):
yum-fastestmirror-1.0.2-2.fc6
yum-3.0.3-1.fc6


How reproducible:
always (on my system)

Steps to Reproduce:
1. enable fastestmirror in file /etc/yum/pluginconf.d/fastestmirror.conf
2. run "yum shell" in bash
3. enter "update" command in yum shell
  
Actual results:
yum dies if fastestmirror is enabled

Cannot use fastestmirror

Expected results:
yum works with yum-fastestmirror enabled


Additional info:
I am including log of yum messages; ulimit and system info included

I started having problems with fastestmirror after yum has been updated to version 
yum.noarch 3.0.3-1.fc6 (Jan 16)
Previous version of yum (that is yum-3.0.1-2.fc6.noarch) worked with
fastestmirror correctly.

So I am not sure, whether the problem is yum or yum-utils...
Comment 1 Wojciech Pilorz 2007-02-07 19:13:23 EST
Created attachment 147618 [details]
yum messages with fastestmirror enabled, then with disabled
Comment 2 Tim Lauridsen 2007-02-08 04:08:10 EST
I dont think this error has anything tp do with yum or the the fastestmirror
plugin. The error happens in the python threading code.
The fastest mirror plugin works by starting a thread for each url in the
mirrorlist, the single thread is timing how long time the connect to the URL
takes. And add it to a list.
When all threads are completted the list is sorted, so the the fastest Url are
first in the list. The list is the written to a cachefile and then returning the
sorted mirrorlist to yum.

The error happens after a while when the plugins try to start a new thread, then
the python threading code make a Traceback because it cant start any more
threads. It looks like there is some kind of limit for the number of concurrent
threads.

Maybe a recent python update have introduced this limit / "Bug".

There can be code some kind of exception handling in the plugin to catch there
kind of errors, but i don't think it is the right thing to do.


Comment 3 Tim Lauridsen 2007-02-08 04:11:01 EST
Luke:

I have added you to the CC, maybe you have some comments to this bug.
Comment 4 Luke Macken 2007-02-09 00:42:29 EST
I am able to reproduce this on FC6 and rawhide.

[...]
301
302
303
Traceback (most recent call last):
  File "foo.py", line 16, in <module>
    y.start()
  File "/usr/lib/python2.5/threading.py", line 434, in start
    _start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread

Are you seriously hitting that limit?  Try setting 'verbose=1' in your
/etc/yum/pluginconf.d/fastestmirror.conf and see if you're really pulling from
that many mirrors.
Comment 5 Luke Macken 2007-02-09 02:16:06 EST
Ah, disregard my suggestion; I didn't see that you already had verbose output
attached to this bug.
Comment 6 Wojciech Pilorz 2007-02-11 17:56:36 EST
The bug appeared after I installed yum version 3.0.3-1.fc6.

Previous yum version I had installed before was 3.0.1-2.fc6, and it did not
exhibit this problem.

I have not changed python version in the meantime (I have python-2.4.4-1.fc6 
installed, and I installed it even before yum version 3.0.1-2 )

While I have reported the problem for yum-fastestmirror-1.0.2-2.fc6,
it was also present with yum-fastestmirror-1.0.1-1.fc6 **after** I updated yum
to 3.0.3. Before yum update fastermirror plugin worked nicely for a few weeks.

This might perhaps suggest the problem is in yum, rather than python.
I had included ulimit -S -a output in the original bug report - I do not believe
there is anything non-standard here.
Comment 7 Wojciech Pilorz 2007-02-20 19:41:48 EST
Tried witn new version of yum and yum-utils - still did not work;
So I run 'yum shell' through strace (this is what I should have done at the very
start) and all became obvious;

Thread creation hit my ulimit -v setting of 425000 - for each thread 10MiB+4KiB
buffer is allocated with mmap, then clone syscall is performed;
What failed was that mmap2 call; 

After increasing address space limit (ulimit -v) to 880000, yum run successfully
once, then again failed.

after
 ulimit -S -v unlimited 
is successfully run, yum works all the time.

I was mislead to believe it is yum version problem because I changed
/etc/security/limits.conf at the same time I upgraded yum.

** However **, with more then 170 mirrors tested for speed, there is no sane
value for ulimit -v; I do not consider unlimited a sane value.

Anyway, requiring user not to have ulimit on address space seems not to be a
very good idea. Perhaps adding a limit on a number of concurrently started
thread, e.g. 20, would be a good idea - it would allow to run fastestmirror with
sane settings (300000 ?) for address space limit.
Comment 8 Wojciech Pilorz 2007-04-11 19:11:48 EDT
For some weeks I run yum with 
ulimit -S -v 999000
and so far the value has always been high enough.
Still I believe that limiting the number of threads running at any given time
would be the proper solution.
Comment 9 Damjan Lango 2007-05-09 08:00:21 EDT
Pilorz: thanks for the work-around, it works with it.
I've seen this problem long time ago, but just disabled fastestmirror, now I
tried it again to see if it has been fixed but still doesn't work...
btw, when using the workaround after fastestmirror goes through all mirrors the
first time, next time the ulimit workaround is not needed as it skips all the
mirrors saying that they have already been timed...
So I agree the proper solution is to limit the number of threads run at the same
time.
Comment 10 Luke Macken 2007-12-03 11:50:40 EST
I added a 'maxthreads' configuration option to fastestmirror, which defaults to 15.
    
http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=commitdiff;h=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e

Please let me know if this solves your problem.  You can get the latest version
here:
http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=blob_plain;f=plugins/fastestmirror/fastestmirror.py;hb=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e
Comment 11 Fedora Update System 2007-12-07 16:30:26 EST
yum-utils-1.1.9-1.fc8 has been pushed to the Fedora 8 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update yum-utils'
Comment 12 Fedora Update System 2007-12-07 16:33:37 EST
yum-utils-1.1.9-1.fc7 has been pushed to the Fedora 7 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update yum-utils'
Comment 13 Wojciech Pilorz 2007-12-10 07:36:46 EST
(In reply to comment #10)
> I added a 'maxthreads' configuration option to fastestmirror, which defaults
to 15.
>     
>
http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=commitdiff;h=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e
> 
> Please let me know if this solves your problem.  You can get the latest version
> here:
>
http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=blob_plain;f=plugins/fastestmirror/fastestmirror.py;hb=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e

I have downloaded the version from link above and put in into
FC6 system;
If I set the maxthreads limit quite low (to 3) I can use yum shell with 
ulimit -S -v 322000
The funny thing is that yum with no shell 
(just yum update) seems to work with much higher value of
maxthreads option.

Thank you for the patch, seems to work for me.

Wojtek
PS. notes for other people willing to test the new version with maxthreads;
1. set ulimit -S -v to some value in range 250000 - 2000000
2. test yum list pattern or yum shell , then command
  like update or list
3. remove /var/cache/yum/timehosts.txt before testing
4. I feel that number of threads that yum wants to create (that is number of
mirrors it wants to try(
depends on whetherthe system is behind proxy/firewall with masquarade or not.
Unfortunately I do not have enough number of configurations or knowledge 
to be reasonably sure it is that way.
Comment 14 Fedora Update System 2007-12-23 17:50:17 EST
yum-utils-1.1.9-1.fc8 has been pushed to the Fedora 8 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update yum-utils'
Comment 15 Fedora Update System 2007-12-23 17:51:01 EST
yum-utils-1.1.9-1.fc7 has been pushed to the Fedora 7 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update yum-utils'
Comment 16 Fedora Update System 2007-12-26 18:51:48 EST
yum-utils-1.1.9-1.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 17 Fedora Update System 2007-12-26 18:52:16 EST
yum-utils-1.1.9-1.fc7 has been pushed to the Fedora 7 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 18 Luke Macken 2008-02-18 17:28:51 EST
Closing, this has been fixed for a while.

Note You need to log in before you can comment on or make changes to this bug.