Bug 227772 - yum-fastermirror breaks yum with "thread.error: can't start new thread"
Summary: yum-fastermirror breaks yum with "thread.error: can't start new thread"
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: yum-utils
Version: 6
Hardware: i386
OS: Linux
medium
high
Target Milestone: ---
Assignee: Seth Vidal
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2007-02-08 00:13 UTC by Wojciech Pilorz
Modified: 2008-02-18 22:28 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2008-02-18 22:28:51 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
yum messages with fastestmirror enabled, then with disabled (7.00 KB, text/plain)
2007-02-08 00:13 UTC, Wojciech Pilorz
no flags Details

Description Wojciech Pilorz 2007-02-08 00:13:22 UTC
Description of problem:

after recent yum update, yum-fastestmirror break yum on my system
yum dies with message
thread.error: can't start new thread

when yum-fastermirror is disabled, yum works well.

Version-Release number of selected component (if applicable):
yum-fastestmirror-1.0.2-2.fc6
yum-3.0.3-1.fc6


How reproducible:
always (on my system)

Steps to Reproduce:
1. enable fastestmirror in file /etc/yum/pluginconf.d/fastestmirror.conf
2. run "yum shell" in bash
3. enter "update" command in yum shell
  
Actual results:
yum dies if fastestmirror is enabled

Cannot use fastestmirror

Expected results:
yum works with yum-fastestmirror enabled


Additional info:
I am including log of yum messages; ulimit and system info included

I started having problems with fastestmirror after yum has been updated to version 
yum.noarch 3.0.3-1.fc6 (Jan 16)
Previous version of yum (that is yum-3.0.1-2.fc6.noarch) worked with
fastestmirror correctly.

So I am not sure, whether the problem is yum or yum-utils...

Comment 1 Wojciech Pilorz 2007-02-08 00:13:23 UTC
Created attachment 147618 [details]
yum messages with fastestmirror enabled, then with disabled

Comment 2 Tim Lauridsen 2007-02-08 09:08:10 UTC
I dont think this error has anything tp do with yum or the the fastestmirror
plugin. The error happens in the python threading code.
The fastest mirror plugin works by starting a thread for each url in the
mirrorlist, the single thread is timing how long time the connect to the URL
takes. And add it to a list.
When all threads are completted the list is sorted, so the the fastest Url are
first in the list. The list is the written to a cachefile and then returning the
sorted mirrorlist to yum.

The error happens after a while when the plugins try to start a new thread, then
the python threading code make a Traceback because it cant start any more
threads. It looks like there is some kind of limit for the number of concurrent
threads.

Maybe a recent python update have introduced this limit / "Bug".

There can be code some kind of exception handling in the plugin to catch there
kind of errors, but i don't think it is the right thing to do.




Comment 3 Tim Lauridsen 2007-02-08 09:11:01 UTC
Luke:

I have added you to the CC, maybe you have some comments to this bug.

Comment 4 Luke Macken 2007-02-09 05:42:29 UTC
I am able to reproduce this on FC6 and rawhide.

[...]
301
302
303
Traceback (most recent call last):
  File "foo.py", line 16, in <module>
    y.start()
  File "/usr/lib/python2.5/threading.py", line 434, in start
    _start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread

Are you seriously hitting that limit?  Try setting 'verbose=1' in your
/etc/yum/pluginconf.d/fastestmirror.conf and see if you're really pulling from
that many mirrors.

Comment 5 Luke Macken 2007-02-09 07:16:06 UTC
Ah, disregard my suggestion; I didn't see that you already had verbose output
attached to this bug.

Comment 6 Wojciech Pilorz 2007-02-11 22:56:36 UTC
The bug appeared after I installed yum version 3.0.3-1.fc6.

Previous yum version I had installed before was 3.0.1-2.fc6, and it did not
exhibit this problem.

I have not changed python version in the meantime (I have python-2.4.4-1.fc6 
installed, and I installed it even before yum version 3.0.1-2 )

While I have reported the problem for yum-fastestmirror-1.0.2-2.fc6,
it was also present with yum-fastestmirror-1.0.1-1.fc6 **after** I updated yum
to 3.0.3. Before yum update fastermirror plugin worked nicely for a few weeks.

This might perhaps suggest the problem is in yum, rather than python.
I had included ulimit -S -a output in the original bug report - I do not believe
there is anything non-standard here.

Comment 7 Wojciech Pilorz 2007-02-21 00:41:48 UTC
Tried witn new version of yum and yum-utils - still did not work;
So I run 'yum shell' through strace (this is what I should have done at the very
start) and all became obvious;

Thread creation hit my ulimit -v setting of 425000 - for each thread 10MiB+4KiB
buffer is allocated with mmap, then clone syscall is performed;
What failed was that mmap2 call; 

After increasing address space limit (ulimit -v) to 880000, yum run successfully
once, then again failed.

after
 ulimit -S -v unlimited 
is successfully run, yum works all the time.

I was mislead to believe it is yum version problem because I changed
/etc/security/limits.conf at the same time I upgraded yum.

** However **, with more then 170 mirrors tested for speed, there is no sane
value for ulimit -v; I do not consider unlimited a sane value.

Anyway, requiring user not to have ulimit on address space seems not to be a
very good idea. Perhaps adding a limit on a number of concurrently started
thread, e.g. 20, would be a good idea - it would allow to run fastestmirror with
sane settings (300000 ?) for address space limit.

Comment 8 Wojciech Pilorz 2007-04-11 23:11:48 UTC
For some weeks I run yum with 
ulimit -S -v 999000
and so far the value has always been high enough.
Still I believe that limiting the number of threads running at any given time
would be the proper solution.


Comment 9 Damjan Lango 2007-05-09 12:00:21 UTC
Pilorz: thanks for the work-around, it works with it.
I've seen this problem long time ago, but just disabled fastestmirror, now I
tried it again to see if it has been fixed but still doesn't work...
btw, when using the workaround after fastestmirror goes through all mirrors the
first time, next time the ulimit workaround is not needed as it skips all the
mirrors saying that they have already been timed...
So I agree the proper solution is to limit the number of threads run at the same
time.

Comment 10 Luke Macken 2007-12-03 16:50:40 UTC
I added a 'maxthreads' configuration option to fastestmirror, which defaults to 15.
    
http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=commitdiff;h=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e

Please let me know if this solves your problem.  You can get the latest version
here:
http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=blob_plain;f=plugins/fastestmirror/fastestmirror.py;hb=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e

Comment 11 Fedora Update System 2007-12-07 21:30:26 UTC
yum-utils-1.1.9-1.fc8 has been pushed to the Fedora 8 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update yum-utils'

Comment 12 Fedora Update System 2007-12-07 21:33:37 UTC
yum-utils-1.1.9-1.fc7 has been pushed to the Fedora 7 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update yum-utils'

Comment 13 Wojciech Pilorz 2007-12-10 12:36:46 UTC
(In reply to comment #10)
> I added a 'maxthreads' configuration option to fastestmirror, which defaults
to 15.
>     
>
http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=commitdiff;h=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e
> 
> Please let me know if this solves your problem.  You can get the latest version
> here:
>
http://devel.linux.duke.edu/gitweb/?p=yum-utils.git;a=blob_plain;f=plugins/fastestmirror/fastestmirror.py;hb=a112f75c93c33b6e8f7b59a1e4694f0aecbf9f5e

I have downloaded the version from link above and put in into
FC6 system;
If I set the maxthreads limit quite low (to 3) I can use yum shell with 
ulimit -S -v 322000
The funny thing is that yum with no shell 
(just yum update) seems to work with much higher value of
maxthreads option.

Thank you for the patch, seems to work for me.

Wojtek
PS. notes for other people willing to test the new version with maxthreads;
1. set ulimit -S -v to some value in range 250000 - 2000000
2. test yum list pattern or yum shell , then command
  like update or list
3. remove /var/cache/yum/timehosts.txt before testing
4. I feel that number of threads that yum wants to create (that is number of
mirrors it wants to try(
depends on whetherthe system is behind proxy/firewall with masquarade or not.
Unfortunately I do not have enough number of configurations or knowledge 
to be reasonably sure it is that way.

Comment 14 Fedora Update System 2007-12-23 22:50:17 UTC
yum-utils-1.1.9-1.fc8 has been pushed to the Fedora 8 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update yum-utils'

Comment 15 Fedora Update System 2007-12-23 22:51:01 UTC
yum-utils-1.1.9-1.fc7 has been pushed to the Fedora 7 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update yum-utils'

Comment 16 Fedora Update System 2007-12-26 23:51:48 UTC
yum-utils-1.1.9-1.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 17 Fedora Update System 2007-12-26 23:52:16 UTC
yum-utils-1.1.9-1.fc7 has been pushed to the Fedora 7 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 18 Luke Macken 2008-02-18 22:28:51 UTC
Closing, this has been fixed for a while.


Note You need to log in before you can comment on or make changes to this bug.