Bug 627686 - runs out of process resources
Summary: runs out of process resources
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: gwibber
Version: 14
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Tom "spot" Callaway
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 628418 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-08-26 16:53 UTC by Bill Nottingham
Modified: 2014-03-17 03:24 UTC (History)
12 users (show)

Fixed In Version: gwibber-2.33.0-7.867bzr.fc12
Clone Of:
Environment:
Last Closed: 2010-10-05 13:21:57 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
un-wtf gwibber service (3.09 KB, patch)
2010-08-26 20:01 UTC, Bill Nottingham
no flags Details | Diff
patch, v2 (2.91 KB, patch)
2010-08-26 20:13 UTC, Bill Nottingham
no flags Details | Diff

Description Bill Nottingham 2010-08-26 16:53:11 UTC
Description of problem:

I'm getting 'Resource temporarily unavailable' when starting processes as a user often with the new kernel. I have the stock user process limit (1024), and there's only about 700-750 *total* processes on the system when this happens.

Usual symptom is:
-bash: fork: retry: Resource temporarily unavailable

root is able to start new processes.

If I try 'su notting -c ls', I get a similar EAGAIN failure:
31445 open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
31445 fstat(3, {st_mode=S_IFREG|0644, st_size=2337, ...}) = 0
31445 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
 0x7fdaab9d2000
31445 read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 2337
31445 close(3)                          = 0
31445 munmap(0x7fdaab9d2000, 4096)      = 0
31445 getuid()                          = 0
31445 getgid()                          = 0
31445 setregid(2167, 4294967295)        = 0
31445 setreuid(2166, 4294967295)        = -1 EAGAIN (Resource temporarily unavailable)

Starting off against kernel... may be libc, may be something else.

I can fix it by (as root) killing off some number of processes.

Version-Release number of selected component (if applicable):

kernel-2.6.35.2-9.fc14.x86_64
glibc-2.12.90-8.x86_64
systemd-7-3.fc14.x86_64

How reproducible:

100% (after a while)

Comment 1 Bill Nottingham 2010-08-26 19:05:50 UTC
Aha, wasn't counting the processes right. I am running against the rlimit for number of active threads (1024). 

Assigning to the offender.

Comment 2 Bill Nottingham 2010-08-26 20:01:49 UTC
Created attachment 441324 [details]
un-wtf gwibber service

gwibber-service was creating a new worker pool of threads/processes for each async request that it decided to do. This fixes that.

Comment 3 Bill Nottingham 2010-08-26 20:13:31 UTC
Created attachment 441326 [details]
patch, v2

Patch v2 cleans up some cruft that's always the same.

Comment 4 Bill Nottingham 2010-08-26 20:32:24 UTC
That being said, the gwibber code has been this way for a while. I wonder why it's suddenly a problem now.

Comment 5 Tom "spot" Callaway 2010-08-26 20:37:28 UTC
Upstream claims that the code is supposed to destroy the pool after the operation finishes. Perhaps this is a python issue?

I'm going to push an update with this patch anyways, since it isn't necessarily wrong. Also opened a ticket with upstream.

Comment 6 Bill Nottingham 2010-08-27 15:35:53 UTC
CC'ing David re: comment #5.

Comment 7 Dave Malcolm 2010-08-27 16:17:33 UTC
Looking briefly at MapAsync.run(), I see that it creates a multiprocessing.Pool() and assigns it to local "pool"; by default it uses the CPU count for the number of child processes.

Looking briefly over multiprocessing.pool.Pool, it doesn't seem to have an explicit __del__ method, but it does have methods like terminate().

So when "pool" goes out of scope, the pool will get cleaned up at some future point (it may well have cyclic references to it by other object, so may have to wait for the garbage collector).

It looks like it may help to add an explicit:
   finally:
      pool.terminate()
or somesuch, to explicitly tell the code that it really ought to try to clean up those threads, rather than waiting for implicit cleanup during garbage collection.

Comment 8 Dave Malcolm 2010-08-27 16:18:04 UTC
(In reply to comment #7)
> up those threads, rather than waiting for implicit cleanup during garbage
s/threads/processes/

Comment 9 Bill Nottingham 2010-08-27 17:23:27 UTC
Note that in my patch, the t.join() is superfluous (and actually makes it more blocking.)

Comment 10 Tom "spot" Callaway 2010-08-30 13:53:01 UTC
*** Bug 628418 has been marked as a duplicate of this bug. ***

Comment 11 Fedora Update System 2010-09-26 17:55:29 UTC
gwibber-2.33.0-7.867bzr.fc13 has been submitted as an update for Fedora 13.
https://admin.fedoraproject.org/updates/gwibber-2.33.0-7.867bzr.fc13

Comment 12 Fedora Update System 2010-09-26 17:56:17 UTC
gwibber-2.33.0-7.867bzr.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/gwibber-2.33.0-7.867bzr.fc14

Comment 13 Fedora Update System 2010-09-26 17:57:05 UTC
gwibber-2.33.0-7.867bzr.fc12 has been submitted as an update for Fedora 12.
https://admin.fedoraproject.org/updates/gwibber-2.33.0-7.867bzr.fc12

Comment 14 Fedora Update System 2010-09-27 03:38:05 UTC
gwibber-2.33.0-7.867bzr.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update gwibber'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/gwibber-2.33.0-7.867bzr.fc14

Comment 15 Fedora Update System 2010-10-05 13:21:06 UTC
gwibber-2.33.0-7.867bzr.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2010-10-08 20:46:30 UTC
gwibber-2.33.0-7.867bzr.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 17 Fedora Update System 2010-10-08 20:52:19 UTC
gwibber-2.33.0-7.867bzr.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.