Bug 532002
| Summary: | QMF agent does not shutdown properly | ||
|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Frantisek Reznicek <freznice> |
| Component: | qpid-qmf | Assignee: | Ken Giusti <kgiusti> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | MRG Quality Engineering <mrgqe-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | Development | CC: | esammons, jsarenik, kgiusti, tross |
| Target Milestone: | 1.3 | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2010-10-22 10:52:09 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Frantisek Reznicek
2009-10-30 09:00:00 UTC
Feel free to push for 1.3 if needed, but be aware that there is most probably dependency to bug 521138, investigating now... One more piece of information to the observation beta] I got qpid::sys::Thread::join() assertion which might be the source of the aborts (this is coming from rhel4.8 x86_64) ... 2009-oct-29 13:32:30 warning Connection closed Resource deadlock avoided qmf-agent: qpid/sys/posix/Thread.cpp:58: void qpid::sys::Thread::join(): Assertion `0' failed. ./test.sh: line 21: 6919 Aborted (core dumped) ( ./qmf-agent 127.0.0.1 99999 ) ... Ha! Good catch - and I immediately see the problem: qmf-agent is written incorrectly. It attempts to do the entire system cleanup from a *signal handler*! This is not good, as lots of code is run during cleanup, and a good chunk of it is not able to run safely from a signal handler. I'm 99.44/100% sure this is the problem (but I'll try to fix/test it shortly). Other than being a bad example for our customers to follow, I don't think that this should be a 1.2 blocking issue. Opinions? -K Suggested fix submitted upstream: https://issues.apache.org/jira/browse/QPID-2179 Frantisek - be aware: After applying a fix, I'm seeing a problem with the test.sh script where it causes many instances of the qmf-agent to be run simultaneously. I point this out because, after a while my system exhausts its thread resources and we abort(). This causes a core file. Note well: this core is not due to the signal bug! Running out of pthread resources is NAB. Just FYI: I'm seeing this output when I run out of threads: 2009-10-30 09:49:58 warning Connection closed 2009-10-30 09:49:58 warning Connecting failed: Connection refused 2009-10-30 09:49:58 warning Connection closed + echo -n 2 2+ '[' 2 -lt 3 ']' + pid=26411 + usleep 24210 + ./qmf-agent 127.0.0.1 99999 + kill -2 26411 ./test.sh: fork: retry: Resource temporarily unavailable <<<<!!!! ++ ls core.20190 core.26376 Resource temporarily unavailable qmf-agent: qpid/sys/posix/Thread.cpp:44: qpid::sys::ThreadPrivate::ThreadPrivate(qpid::sys::Runnable*): Assertion `0' failed. Resource temporarily unavailable qmf-agent: qpid/sys/posix/Thread.cpp:44: qpid::sys::ThreadPrivate::ThreadPrivate(qpid::sys::Runnable*): Assertion `0' failed. Resource temporarily unavailable qmf-agent: qpid/sys/posix/Thread.cpp:44: qpid::sys::ThreadPrivate::ThreadPrivate(qpid::sys::Runnable*): Assertion `0' failed. 2009-10-30 09:49:59 warning Connecting failed: Connection refused 2009-10-30 09:49:59 warning Connection closed ++ wc -l + cnt=2 + usleep 2624800 2009-10-30 09:50:00 warning Connecting failed: Connection refused 2009-10-30 09:50:00 warning Connecting failed: Connection refused ... 2009-10-30 09:50:02 warning Connecting failed: Connection refused 2009-10-30 09:50:02 warning Connection closed ./test.sh: line 32: 19262 Aborted (core dumped) ( ./qmf-agent 127.0.0.1 99999 ) ./test.sh: line 32: 25331 Aborted (core dumped) ( ./qmf-agent 127.0.0.1 99999 ) ./test.sh: line 32: 26150 Aborted (core dumped) ( ./qmf-agent 127.0.0.1 99999 ) Doing a "ps -ef | grep qmf-agent" shows the test has created thousands of qmf-agents: kgiusti 16110 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16127 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16135 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16158 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16206 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16216 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16267 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16284 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16292 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16328 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16406 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 kgiusti 16415 1 0 09:27 pts/0 00:00:00 ./qmf-agent 127.0.0.1 99999 ..... ..... etc. -K |