Bug 959189 - Corosync doesn't preempt in confdb properly and can hang on exit
Summary: Corosync doesn't preempt in confdb properly and can hang on exit
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: corosync
Version: 6.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Jan Friesse
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-05-03 10:15 UTC by Jan Friesse
Modified: 2013-11-21 04:34 UTC (History)
4 users (show)

Fixed In Version: corosync-1.4.1-16.el6
Doc Type: Bug Fix
Doc Text:
Cause: Confdb client uses corosync heavily and corosync is exited (sending SIGINT or corosync-cfgtool). Consequence: Corosync will never exit. Fix: Make sure all confdb IPC connections are closed on corosync exit. Result: Corosync will exit properly.
Clone Of:
Environment:
Last Closed: 2013-11-21 04:34:16 UTC


Attachments (Terms of Use)
Proposed patch - part 1 - Free confdb message holder list on confdb exit (1.63 KB, patch)
2013-05-03 10:15 UTC, Jan Friesse
no flags Details | Diff
Proposed patch - part 2 - Make objdb_notify_dispatch preemptable (3.87 KB, patch)
2013-05-03 10:16 UTC, Jan Friesse
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1531 normal SHIPPED_LIVE corosync bug fix and enhancement update 2013-11-21 00:40:57 UTC

Description Jan Friesse 2013-05-03 10:15:27 UTC
Created attachment 743143 [details]
Proposed patch - part 1 - Free confdb message holder list on confdb exit

Description of problem:
objdb_notify_dispatch traverse thru confdb_ipc_message_holder_list list
until it's empty. If IPC connection is still feeding confdb/objdb with
changes then iteration will never end. To prevent this and let corosync
properly preempt, terminal condition is enhanced with maximum number of
processed items from list. Change makes confdb preemptable so corosync
can accept new connections, receive/send token, ...

Also when confdb service is unloaded. it's needed to free confdb message holder list. This operation will not only properly free memory, but also decreases
connection reference(s). This should solve deadloop in
coroipcs_ipc_service_exit, because confdb_exit_fn removes not only
notify_pipe but also deletes poll_dispatch, so nobody is left to
unreference connections.

Version-Release number of selected component (if applicable):
Flatiron

How reproducible:
98.76

Steps to Reproduce:
1. Start corosync -p
2. Start IPC client which track key changes and also make changes of that key
3. Exit corosync (sending SIGINT)

"Unit" test:
https://github.com/jfriesse/csts/blob/master/tests/stop-with-confdb-load.sh

Actual results:
Corosync will not exit (unit test $? != 0)

Expected results:
Corosync will properly exit (unit test $? == 0)

Additional info:

Comment 1 Jan Friesse 2013-05-03 10:16:10 UTC
Created attachment 743144 [details]
Proposed patch - part 2 - Make objdb_notify_dispatch preemptable

Comment 8 Jaroslav Kortus 2013-09-11 15:33:46 UTC
Verified with stop-with-confdb-load.sh test:
FAIL on corosync-1.4.1-15.el6.x86_64 (RHEL6.4)
PASS on corosync-1.4.1-17.el6.x86_64 (RHEL6.5)

Comment 10 errata-xmlrpc 2013-11-21 04:34:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1531.html


Note You need to log in before you can comment on or make changes to this bug.