Bug 959189

Summary: Corosync doesn't preempt in confdb properly and can hang on exit
Product: Red Hat Enterprise Linux 6 Reporter: Jan Friesse <jfriesse>
Component: corosyncAssignee: Jan Friesse <jfriesse>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.4CC: cluster-maint, jkortus, lnovich, sdake
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: corosync-1.4.1-16.el6 Doc Type: Bug Fix
Doc Text:
Cause: Confdb client uses corosync heavily and corosync is exited (sending SIGINT or corosync-cfgtool). Consequence: Corosync will never exit. Fix: Make sure all confdb IPC connections are closed on corosync exit. Result: Corosync will exit properly.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-21 04:34:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Proposed patch - part 1 - Free confdb message holder list on confdb exit
none
Proposed patch - part 2 - Make objdb_notify_dispatch preemptable none

Description Jan Friesse 2013-05-03 10:15:27 UTC 
Created attachment 743143 [details]
Proposed patch - part 1 - Free confdb message holder list on confdb exit

Description of problem:
objdb_notify_dispatch traverse thru confdb_ipc_message_holder_list list
until it's empty. If IPC connection is still feeding confdb/objdb with
changes then iteration will never end. To prevent this and let corosync
properly preempt, terminal condition is enhanced with maximum number of
processed items from list. Change makes confdb preemptable so corosync
can accept new connections, receive/send token, ...

Also when confdb service is unloaded. it's needed to free confdb message holder list. This operation will not only properly free memory, but also decreases
connection reference(s). This should solve deadloop in
coroipcs_ipc_service_exit, because confdb_exit_fn removes not only
notify_pipe but also deletes poll_dispatch, so nobody is left to
unreference connections.

Version-Release number of selected component (if applicable):
Flatiron

How reproducible:
98.76

Steps to Reproduce:
1. Start corosync -p
2. Start IPC client which track key changes and also make changes of that key
3. Exit corosync (sending SIGINT)

"Unit" test:
https://github.com/jfriesse/csts/blob/master/tests/stop-with-confdb-load.sh

Actual results:
Corosync will not exit (unit test $? != 0)

Expected results:
Corosync will properly exit (unit test $? == 0)

Additional info:
Comment 1 Jan Friesse 2013-05-03 10:16:10 UTC 
Created attachment 743144 [details]
Proposed patch - part 2 - Make objdb_notify_dispatch preemptable
Comment 8 Jaroslav Kortus 2013-09-11 15:33:46 UTC 
Verified with stop-with-confdb-load.sh test:
FAIL on corosync-1.4.1-15.el6.x86_64 (RHEL6.4)
PASS on corosync-1.4.1-17.el6.x86_64 (RHEL6.5)
Comment 10 errata-xmlrpc 2013-11-21 04:34:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1531.html