Bug 631835

Summary: aisexec triggers abort from memb_state_commit_token_update
Product: Red Hat Enterprise Linux 5 Reporter: ppecka <ppecka>
Component: openaisAssignee: Steven Dake <sdake>
Status: CLOSED DUPLICATE QA Contact: Cluster QE <mspqa-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.5CC: cluster-maint, edamato
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-02-07 18:31:29 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
gdb examined coredump none

Description ppecka 2010-09-08 13:54:58 UTC 
Created attachment 445992 [details]
gdb examined coredump

Description of problem:
On attempt to reproduce bug 613753. qpid-perftest made huge network load, after approx. 15 minutes openais service aborted with:

(for full trace see attach):
===snip===
Thread 1 (Thread 15636):
#0  0x00000034d6a30265 in raise () from /lib64/libc.so.6
#1  0x00000034d6a31d10 in abort () from /lib64/libc.so.6
#2  0x00000034d6a296e6 in __assert_fail () from /lib64/libc.so.6
#3  0x000000000040d1a5 in memb_state_commit_token_update (
    instance=0x2aaaab2bd010, commit_token=0x1249e74) at totemsrp.c:2681
#4  memb_state_commit_enter (instance=0x2aaaab2bd010, commit_token=0x1249e74)
    at totemsrp.c:1754
#5  0x000000000040f9f3 in message_handler_memb_commit_token (
    instance=0x2aaaab2bd010, msg=0x1249e74, msg_len=<value optimized out>, 
    endian_conversion_needed=<value optimized out>) at totemsrp.c:4062
#6  0x0000000000409f5e in rrp_deliver_fn (context=0x12295b0, msg=0x1249e74, 
    msg_len=312) at totemrrp.c:1319
#7  0x00000000004084eb in net_deliver_fn (handle=<value optimized out>, 
    fd=<value optimized out>, revents=<value optimized out>, data=0x12497f0)
    at totemnet.c:695
#8  0x0000000000405d00 in poll_run (handle=0) at aispoll.c:402
#9  0x000000000041898e in main (argc=<value optimized out>, 
    argv=<value optimized out>) at main.c:628
===snip===


cat /etc/ais/openais.conf 
# Please read the openais.conf.5 manual page

totem {
	version: 2
	secauth: off
	threads: 0
	interface {
		ringnumber: 0
		bindnetaddr: 10.16.64.0 
		mcastaddr: 226.94.1.1
		mcastport: 5405
	}
}

logging {
	debug: on
	timestamp: on
}

amf {
	mode: disabled
}


#cat /etc/qpidd.conf
log-to-file=/tmp/qpidd.log
log-enable=debug+
cluster-name=test_cluster_01
cluster-mechanism=ANONYMOUS
auth=no




Version-Release number of selected component (if applicable):
openais-0.80.6-16.el5_5.7
openais-debuginfo-0.80.6-16.el5_5.7
openais-devel-0.80.6-16.el5_5.7




How reproducible:
10%

Steps to Reproduce:
1. setenforce 0
2. service openais start
3. follow bug 613753

Actual results:
openais aborts

Expected results:
no aborts

Additional info:
core file found:
hp-dl385g5p-01.rhts.eng.bos.redhat.com

other hosts involved in test:
hp-bl685c-01.rhts.eng.bos.redhat.com
nec-em13.rhts.eng.bos.redhat.com
hp-bl480c-01.rhts.eng.bos.redhat.com
hp-dl360g5-02.rhts.eng.bos.redhat.com
dell-pe650-02.rhts.eng.bos.redhat.com


qpid packages used in bug 613753:
python-qpid-0.7.946106-4.el5
qpid-cpp-client-0.7.946106-6.el5
qpid-cpp-client-devel-0.7.946106-6.el5
qpid-cpp-client-devel-docs-0.7.946106-6.el5
qpid-cpp-client-ssl-0.7.946106-6.el5
qpid-cpp-server-0.7.946106-6.el5
qpid-cpp-server-cluster-0.7.946106-6.el5
qpid-cpp-server-devel-0.7.946106-6.el5
qpid-cpp-server-ssl-0.7.946106-6.el5
qpid-cpp-server-store-0.7.946106-6.el5
qpid-cpp-server-xml-0.7.946106-6.el5
qpid-java-client-0.7.946106-5.el5
qpid-java-common-0.7.946106-5.el5
qpid-tools-0.7.946106-6.el5
Comment 1 Steven Dake 2010-09-08 17:14:36 UTC 
(gdb) print *commit_token
$3 = {header = {type = 4 '\004', encapsulated = 0 '\000', 
    endian_detector = 65314, nodeid = 1363152906}, token_seq = 3, ring_id = {
    rep = {nodeid = 1363152906, family = 2, 
      addr = "\n\020@Q\b\000\002\000\n\020@Q\b\000\004"}, seq = 712}, 
  retrans_flg = 0, memb_index = 4, addr_entries = 3, 
  end_of_commit_token = 0x967abd2 "\n\020@Q\002"}
Comment 2 Steven Dake 2010-09-08 17:19:07 UTC 
Need access to machines to run special builds - looks environmental and there are no logs on the target listed.
Comment 4 Steven Dake 2011-02-07 18:31:29 UTC 

*** This bug has been marked as a duplicate of bug 623176 ***