Bug 631890

Created attachment 446013 [details]
gdb examined coredump

Description of problem:
On attempt to reproduce bug 613753. qpid-perftest made huge network load, after
approx. 30 minutes openais service aborted with:
(this one is different run, and happened on different node than bug 631835)

(for full trace see attach):
===snip===
Thread 1 (Thread 15504):
#0  0x00f32402 in __kernel_vsyscall ()
#1  0x002b2df0 in raise () from /lib/libc.so.6
#2  0x002b4701 in abort () from /lib/libc.so.6
#3  0x002eb28b in __libc_message () from /lib/libc.so.6
#4  0x003702d1 in __chk_fail () from /lib/libc.so.6
#5  0x0805e5ab in totempg_deliver_fn (nodeid=4460554, iovec=0xb71fd158, 
    iov_len=1, endian_conversion_required=0) at totempg.c:530
#6  0x0805c8a5 in totemmrp_deliver_fn (nodeid=4460554, iovec=0xb71fd158, 
    iov_len=1, endian_conversion_required=0) at totemmrp.c:82
#7  0x08058781 in messages_deliver_to_app (instance=0xb74af008, skip=0, 
    end_point=<value optimized out>) at totemsrp.c:3550
#8  0x0805911c in memb_state_operational_enter (instance=0xb74af008)
    at totemsrp.c:1613
#9  0x0805ae94 in message_handler_orf_token (instance=0xb74af008, 
    msg=0x8c6ab9c, msg_len=70, endian_conversion_needed=0) at totemsrp.c:3395
#10 0x08053a02 in main_deliver_fn (context=0xb74af008, msg=0x8c6ab9c, 
    msg_len=15504) at totemsrp.c:4135
#11 0x08051240 in none_token_recv (rrp_instance=0x8c48160, iface_no=0, 
    context=0xb74af008, msg=0x8c6ab9c, msg_len=70, token_seq=27)
    at totemrrp.c:506
#12 0x08051316 in rrp_deliver_fn (context=0x8c485d8, msg=0x8c6ab9c, msg_len=70)
    at totemrrp.c:1308
#13 0x0804f752 in net_deliver_fn (handle=0, fd=2, revents=1, data=0x8c6a570)
    at totemnet.c:695
#14 0x0804c8c2 in poll_run (handle=0) at aispoll.c:402
#15 0x08062295 in main (argc=Cannot access memory at address 0x3c90
) at main.c:628
===snip===

cat /etc/ais/openais.conf 
# Please read the openais.conf.5 manual page

totem {
 version: 2
 secauth: off
 threads: 0
 interface {
  ringnumber: 0
  bindnetaddr: 10.16.64.0 
  mcastaddr: 226.94.1.1
  mcastport: 5405
 }
}

logging {
 debug: on
 timestamp: on
}

amf {
 mode: disabled
}


#cat /etc/qpidd.conf
log-to-file=/tmp/qpidd.log
log-enable=debug+
cluster-name=test_cluster_01
cluster-mechanism=ANONYMOUS
auth=no




Version-Release number of selected component (if applicable):
openais-0.80.6-16.el5_5.7
openais-debuginfo-0.80.6-16.el5_5.7
openais-devel-0.80.6-16.el5_5.7




How reproducible:
5%

Steps to Reproduce:
1. setenforce 0
2. service openais start
3. follow bug 613753

Actual results:
openais aborts

Expected results:
no aborts

Additional info:
core file found:
dell-pe650-02.rhts.eng.bos.redhat.com


other hosts involved in test:
hp-bl685c-01.rhts.eng.bos.redhat.com
nec-em13.rhts.eng.bos.redhat.com
hp-bl480c-01.rhts.eng.bos.redhat.com
hp-dl360g5-02.rhts.eng.bos.redhat.com
hp-dl385g5p-01.rhts.eng.bos.redhat.com


qpid packages used in bug 613753:
python-qpid-0.7.946106-4.el5
qpid-cpp-client-0.7.946106-6.el5
qpid-cpp-client-devel-0.7.946106-6.el5
qpid-cpp-client-devel-docs-0.7.946106-6.el5
qpid-cpp-client-ssl-0.7.946106-6.el5
qpid-cpp-server-0.7.946106-6.el5
qpid-cpp-server-cluster-0.7.946106-6.el5
qpid-cpp-server-devel-0.7.946106-6.el5
qpid-cpp-server-ssl-0.7.946106-6.el5
qpid-cpp-server-store-0.7.946106-6.el5
qpid-cpp-server-xml-0.7.946106-6.el5
qpid-java-client-0.7.946106-5.el5
qpid-java-common-0.7.946106-5.el5
qpid-tools-0.7.946106-6.el5