728620 – rgmanager crash during HA LVM shutdown

Bug 728620 - rgmanager crash during HA LVM shutdown

Summary: rgmanager crash during HA LVM shutdown

Keywords:
Status:	CLOSED DUPLICATE of bug 697446
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	rgmanager
Sub Component:
Version:	6.1
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Lon Hohberger
QA Contact:	Cluster QE
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2011-08-05 20:33 UTC by Corey Marthaler
Modified:	2011-08-08 14:32 UTC (History)
CC List:	1 user (show)
Fixed In Version:	rgmanager-3.0.12.1-3.el6
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2011-08-08 14:32:35 UTC
Target Upstream Version:

Attachments	(Terms of Use)
rgmanager coredump from taft-02 (188.17 KB, application/x-gzip) 2011-08-05 20:40 UTC, Corey Marthaler	no flags	Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Corey Marthaler 2011-08-05 20:33:01 UTC

Description of problem:
Saw this while shutting down rgmanager with 3 active services (though not all were active on taft-02). I'll attach the core dump.

Aug  5 14:52:07 taft-02 rgmanager[28382]: [lvm] Getting status
Aug  5 14:58:54 taft-02 rgmanager[3474]: Shutting down
Aug  5 14:58:54 taft-02 rgmanager[3474]: Stopping service service:halvm1
Aug  5 14:58:54 taft-02 rgmanager[3474]: Member 1 shutting down
Aug  5 14:58:54 taft-02 rgmanager[3474]: Member 3 shutting down
Aug  5 14:58:54 taft-02 rgmanager[3474]: State change: taft-03 UP
Aug  5 14:58:54 taft-02 rgmanager[30144]: [fs] unmounting /mnt/fs1
Aug  5 14:58:54 taft-02 rgmanager[3474]: State change: taft-03 DOWN
Aug  5 14:58:56 taft-02 lvm[1341]: No longer monitoring mirror device TAFT1-ha for events.
Aug  5 14:58:57 taft-02 rgmanager[30217]: [lvm] Stripping tag, taft-02
Aug  5 14:58:58 taft-02 rgmanager[3474]: Service service:halvm1 is stopped
Aug  5 14:58:58 taft-02 rgmanager[3474]: Disconnecting from CMAN


(gdb) bt
#0  0x00000033a7a32945 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00000033a7a34125 in abort () at abort.c:92
#2  0x0000003d0d22a975 in _dbus_abort () at dbus-sysdeps.c:88
#3  0x0000003d0d226845 in _dbus_warn_check_failed (
    format=0x3d0d2339e0 "arguments to %s() were incorrect, assertion \"%s\" failed in file %s line %d.\nThis is normally a bug in some application using the D-Bus library.\n") at dbus-internals.c:283
#4  0x0000003d0d20d973 in dbus_connection_get_is_connected (connection=0x0) at dbus-connection.c:2876
#5  0x000000000041a4d0 in _rgm_dbus_notify (key=0x7f43b0001400 "rg=\"service:halvm2\"", view=<value optimized out>, data=0x7f43b0001240,
    size=<value optimized out>) at /usr/src/debug/rgmanager-3.0.12.1/rgmanager/src/daemons/update-dbus.c:166
#6  rgm_dbus_update (key=0x7f43b0001400 "rg=\"service:halvm2\"", view=<value optimized out>, data=0x7f43b0001240, size=<value optimized out>)
    at /usr/src/debug/rgmanager-3.0.12.1/rgmanager/src/daemons/update-dbus.c:250
#7  0x000000000041dbea in vf_resolve_views (key_node=0x7f43b00013a0) at /usr/src/debug/rgmanager-3.0.12.1/rgmanager/src/clulib/vft.c:272
#8  0x000000000041e7ed in vf_process_msg (ctx=<value optimized out>, nodeid=<value optimized out>, msgp=0x7f43b00008c0, nbytes=<value optimized out>)
    at /usr/src/debug/rgmanager-3.0.12.1/rgmanager/src/clulib/vft.c:1410
#9  0x000000000041eeca in vf_event_loop (arg=<value optimized out>) at /usr/src/debug/rgmanager-3.0.12.1/rgmanager/src/clulib/vft.c:812
#10 vf_server (arg=<value optimized out>) at /usr/src/debug/rgmanager-3.0.12.1/rgmanager/src/clulib/vft.c:871
#11 0x00000033a7e077e1 in start_thread (arg=0x7f43bf5fe700) at pthread_create.c:301
#12 0x00000033a7ae573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb)


Other nodes:
Aug  5 14:58:17 taft-01 rgmanager[3299]: Shutting down
Aug  5 14:58:17 taft-01 rgmanager[3299]: Disconnecting from CMAN
Aug  5 14:58:17 taft-01 rgmanager[3299]: Exiting
Aug  5 14:58:32 taft-01 corosync[3080]:   [TOTEM ] A processor failed, forming new configuration.
Aug  5 14:58:44 taft-01 corosync[3080]:   [QUORUM] Members[3]: 1 3 4
Aug  5 14:58:44 taft-01 corosync[3080]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug  5 14:58:44 taft-01 kernel: dlm: closing connection to node 2
Aug  5 14:58:44 taft-01 corosync[3080]:   [CPG   ] chosen downlist: sender r(0) ip(10.15.89.67) ; members(old:4 left:1)
Aug  5 14:58:44 taft-01 corosync[3080]:   [MAIN  ] Completed service synchronization, ready to provide service.
Aug  5 14:58:44 taft-01 fenced[3136]: fencing node taft-02


Aug  5 14:58:48 taft-03 rgmanager[3451]: Shutting down
Aug  5 14:58:48 taft-03 dlm_controld[3304]: cpg_mcast_joined error 12 handle 7724c67e00000001 start
Aug  5 14:58:48 taft-03 rgmanager[3451]: Disconnecting from CMAN
Aug  5 14:58:48 taft-03 rgmanager[3451]: Exiting
Aug  5 14:59:02 taft-03 corosync[3232]:   [TOTEM ] A processor failed, forming new configuration.
Aug  5 14:59:14 taft-03 corosync[3232]:   [QUORUM] Members[3]: 1 3 4
Aug  5 14:59:14 taft-03 corosync[3232]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug  5 14:59:14 taft-03 kernel: dlm: closing connection to node 2
Aug  5 14:59:14 taft-03 corosync[3232]:   [CPG   ] chosen downlist: sender r(0) ip(10.15.89.67) ; members(old:4 left:1)
Aug  5 14:59:14 taft-03 corosync[3232]:   [MAIN  ] Completed service synchronization, ready to provide service.
Aug  5 14:59:14 taft-03 fenced[3288]: fencing deferred to taft-01


Aug  5 14:58:52 taft-04 rgmanager[3425]: Shutting down
Aug  5 14:58:52 taft-04 rgmanager[3425]: Member 1 shutting down
Aug  5 14:58:52 taft-04 rgmanager[3425]: Member 3 shutting down
Aug  5 14:58:52 taft-04 rgmanager[3425]: Stopping service service:halvm2
Aug  5 14:58:52 taft-04 rgmanager[3425]: State change: taft-03 UP
Aug  5 14:58:52 taft-04 rgmanager[3425]: Stopping service service:halvm3
Aug  5 14:58:53 taft-04 rgmanager[3425]: State change: taft-03 DOWN
Aug  5 14:58:53 taft-04 rgmanager[19846]: [fs] unmounting /mnt/fs2
Aug  5 14:58:53 taft-04 rgmanager[19885]: [fs] unmounting /mnt/fs3
Aug  5 14:58:55 taft-04 lvm[5141]: No longer monitoring mirror device TAFT3-ha for events.
Aug  5 14:58:55 taft-04 lvm[5141]: No longer monitoring mirror device TAFT2-ha for events.
Aug  5 14:58:56 taft-04 rgmanager[20023]: [lvm] Stripping tag, taft-04
Aug  5 14:58:56 taft-04 rgmanager[20030]: [lvm] Stripping tag, taft-04
Aug  5 14:58:56 taft-04 rgmanager[3425]: Member 2 shutting down
Aug  5 14:58:57 taft-04 rgmanager[3425]: Service service:halvm2 is stopped
Aug  5 14:58:57 taft-04 rgmanager[3425]: Service service:halvm3 is stopped
Aug  5 14:58:57 taft-04 rgmanager[3425]: Disconnecting from CMAN
Aug  5 14:59:07 taft-04 corosync[3206]:   [TOTEM ] A processor failed, forming new configuration.
Aug  5 14:59:12 taft-04 rgmanager[3425]: Exiting
Aug  5 14:59:19 taft-04 kernel: dlm: closing connection to node 2
Aug  5 14:59:19 taft-04 corosync[3206]:   [QUORUM] Members[3]: 1 3 4
Aug  5 14:59:19 taft-04 corosync[3206]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug  5 14:59:19 taft-04 corosync[3206]:   [CPG   ] chosen downlist: sender r(0) ip(10.15.89.67) ; members(old:4 left:1)
Aug  5 14:59:19 taft-04 corosync[3206]:   [MAIN  ] Completed service synchronization, ready to provide service.
Aug  5 14:59:19 taft-04 fenced[3262]: fencing deferred to taft-01


Version-Release number of selected component (if applicable):
rgmanager-3.0.12.1-2.el6.x86_64
corosync-1.4.1-1.el6.x86_64

Comment 1 Corey Marthaler 2011-08-05 20:40:09 UTC

Created attachment 516950 [details]
rgmanager coredump from taft-02

Comment 2 Lon Hohberger 2011-08-08 14:32:35 UTC


*** This bug has been marked as a duplicate of bug 697446 ***

Note You need to log in before you can comment on or make changes to this bug.