RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 917914 - the cluster is down because of [TOTEM ] FAILED TO RECEIVE and INFO: task clvmd:4741 blocked
Summary: the cluster is down because of [TOTEM ] FAILED TO RECEIVE and INFO: task clv...
Keywords:
Status: CLOSED DUPLICATE of bug 854216
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: corosync
Version: 6.4
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Jan Friesse
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-03-05 06:25 UTC by muse
Modified: 2013-03-05 10:27 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-03-05 10:27:18 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description muse 2013-03-05 06:25:39 UTC 
Description of problem:
The RHEL6.4 is installed on the Cisco UCS B200 NIC 1240. Before I create the cluster, the OS is working well, but after I create the cluster node from luci and within 2 minutes, the cluster service will be no responsed and crashed after the cluster is created. The cluster service can not be stopped by useing service XXX stop, and the reboot is very very slow and unsuccessful. After serveral hours, the cluster service management can be closed with errors.
Actually, the cluster only has the nodes name. There is no possible to add other resources because of the service crashed.

OS info 
rpm -q corosync cman rgmanager fence-agents gfs2-utils lvm2-cluster
corosync-1.4.1-15.el6.x86_64
cman-3.0.12.1-49.el6.x86_64
rgmanager-3.0.12.1-17.el6.x86_64
fence-agents-3.1.5-25.el6.x86_64
gfs2-utils-3.0.12.1-49.el6.x86_64
lvm2-cluster-2.02.98-9.el6.x86_64

uname -a
Linux 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux


Error Log:
Mar  4 16:10:22 ARCUCSB2007712M corosync[4519]:   [TOTEM ] Retransmit List: 78 7a 7b 7c 7e 7f 80 82 83
Mar  4 16:10:24 ARCUCSB2007712M corosync[4519]:   [TOTEM ] Retransmit List: 78 7a 7b 7c 7e 7f 80 82 83
Mar  4 16:10:26 ARCUCSB2007712M corosync[4519]:   [TOTEM ] Retransmit List: 78 7a 7b 7c 7e 7f 80 82 83
Mar  4 16:10:28 ARCUCSB2007712M corosync[4519]:   [TOTEM ] Retransmit List: 78 7a 7b 7c 7e 7f 80 82 83
Mar  4 16:10:28 ARCUCSB2007712M corosync[4519]:   [TOTEM ] FAILED TO RECEIVE
Mar  4 16:10:30 ARCUCSB2007712M abrt[8051]: File '/usr/sbin/corosync' seems to be deleted
Mar  4 16:10:30 ARCUCSB2007712M abrt[8051]: Saved core dump of pid 4519 (/usr/sbin/corosync) to /var/spool/abrt/ccpp-2013-03-04-16:10:30-451                9 (67997696 bytes)
Mar  4 16:10:30 ARCUCSB2007712M abrtd: Directory 'ccpp-2013-03-04-16:10:30-4519' creation detected
Mar  4 16:10:30 ARCUCSB2007712M fenced[4582]: cluster is down, exiting
Mar  4 16:10:30 ARCUCSB2007712M fenced[4582]: daemon cpg_dispatch error 2
Mar  4 16:10:30 ARCUCSB2007712M dlm_controld[4604]: cluster is down, exiting
Mar  4 16:10:30 ARCUCSB2007712M dlm_controld[4604]: daemon cpg_dispatch error 2
Mar  4 16:10:30 ARCUCSB2007712M dlm_controld[4604]: cpg_dispatch error 2
Mar  4 16:10:30 ARCUCSB2007712M gfs_controld[4657]: cluster is down, exiting
Mar  4 16:10:30 ARCUCSB2007712M gfs_controld[4657]: daemon cpg_dispatch error 2
Mar  4 16:10:31 ARCUCSB2007712M kernel: Bridge firewalling registered
Mar  4 16:10:36 ARCUCSB2007712M kernel: dlm: closing connection to node 2
Mar  4 16:10:36 ARCUCSB2007712M kernel: dlm: closing connection to node 1
Mar  4 16:10:36 ARCUCSB2007712M kernel: dlm: rgmanager: no userland control daemon, stopping lockspace
Mar  4 16:10:36 ARCUCSB2007712M kernel: dlm: clvmd: no userland control daemon, stopping lockspace
Mar  4 16:12:56 ARCUCSB2007712M kernel: INFO: task clvmd:4741 blocked for more than 120 seconds.
Mar  4 16:12:56 ARCUCSB2007712M kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  4 16:12:56 ARCUCSB2007712M kernel: clvmd         D 0000000000000002     0  4741      1 0x00000080
Mar  4 16:12:56 ARCUCSB2007712M kernel: ffff881070defd68 0000000000000082 ffff881070defd30 ffff881070defd2c
Mar  4 16:12:56 ARCUCSB2007712M kernel: 0000000000000000 ffff88087fc28800 ffff880028276700 0000000000000400
Mar  4 16:12:56 ARCUCSB2007712M kernel: ffff881070c4d058 ffff881070deffd8 000000000000fb88 ffff881070c4d058
Mar  4 16:12:56 ARCUCSB2007712M kernel: Call Trace:
Mar  4 16:12:56 ARCUCSB2007712M kernel: [<ffffffff8150ed3e>] __mutex_lock_slowpath+0x13e/0x180
Mar  4 16:12:56 ARCUCSB2007712M kernel: [<ffffffff8150ebdb>] mutex_lock+0x2b/0x50
Mar  4 16:12:56 ARCUCSB2007712M kernel: [<ffffffffa027cea8>] dlm_release_lockspace+0x48/0x480 [dlm]
Mar  4 16:12:56 ARCUCSB2007712M kernel: [<ffffffffa0285f92>] device_write+0x2b2/0x720 [dlm]
Mar  4 16:12:56 ARCUCSB2007712M kernel: [<ffffffff811a1ae0>] ? mntput_no_expire+0x30/0x110
Mar  4 16:12:56 ARCUCSB2007712M kernel: [<ffffffff8121baf6>] ? security_file_permission+0x16/0x20
Mar  4 16:12:56 ARCUCSB2007712M kernel: [<ffffffff81180f98>] vfs_write+0xb8/0x1a0
Mar  4 16:12:56 ARCUCSB2007712M kernel: [<ffffffff81181891>] sys_write+0x51/0x90
Mar  4 16:12:56 ARCUCSB2007712M kernel: [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
Mar  4 16:12:56 ARCUCSB2007712M kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Mar  4 16:14:56 ARCUCSB2007712M kernel: INFO: task clvmd:4741 blocked for more than 120 seconds.
Mar  4 16:14:56 ARCUCSB2007712M kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  4 16:14:56 ARCUCSB2007712M kernel: clvmd         D 0000000000000002     0  4741      1 0x00000080
Mar  4 16:14:56 ARCUCSB2007712M kernel: ffff881070defd68 0000000000000082 ffff881070defd30 ffff881070defd2c
Mar  4 16:14:56 ARCUCSB2007712M kernel: 0000000000000000 ffff88087fc28800 ffff880028276700 0000000000000400
Mar  4 16:14:56 ARCUCSB2007712M kernel: ffff881070c4d058 ffff881070deffd8 000000000000fb88 ffff881070c4d058
Mar  4 16:14:56 ARCUCSB2007712M kernel: Call Trace:
Mar  4 16:14:56 ARCUCSB2007712M kernel: [<ffffffff8150ed3e>] __mutex_lock_slowpath+0x13e/0x180
Mar  4 16:14:56 ARCUCSB2007712M kernel: [<ffffffff8150ebdb>] mutex_lock+0x2b/0x50
Mar  4 16:14:56 ARCUCSB2007712M kernel: [<ffffffffa027cea8>] dlm_release_lockspace+0x48/0x480 [dlm]
Mar  4 16:14:56 ARCUCSB2007712M kernel: [<ffffffffa0285f92>] device_write+0x2b2/0x720 [dlm]
Mar  4 16:14:56 ARCUCSB2007712M kernel: [<ffffffff811a1ae0>] ? mntput_no_expire+0x30/0x110
Mar  4 16:14:56 ARCUCSB2007712M kernel: [<ffffffff8121baf6>] ? security_file_permission+0x16/0x20
Mar  4 16:14:56 ARCUCSB2007712M kernel: [<ffffffff81180f98>] vfs_write+0xb8/0x1a0
Mar  4 16:14:56 ARCUCSB2007712M kernel: [<ffffffff81181891>] sys_write+0x51/0x90
Mar  4 16:14:56 ARCUCSB2007712M kernel: [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
Mar  4 16:14:56 ARCUCSB2007712M kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Mar  4 16:16:56 ARCUCSB2007712M kernel: INFO: task clvmd:4741 blocked for more than 120 seconds.
Mar  4 16:16:56 ARCUCSB2007712M kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  4 16:16:56 ARCUCSB2007712M kernel: clvmd         D 0000000000000002     0  4741      1 0x00000080
Mar  4 16:16:56 ARCUCSB2007712M kernel: ffff881070defd68 0000000000000082 ffff881070defd30 ffff881070defd2c
Mar  4 16:16:56 ARCUCSB2007712M kernel: 0000000000000000 ffff88087fc28800 ffff880028276700 0000000000000400
Mar  4 16:16:56 ARCUCSB2007712M kernel: ffff881070c4d058 ffff881070deffd8 000000000000fb88 ffff881070c4d058
Mar  4 16:16:56 ARCUCSB2007712M kernel: Call Trace:
Mar  4 16:16:56 ARCUCSB2007712M kernel: [<ffffffff8150ed3e>] __mutex_lock_slowpath+0x13e/0x180
Mar  4 16:16:56 ARCUCSB2007712M kernel: [<ffffffff8150ebdb>] mutex_lock+0x2b/0x50
Mar  4 16:16:56 ARCUCSB2007712M kernel: [<ffffffffa027cea8>] dlm_release_lockspace+0x48/0x480 [dlm]
Mar  4 16:16:56 ARCUCSB2007712M kernel: [<ffffffffa0285f92>] device_write+0x2b2/0x720 [dlm]
Mar  4 16:16:56 ARCUCSB2007712M kernel: [<ffffffff811a1ae0>] ? mntput_no_expire+0x30/0x110
Mar  4 16:16:56 ARCUCSB2007712M kernel: [<ffffffff8121baf6>] ? security_file_permission+0x16/0x20
Mar  4 16:16:56 ARCUCSB2007712M kernel: [<ffffffff81180f98>] vfs_write+0xb8/0x1a0
Mar  4 16:16:56 ARCUCSB2007712M kernel: [<ffffffff81181891>] sys_write+0x51/0x90
Mar  4 16:16:56 ARCUCSB2007712M kernel: [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
Mar  4 16:16:56 ARCUCSB2007712M kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Mar  4 16:18:56 ARCUCSB2007712M kernel: INFO: task clvmd:4741 blocked for more than 120 seconds.
Mar  4 16:18:56 ARCUCSB2007712M kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  4 16:18:56 ARCUCSB2007712M kernel: clvmd         D 0000000000000002     0  4741      1 0x00000080
Mar  4 16:18:56 ARCUCSB2007712M kernel: ffff881070defd68 0000000000000082 ffff881070defd30 ffff881070defd2c
Mar  4 16:18:56 ARCUCSB2007712M kernel: 0000000000000000 ffff88087fc28800 ffff880028276700 0000000000000400
Mar  4 16:18:56 ARCUCSB2007712M kernel: ffff881070c4d058 ffff881070deffd8 000000000000fb88 ffff881070c4d058
Mar  4 16:18:56 ARCUCSB2007712M kernel: Call Trace:
Mar  4 16:18:56 ARCUCSB2007712M kernel: [<ffffffff8150ed3e>] __mutex_lock_slowpath+0x13e/0x180
Mar  4 16:18:56 ARCUCSB2007712M kernel: [<ffffffff8150ebdb>] mutex_lock+0x2b/0x50
Mar  4 16:18:56 ARCUCSB2007712M kernel: [<ffffffffa027cea8>] dlm_release_lockspace+0x48/0x480 [dlm]
Mar  4 16:18:56 ARCUCSB2007712M kernel: [<ffffffffa0285f92>] device_write+0x2b2/0x720 [dlm]
Mar  4 16:18:56 ARCUCSB2007712M kernel: [<ffffffff811a1ae0>] ? mntput_no_expire+0x30/0x110
Mar  4 16:18:56 ARCUCSB2007712M kernel: [<ffffffff8121baf6>] ? security_file_permission+0x16/0x20
Mar  4 16:18:56 ARCUCSB2007712M kernel: [<ffffffff81180f98>] vfs_write+0xb8/0x1a0
Mar  4 16:18:56 ARCUCSB2007712M kernel: [<ffffffff81181891>] sys_write+0x51/0x90
Mar  4 16:18:56 ARCUCSB2007712M kernel: [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
Mar  4 16:18:56 ARCUCSB2007712M kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Comment 1 Fabio Massimo Di Nitto 2013-03-05 06:32:09 UTC 
(In reply to comment #0)
> Description of problem:
> The RHEL6.4 is installed on the Cisco UCS B200 NIC 1240. Before I create the
> cluster, the OS is working well, but after I create the cluster node from
> luci and within 2 minutes, the cluster service will be no responsed and
> crashed after the cluster is created. The cluster service can not be stopped
> by useing service XXX stop, and the reboot is very very slow and
> unsuccessful. After serveral hours, the cluster service management can be
> closed with errors.
> Actually, the cluster only has the nodes name. There is no possible to add
> other resources because of the service crashed.
> 
> OS info 
> rpm -q corosync cman rgmanager fence-agents gfs2-utils lvm2-cluster
> corosync-1.4.1-15.el6.x86_64
> cman-3.0.12.1-49.el6.x86_64
> rgmanager-3.0.12.1-17.el6.x86_64
> fence-agents-3.1.5-25.el6.x86_64
> gfs2-utils-3.0.12.1-49.el6.x86_64
> lvm2-cluster-2.02.98-9.el6.x86_64
> 
> uname -a
> Linux 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64
> x86_64 x86_64 GNU/Linux
> 
> 
> Error Log:
> Mar  4 16:10:22 ARCUCSB2007712M corosync[4519]:   [TOTEM ] Retransmit List:
> 78 7a 7b 7c 7e 7f 80 82 83
> Mar  4 16:10:24 ARCUCSB2007712M corosync[4519]:   [TOTEM ] Retransmit List:
> 78 7a 7b 7c 7e 7f 80 82 83
> Mar  4 16:10:26 ARCUCSB2007712M corosync[4519]:   [TOTEM ] Retransmit List:
> 78 7a 7b 7c 7e 7f 80 82 83
> Mar  4 16:10:28 ARCUCSB2007712M corosync[4519]:   [TOTEM ] Retransmit List:
> 78 7a 7b 7c 7e 7f 80 82 83
> Mar  4 16:10:28 ARCUCSB2007712M corosync[4519]:   [TOTEM ] FAILED TO RECEIVE
> Mar  4 16:10:30 ARCUCSB2007712M abrt[8051]: File '/usr/sbin/corosync' seems
> to be deleted


This generally indicates a network (multicast) issue between the nodes.

Also, why would abrt reports that corosync has been deleted?
Comment 2 Jan Friesse 2013-03-05 10:27:18 UTC 

*** This bug has been marked as a duplicate of bug 854216 ***
Note You need to log in before you can comment on or make changes to this bug.