The xend init script hangs the network when starting up, causing multicast traffic to be lost, resulting in confused cluster nodes. Seems to be the same problem (or related to it) discussed here: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=431 When the xend service is enabled at boot, it starts after cman, and hangs the network long enough to cause the cluster to become unstable. On the three-node cluster we're testing with, each of the three nodes reports a different view of cluster membership. If xend is disabled at boot time, or xend is enabled, and cman is disabled, but started after all nodes have booted, the problem does not occur. 'cman_tool nodes' output when xend and cman services are enabled at boot: [root@huey ~]# cman_tool nodes NOTE: There are 1 disallowed nodes, members list may seem inconsistent across the cluster Node Sts Inc Joined Name 1 X 12 louey.lab.boston.redhat.com 2 M 4 2007-03-02 15:49:44 huey.lab.boston.redhat.com 3 d 12 2007-03-02 15:49:44 dewey.lab.boston.redhat.com [root@dewey ~]# cman_tool nodes Node Sts Inc Joined Name 1 X 8 louey.lab.boston.redhat.com 2 M 12 2007-03-02 15:49:42 huey.lab.boston.redhat.com 3 M 4 2007-03-02 15:49:39 dewey.lab.boston.redhat.com [root@louey ~]# cman_tool nodes Node Sts Inc Joined Name 1 M 4 2001-12-31 19:00:53 louey.lab.boston.redhat.com 2 M 12 2001-12-31 19:00:55 huey.lab.boston.redhat.com 3 M 8 2001-12-31 19:00:53 dewey.lab.boston.redhat.com I'll attach /var/log/messages output for each of the three nodes.
Created attachment 149150 [details] logs from huey.lab.boston.redhat.com
Created attachment 149151 [details] logs from dewey.lab.boston.redhat.com
Created attachment 149152 [details] logs from louey.lab.boston.redhat.com
Created attachment 149153 [details] cluster.conf file used for the cluster
Can you please supply "ifconfig" and "ip route" output, both after a successful (non-Xen) boot, and after xend has started?
Before starting xend: [root@huey ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:04:23:B5:3F:DA inet addr:192.168.77.141 Bcast:192.168.79.255 Mask:255.255.252.0 inet6 addr: fe80::204:23ff:feb5:3fda/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1690 errors:0 dropped:0 overruns:0 frame:0 TX packets:1476 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:393906 (384.6 KiB) TX bytes:271689 (265.3 KiB) Base address:0xd880 Memory:fcfa0000-fcfc0000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:81 errors:0 dropped:0 overruns:0 frame:0 TX packets:81 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:9762 (9.5 KiB) TX bytes:9762 (9.5 KiB) [root@huey ~]# ip route 192.168.76.0/22 dev eth0 proto kernel scope link src 192.168.77.141 169.254.0.0/16 dev eth0 scope link default via 192.168.79.254 dev eth0 ----- [root@dewey ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:04:23:B5:41:1E inet addr:192.168.77.142 Bcast:192.168.79.255 Mask:255.255.252.0 inet6 addr: fe80::204:23ff:feb5:411e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2140 errors:0 dropped:0 overruns:0 frame:0 TX packets:1540 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:455460 (444.7 KiB) TX bytes:263000 (256.8 KiB) Base address:0xd880 Memory:fcfa0000-fcfc0000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:148 errors:0 dropped:0 overruns:0 frame:0 TX packets:148 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:18740 (18.3 KiB) TX bytes:18740 (18.3 KiB) [root@dewey ~]# ip route 192.168.76.0/22 dev eth0 proto kernel scope link src 192.168.77.142 169.254.0.0/16 dev eth0 scope link default via 192.168.79.254 dev eth0 ----- [root@louey ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:04:23:B5:47:70 inet addr:192.168.77.143 Bcast:192.168.79.255 Mask:255.255.252.0 inet6 addr: fe80::204:23ff:feb5:4770/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2267 errors:0 dropped:0 overruns:0 frame:0 TX packets:1610 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:476602 (465.4 KiB) TX bytes:275827 (269.3 KiB) Base address:0xd880 Memory:fcfa0000-fcfc0000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:81 errors:0 dropped:0 overruns:0 frame:0 TX packets:81 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:9762 (9.5 KiB) TX bytes:9762 (9.5 KiB) [root@louey ~]# ip route 192.168.76.0/22 dev eth0 proto kernel scope link src 192.168.77.143 169.254.0.0/16 dev eth0 scope link default via 192.168.79.254 dev eth0 After starting xend: [root@huey ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:04:23:B5:3F:DA inet addr:192.168.77.141 Bcast:192.168.79.255 Mask:255.255.252.0 inet6 addr: fe80::204:23ff:feb5:3fda/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:449 errors:0 dropped:0 overruns:0 frame:0 TX packets:117 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:57429 (56.0 KiB) TX bytes:21966 (21.4 KiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:81 errors:0 dropped:0 overruns:0 frame:0 TX packets:81 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:9762 (9.5 KiB) TX bytes:9762 (9.5 KiB) peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:2965 errors:0 dropped:0 overruns:0 frame:0 TX packets:2297 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:569858 (556.5 KiB) TX bytes:422530 (412.6 KiB) Base address:0xd880 Memory:fcfa0000-fcfc0000 vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:119 errors:0 dropped:0 overruns:0 frame:0 TX packets:449 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:22370 (21.8 KiB) TX bytes:57429 (56.0 KiB) xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:310 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:34365 (33.5 KiB) TX bytes:0 (0.0 b) [root@huey ~]# ip route 192.168.76.0/22 dev eth0 proto kernel scope link src 192.168.77.141 169.254.0.0/16 dev eth0 scope link default via 192.168.79.254 dev eth0 ----- [root@dewey ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:04:23:B5:41:1E inet addr:192.168.77.142 Bcast:192.168.79.255 Mask:255.255.252.0 inet6 addr: fe80::204:23ff:feb5:411e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:219 errors:0 dropped:0 overruns:0 frame:0 TX packets:64 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:30235 (29.5 KiB) TX bytes:11832 (11.5 KiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:148 errors:0 dropped:0 overruns:0 frame:0 TX packets:148 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:18740 (18.3 KiB) TX bytes:18740 (18.3 KiB) peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:3605 errors:0 dropped:0 overruns:0 frame:0 TX packets:2193 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:668610 (652.9 KiB) TX bytes:385786 (376.7 KiB) Base address:0xd880 Memory:fcfa0000-fcfc0000 vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:66 errors:0 dropped:0 overruns:0 frame:0 TX packets:219 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:12236 (11.9 KiB) TX bytes:30235 (29.5 KiB) xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:172 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:19359 (18.9 KiB) TX bytes:0 (0.0 b) [root@dewey ~]# ip route 192.168.76.0/22 dev eth0 proto kernel scope link src 192.168.77.142 169.254.0.0/16 dev eth0 scope link default via 192.168.79.254 dev eth0 ----- [root@louey ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:04:23:B5:47:70 inet addr:192.168.77.143 Bcast:192.168.79.255 Mask:255.255.252.0 inet6 addr: fe80::204:23ff:feb5:4770/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:172 errors:0 dropped:0 overruns:0 frame:0 TX packets:54 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:22707 (22.1 KiB) TX bytes:8838 (8.6 KiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:81 errors:0 dropped:0 overruns:0 frame:0 TX packets:81 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:9762 (9.5 KiB) TX bytes:9762 (9.5 KiB) peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:4013 errors:0 dropped:0 overruns:0 frame:0 TX packets:2433 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:730724 (713.5 KiB) TX bytes:428606 (418.5 KiB) Base address:0xd880 Memory:fcfa0000-fcfc0000 vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:56 errors:0 dropped:0 overruns:0 frame:0 TX packets:172 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:9242 (9.0 KiB) TX bytes:22707 (22.1 KiB) xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:136 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:14485 (14.1 KiB) TX bytes:0 (0.0 b) [root@louey ~]# ip route 192.168.76.0/22 dev eth0 proto kernel scope link src 192.168.77.143 169.254.0.0/16 dev eth0 scope link default via 192.168.79.254 dev eth0
Fixing Product Name. Cluster Suite components were integrated into Enterprise Linux version 5.0.
Created attachment 153296 [details] patch to work around xend bridged networking brain damage I've attached a patch to work around the xend network-bridge issues. I modified the cman init script to check whether xend will start and if it will start and is configured to use bridged networking. If both conditions are satisfied, the cman init script will run '/etc/xen/scripts/network-bridge start' before doing anything else. xend will do the same thing when it starts, but it'll essentially be a noop. As far as I can tell, nothing needs to be done when cman stops.
Ryan Since your working issue I'm reassigning to you. Thanks -steve
The workaround (posted above) has been committed to CVS now.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2007-0575.html