Bug 453000 - Rgmanager stops heartbeat interface is Virtual IP is substring match to the broadcast address
Rgmanager stops heartbeat interface is Virtual IP is substring match to the b...
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: rgmanager (Show other bugs)
All Linux
medium Severity medium
: rc
: ---
Assigned To: Lon Hohberger
Cluster QE
: 460541 (view as bug list)
Depends On:
  Show dependency treegraph
Reported: 2008-06-26 10:48 EDT by Shane Bradley
Modified: 2010-10-22 22:15 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2009-01-20 15:56:32 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
rgmanager mismatch ip (472 bytes, patch)
2008-06-26 11:02 EDT, Shane Bradley
no flags Details | Diff

  None (edit)
Description Shane Bradley 2008-06-26 10:48:35 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9) Gecko/2008061712 Fedora/3.0-1.fc9 Firefox/3.0

Description of problem:

Problem description: On a cluster node where there is an rgmanager IP resource that is a substring match to the broadcast address of any interface, stopping or relocating that service will bring down that entire interface.  If this happens to be the interface used for cluster communication then the node misses its heartbeats and gets fenced.

Example: ashprdgfs01 has eth0 =  This means the broadcast address is  Stopping the following resource:

  <ip address="" monitor_link="1"/>

causes the wrong ip address to be removed:

  Jun 23 15:18:27 ashprdgfs01 clurgmgrd[3889]: <notice> Stopping service service:VIP
  Jun 23 15:18:27 ashprdgfs01 clurgmgrd: [3889]: <info> Removing IPv4 address from eth0

Since this is the main ip for eth0, this kills cluster communication and the node gets fenced:

  Jun 23 15:18:42 ashprdgfs02 kernel: dlm: closing connection to node 2
  Jun 23 15:18:42 ashprdgfs02 fenced[3199]: fencing node "ashprdgfs01.gspt.net"

The problem is on line 714 of ip.sh:

  addr=`/sbin/ip addr list | grep "$addr" | head -n 1 | awk '{print $2}'`

Because is a substring of, this actually returns 2 lines:

   inet brd scope global eth0
   inet scope global secondary eth0

and the first one is chosen mistakenly, causing it to be the one removed (718):

   /sbin/ip -f inet addr del dev $dev $addr

The address we want will always be follwed by a '/' for the subnet, so this can be fixed easily like so:

   addr=`/sbin/ip addr list | grep "$addr/" | head -n 1 | awk '{print $2}'`

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1) Configure an interface with a netmask
2) Configure an IP resource on that interface with last octet of .25 or .2
3) Stop the service:

     # clusvcadm -s vip

Actual Results:
VIP service should stop, leaving the main ip on that interface intact. If this is the heartbeat interface then the node loses communication with cluster and node is fenced off.

Expected Results:
The entire interface is taken down.

Additional info:
Workaround is to not use VIP that is substring of broadcast.
Comment 1 Shane Bradley 2008-06-26 11:02:20 EDT
Created attachment 310344 [details]
rgmanager mismatch ip
Comment 2 Lon Hohberger 2008-06-26 13:58:06 EDT

On RHEL4, that line isn't even present.
Comment 3 RHEL Product and Program Management 2008-07-03 14:21:38 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
Comment 4 Fedora Update System 2008-07-30 16:03:55 EDT
gfs2-utils-2.03.05-1.fc9, rgmanager-2.03.05-1.fc9, cman-2.03.05-1.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 5 Lon Hohberger 2008-08-01 12:13:36 EDT
This breaks when netmasks are used.
Comment 6 Lon Hohberger 2008-08-01 12:16:29 EDT
@@ -702,7 +702,7 @@
 			if [ "${addr/\/*/}" != "$ifaddr" ]; then
-			addr=`/sbin/ip addr list | grep "$addr" | head -n 1 | awk '{print $2}'`
+			addr=`/sbin/ip addr list | grep "$addr/" | head -n 1 | awk '{print $2}'`

... should be:

+			addr=`/sbin/ip addr list | grep "$ifaddr/" | head -n 1 | awk '{print $2}'`

Comment 7 Lon Hohberger 2008-08-22 14:04:24 EDT

Ancillary patch has been pushed to RHEL5 branch.
Comment 8 Lon Hohberger 2008-08-22 14:07:06 EDT
Pushed to STABLE2 and master branches as well.
Comment 9 Lon Hohberger 2008-09-03 11:31:15 EDT
*** Bug 460541 has been marked as a duplicate of this bug. ***
Comment 12 errata-xmlrpc 2009-01-20 15:56:32 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.