This service will be undergoing maintenance at 03:30 UTC, 2016-05-27. It is expected to last about 2 hours
Bug 453000 - Rgmanager stops heartbeat interface is Virtual IP is substring match to the broadcast address
Rgmanager stops heartbeat interface is Virtual IP is substring match to the b...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: rgmanager (Show other bugs)
5.2.z
All Linux
medium Severity medium
: rc
: ---
Assigned To: Lon Hohberger
Cluster QE
:
: 460541 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-06-26 10:48 EDT by Shane Bradley
Modified: 2010-10-22 22:15 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-20 15:56:32 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
rgmanager mismatch ip (472 bytes, patch)
2008-06-26 11:02 EDT, Shane Bradley
no flags Details | Diff

  None (edit)
Description Shane Bradley 2008-06-26 10:48:35 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9) Gecko/2008061712 Fedora/3.0-1.fc9 Firefox/3.0

Description of problem:

Problem description: On a cluster node where there is an rgmanager IP resource that is a substring match to the broadcast address of any interface, stopping or relocating that service will bring down that entire interface.  If this happens to be the interface used for cluster communication then the node misses its heartbeats and gets fenced.

Example: ashprdgfs01 has eth0 = 172.20.200.21/24.  This means the broadcast address is 172.20.200.255.  Stopping the following resource:

  <ip address="172.20.200.25" monitor_link="1"/>

causes the wrong ip address to be removed:

  Jun 23 15:18:27 ashprdgfs01 clurgmgrd[3889]: <notice> Stopping service service:VIP
  Jun 23 15:18:27 ashprdgfs01 clurgmgrd: [3889]: <info> Removing IPv4 address 172.20.200.21/24 from eth0

Since this is the main ip for eth0, this kills cluster communication and the node gets fenced:

  Jun 23 15:18:42 ashprdgfs02 kernel: dlm: closing connection to node 2
  Jun 23 15:18:42 ashprdgfs02 fenced[3199]: fencing node "ashprdgfs01.gspt.net"

The problem is on line 714 of ip.sh:

  addr=`/sbin/ip addr list | grep "$addr" | head -n 1 | awk '{print $2}'`

Because 172.20.200.25 is a substring of 172.20.200.255, this actually returns 2 lines:

   inet 172.20.200.21/24 brd 172.20.200.255 scope global eth0
   inet 172.20.200.25/24 scope global secondary eth0

and the first one is chosen mistakenly, causing it to be the one removed (718):

   /sbin/ip -f inet addr del dev $dev $addr

The address we want will always be follwed by a '/' for the subnet, so this can be fixed easily like so:

   addr=`/sbin/ip addr list | grep "$addr/" | head -n 1 | awk '{print $2}'`

Version-Release number of selected component (if applicable):
rgmanager-2.0.38-2

How reproducible:
Always


Steps to Reproduce:
1) Configure an interface with a 255.255.255.0 netmask
2) Configure an IP resource on that interface with last octet of .25 or .2
3) Stop the service:

     # clusvcadm -s vip

Actual Results:
VIP service should stop, leaving the main ip on that interface intact. If this is the heartbeat interface then the node loses communication with cluster and node is fenced off.



Expected Results:
The entire interface is taken down.

Additional info:
Workaround is to not use VIP that is substring of broadcast.
Comment 1 Shane Bradley 2008-06-26 11:02:20 EDT
Created attachment 310344 [details]
rgmanager mismatch ip
Comment 2 Lon Hohberger 2008-06-26 13:58:06 EDT
Applied.

On RHEL4, that line isn't even present.
Comment 3 RHEL Product and Program Management 2008-07-03 14:21:38 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 4 Fedora Update System 2008-07-30 16:03:55 EDT
gfs2-utils-2.03.05-1.fc9, rgmanager-2.03.05-1.fc9, cman-2.03.05-1.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 5 Lon Hohberger 2008-08-01 12:13:36 EDT
This breaks when netmasks are used.
Comment 6 Lon Hohberger 2008-08-01 12:16:29 EDT
@@ -702,7 +702,7 @@
 			if [ "${addr/\/*/}" != "$ifaddr" ]; then
 			        continue
 			fi
-			addr=`/sbin/ip addr list | grep "$addr" | head -n 1 | awk '{print $2}'`
+			addr=`/sbin/ip addr list | grep "$addr/" | head -n 1 | awk '{print $2}'`

... should be:

+			addr=`/sbin/ip addr list | grep "$ifaddr/" | head -n 1 | awk '{print $2}'`
                                                         ^^^^^^^^

Comment 7 Lon Hohberger 2008-08-22 14:04:24 EDT
http://git.fedorahosted.org/git/?p=cluster.git;a=commit;h=40e34ea406411c10a53aa1a147a0c94a62c3d12b

Ancillary patch has been pushed to RHEL5 branch.
Comment 8 Lon Hohberger 2008-08-22 14:07:06 EDT
Pushed to STABLE2 and master branches as well.
Comment 9 Lon Hohberger 2008-09-03 11:31:15 EDT
*** Bug 460541 has been marked as a duplicate of this bug. ***
Comment 12 errata-xmlrpc 2009-01-20 15:56:32 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0101.html

Note You need to log in before you can comment on or make changes to this bug.