| Summary: | [RHEV-RHS] Gateway not configured on RHSS Node, after adding it to gluster enabled cluster | ||
|---|---|---|---|
| Product: | Red Hat Gluster Storage | Reporter: | SATHEESARAN <sasundar> |
| Component: | vdsm | Assignee: | Timothy Asir <tjeyasin> |
| Status: | CLOSED ERRATA | QA Contact: | SATHEESARAN <sasundar> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 2.1 | CC: | acathrow, barumuga, dpati, ecohen, grajaiya, iheim, rcyriac, Rhev-m-bugs, rhs-bugs, sabose, sasundar, scohen, tjeyasin, yeylon |
| Target Milestone: | --- | Keywords: | Regression, TestBlocker, ZStream |
| Target Release: | RHGS 2.1.2 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | gluster | ||
| Fixed In Version: | 4.13.0-24 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: |
virt rhev integration
|
|
| Last Closed: | 2014-02-25 08:07:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
I see this issue only with latest RHSS Version glusterfs-3.4.0.44.1u2.el6rhs. Probably something bad happening with latest RHSS 2.1 u2. When I add RHSS 2.1 Update1 Nodes to RHEVM 3.3, all works fine. I would be changing the product and component once I am sure about this issue DEFROUTE key in the ifcfg-rhevm was causing this issue. This key was disabled by default, when I enabled it, and restarted the network, then then default gateway was added and network ping behaved normal.
1. Following are the steps which helped to restore the gateway,
[root@rhss1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-rhevm
# Generated by VDSM version 4.13.0-21.el6rhs
DEVICE=rhevm
ONBOOT=yes
TYPE=Bridge
DELAY=0
BOOTPROTO=dhcp
DEFROUTE=no
NM_CONTROLLED=no
IPV6_AUTOCONF=yes
UUID=7d9ccb0c-e0e9-4a35-85cf-f183180c4169
IPV6INIT=yes
<Edited the ifcfg-rhevm file, and enabled DEFROUTE
[root@rhss1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-rhevm
# Generated by VDSM version 4.13.0-21.el6rhs
DEVICE=rhevm
ONBOOT=yes
TYPE=Bridge
DELAY=0
BOOTPROTO=dhcp
DEFROUTE=yes
NM_CONTROLLED=no
IPV6_AUTOCONF=yes
UUID=7d9ccb0c-e0e9-4a35-85cf-f183180c4169
IPV6INIT=yes
[root@rhss1 ~]# service network restart
Shutting down interface rhevm: [ OK ]
Shutting down interface eth0: [ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: [ OK ]
Bringing up interface rhevm:
Determining IP information for rhevm... done.
[ OK ]
[root@rhss1 ~]# ping google.com
PING google.com (173.194.38.165) 56(84) bytes of data.
64 bytes from sin04s02-in-f5.1e100.net (173.194.38.165): icmp_seq=1 ttl=48 time=119 ms
64 bytes from sin04s02-in-f5.1e100.net (173.194.38.165): icmp_seq=2 ttl=48 time=73.8 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1500ms
rtt min/avg/max/mdev = 73.814/96.618/119.423/22.806 ms
[root@rhss1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
10.70.36.0 0.0.0.0 255.255.254.0 U 0 0 0 rhevm
169.254.0.0 0.0.0.0 255.255.0.0 U 1008 0 0 rhevm
0.0.0.0 10.70.37.254 0.0.0.0 UG 0 0 0 rhevm
RHS 2.1 Update1, when added to gluster-enabled cluster never added "DEFROUTE" to /etc/sysconfig/network-scripts/ifcfg-rhevm
Contents of /etc/sysconfig/network-scripts/ifcfg-rhevm, is given as below :
DEVICE=rhevm
ONBOOT=yes
TYPE=Bridge
DELAY=0
BOOTPROTO=dhcp
NM_CONTROLLED=no
IPV6_AUTOCONF=yes
UUID=2f680125-a400-4f14-a0b6-20bcd7151568
IPV6INIT=yes
Hope that the Bootstrapping code adds this "DEFROUTE" to disabled.
Adding the comment from Bala, <snip> The bridge name is hardcoded in rhs-vdsm. It expects 'ovirtmgmt' bridge name whereas rhevm passes 'rhevm'. I think this has to be fixed in both rhev and rhs side. Regards, Bala </snip> Patch sent to downstream: https://code.engineering.redhat.com/gerrit/18120 This issue is still seen with glusterfs-3.4.0.55rhs-1 There is no default gateway set when adding RHSS Nodes to the gluster-enabled cluster in RHEVM (IS30). So, when RHSS Nodes are in different network ( subnet ), then they can't ping each other. Hence requesting this as a blocker Timothy, could you change to the correct Product/Component for this issue, as I am not sure about that Adding NEED INFO change of Product/Component Is this a regression ? (In reply to Gowrishankar Rajaiyan from comment #11) > Is this a regression ? No (In reply to SATHEESARAN from comment #12) > (In reply to Gowrishankar Rajaiyan from comment #11) > > Is this a regression ? > > No I had a wrong understanding about, what is "regression". Yes, this issue was not seen with RHS 2.1 Update1 and seen with U2 RHSS ISO. Hence adding a regression keyword Tested with RHSS 2.1 Update2 RC1 ISO - RHSS-2.1-20140116.2 VDSM - 4.13.0 1. Added the RHSS Node to gluster enabled cluster in RHEVM RHSS Nodes defaults gateway was set as DEFROUTE for RHEVM was YES [Fri Jan 17 18:00:24 UTC 2014 root@:~ ] # rpm -qa | grep vdsm vdsm-python-4.13.0-24.el6rhs.x86_64 vdsm-xmlrpc-4.13.0-24.el6rhs.noarch vdsm-4.13.0-24.el6rhs.x86_64 vdsm-reg-4.13.0-24.el6rhs.noarch vdsm-gluster-4.13.0-24.el6rhs.noarch vdsm-python-cpopen-4.13.0-24.el6rhs.x86_64 vdsm-cli-4.13.0-24.el6rhs.noarch [Fri Jan 17 18:01:34 UTC 2014 root@:~ ] # cat /etc/sysconfig/network-scripts/ifcfg-rhevm # Generated by VDSM version 4.13.0-24.el6rhs DEVICE=rhevm ONBOOT=yes TYPE=Bridge DELAY=0 BOOTPROTO=dhcp DEFROUTE=yes NM_CONTROLLED=no STP=no [Fri Jan 17 18:10:23 UTC 2014 root@:~ ] # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.70.36.0 0.0.0.0 255.255.254.0 U 0 0 0 rhevm 169.254.0.0 0.0.0.0 255.255.0.0 U 1012 0 0 rhevm 0.0.0.0 10.70.37.254 0.0.0.0 UG 0 0 0 rhevm However, there was a crash report generated after adding RHSS Node to gluster enabled cluster and this defect is reported as a new bug - https://bugzilla.redhat.com/show_bug.cgi?id=1054759 Since the cause for this defect(this bug) is fixed, moving this bug to VERIFIED Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-0208.html |
Description of problem: Network Gateway was not configured after adding RHSS Node to the gluster enabled cluster. As a result, all outbound traffic getting affected. Version-Release number of selected component (if applicable): RHEVM Version - 3.3.0-0.37.beta1.el6ev (IS 25) RHSS Version - glusterfs-3.4.0.44.1u2rhs How reproducible: Always Steps to Reproduce: 1. Create a POSIXFS Datacenter with compatibility 3.3 2. Create a gluster enabled cluster with compatibility 3.3 3. Add RHSS Node to the gluster enabled gluster 4. Get the gateway information using, "route -n" command in RHSS Node Actual results: Gateway is not available in "route -n" command. Expected results: Gateway information available before adding the node the cluster should be preserved. Additional info: 1. Outputs of required commands on RHSS Node before adding it to gluster enabled cluster [root@rhss1 ~]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:b2:bb:da brd ff:ff:ff:ff:ff:ff inet 10.70.37.128/23 brd 10.70.37.255 scope global eth0 3: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 52:54:00:1d:8a:34 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500 link/ether 52:54:00:1d:8a:34 brd ff:ff:ff:ff:ff:ff [root@rhss1 ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@rhss1 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 10.70.36.0 0.0.0.0 255.255.254.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0 0.0.0.0 10.70.37.254 0.0.0.0 UG 0 0 0 eth0 2. After adding the RHSS Node to the cluster, [root@rhss1 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.70.36.0 0.0.0.0 255.255.254.0 U 0 0 0 rhevm 169.254.0.0 0.0.0.0 255.255.0.0 U 1011 0 0 rhevm