Hide Forgot
Description of problem: Network Gateway was not configured after adding RHSS Node to the gluster enabled cluster. As a result, all outbound traffic getting affected. Version-Release number of selected component (if applicable): RHEVM Version - 3.3.0-0.37.beta1.el6ev (IS 25) RHSS Version - glusterfs-3.4.0.44.1u2rhs How reproducible: Always Steps to Reproduce: 1. Create a POSIXFS Datacenter with compatibility 3.3 2. Create a gluster enabled cluster with compatibility 3.3 3. Add RHSS Node to the gluster enabled gluster 4. Get the gateway information using, "route -n" command in RHSS Node Actual results: Gateway is not available in "route -n" command. Expected results: Gateway information available before adding the node the cluster should be preserved. Additional info: 1. Outputs of required commands on RHSS Node before adding it to gluster enabled cluster [root@rhss1 ~]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:b2:bb:da brd ff:ff:ff:ff:ff:ff inet 10.70.37.128/23 brd 10.70.37.255 scope global eth0 3: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 52:54:00:1d:8a:34 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500 link/ether 52:54:00:1d:8a:34 brd ff:ff:ff:ff:ff:ff [root@rhss1 ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@rhss1 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 10.70.36.0 0.0.0.0 255.255.254.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0 0.0.0.0 10.70.37.254 0.0.0.0 UG 0 0 0 eth0 2. After adding the RHSS Node to the cluster, [root@rhss1 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.70.36.0 0.0.0.0 255.255.254.0 U 0 0 0 rhevm 169.254.0.0 0.0.0.0 255.255.0.0 U 1011 0 0 rhevm
I see this issue only with latest RHSS Version glusterfs-3.4.0.44.1u2.el6rhs. Probably something bad happening with latest RHSS 2.1 u2. When I add RHSS 2.1 Update1 Nodes to RHEVM 3.3, all works fine. I would be changing the product and component once I am sure about this issue
DEFROUTE key in the ifcfg-rhevm was causing this issue. This key was disabled by default, when I enabled it, and restarted the network, then then default gateway was added and network ping behaved normal. 1. Following are the steps which helped to restore the gateway, [root@rhss1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-rhevm # Generated by VDSM version 4.13.0-21.el6rhs DEVICE=rhevm ONBOOT=yes TYPE=Bridge DELAY=0 BOOTPROTO=dhcp DEFROUTE=no NM_CONTROLLED=no IPV6_AUTOCONF=yes UUID=7d9ccb0c-e0e9-4a35-85cf-f183180c4169 IPV6INIT=yes <Edited the ifcfg-rhevm file, and enabled DEFROUTE [root@rhss1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-rhevm # Generated by VDSM version 4.13.0-21.el6rhs DEVICE=rhevm ONBOOT=yes TYPE=Bridge DELAY=0 BOOTPROTO=dhcp DEFROUTE=yes NM_CONTROLLED=no IPV6_AUTOCONF=yes UUID=7d9ccb0c-e0e9-4a35-85cf-f183180c4169 IPV6INIT=yes [root@rhss1 ~]# service network restart Shutting down interface rhevm: [ OK ] Shutting down interface eth0: [ OK ] Shutting down loopback interface: [ OK ] Bringing up loopback interface: [ OK ] Bringing up interface eth0: [ OK ] Bringing up interface rhevm: Determining IP information for rhevm... done. [ OK ] [root@rhss1 ~]# ping google.com PING google.com (173.194.38.165) 56(84) bytes of data. 64 bytes from sin04s02-in-f5.1e100.net (173.194.38.165): icmp_seq=1 ttl=48 time=119 ms 64 bytes from sin04s02-in-f5.1e100.net (173.194.38.165): icmp_seq=2 ttl=48 time=73.8 ms ^C --- google.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1500ms rtt min/avg/max/mdev = 73.814/96.618/119.423/22.806 ms [root@rhss1 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 10.70.36.0 0.0.0.0 255.255.254.0 U 0 0 0 rhevm 169.254.0.0 0.0.0.0 255.255.0.0 U 1008 0 0 rhevm 0.0.0.0 10.70.37.254 0.0.0.0 UG 0 0 0 rhevm RHS 2.1 Update1, when added to gluster-enabled cluster never added "DEFROUTE" to /etc/sysconfig/network-scripts/ifcfg-rhevm Contents of /etc/sysconfig/network-scripts/ifcfg-rhevm, is given as below : DEVICE=rhevm ONBOOT=yes TYPE=Bridge DELAY=0 BOOTPROTO=dhcp NM_CONTROLLED=no IPV6_AUTOCONF=yes UUID=2f680125-a400-4f14-a0b6-20bcd7151568 IPV6INIT=yes Hope that the Bootstrapping code adds this "DEFROUTE" to disabled.
Adding the comment from Bala, <snip> The bridge name is hardcoded in rhs-vdsm. It expects 'ovirtmgmt' bridge name whereas rhevm passes 'rhevm'. I think this has to be fixed in both rhev and rhs side. Regards, Bala </snip>
Patch sent to downstream: https://code.engineering.redhat.com/gerrit/18120
This issue is still seen with glusterfs-3.4.0.55rhs-1 There is no default gateway set when adding RHSS Nodes to the gluster-enabled cluster in RHEVM (IS30). So, when RHSS Nodes are in different network ( subnet ), then they can't ping each other. Hence requesting this as a blocker Timothy, could you change to the correct Product/Component for this issue, as I am not sure about that
Adding NEED INFO change of Product/Component
Is this a regression ?
(In reply to Gowrishankar Rajaiyan from comment #11) > Is this a regression ? No
(In reply to SATHEESARAN from comment #12) > (In reply to Gowrishankar Rajaiyan from comment #11) > > Is this a regression ? > > No I had a wrong understanding about, what is "regression". Yes, this issue was not seen with RHS 2.1 Update1 and seen with U2 RHSS ISO. Hence adding a regression keyword
Tested with RHSS 2.1 Update2 RC1 ISO - RHSS-2.1-20140116.2 VDSM - 4.13.0 1. Added the RHSS Node to gluster enabled cluster in RHEVM RHSS Nodes defaults gateway was set as DEFROUTE for RHEVM was YES [Fri Jan 17 18:00:24 UTC 2014 root@:~ ] # rpm -qa | grep vdsm vdsm-python-4.13.0-24.el6rhs.x86_64 vdsm-xmlrpc-4.13.0-24.el6rhs.noarch vdsm-4.13.0-24.el6rhs.x86_64 vdsm-reg-4.13.0-24.el6rhs.noarch vdsm-gluster-4.13.0-24.el6rhs.noarch vdsm-python-cpopen-4.13.0-24.el6rhs.x86_64 vdsm-cli-4.13.0-24.el6rhs.noarch [Fri Jan 17 18:01:34 UTC 2014 root@:~ ] # cat /etc/sysconfig/network-scripts/ifcfg-rhevm # Generated by VDSM version 4.13.0-24.el6rhs DEVICE=rhevm ONBOOT=yes TYPE=Bridge DELAY=0 BOOTPROTO=dhcp DEFROUTE=yes NM_CONTROLLED=no STP=no [Fri Jan 17 18:10:23 UTC 2014 root@:~ ] # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.70.36.0 0.0.0.0 255.255.254.0 U 0 0 0 rhevm 169.254.0.0 0.0.0.0 255.255.0.0 U 1012 0 0 rhevm 0.0.0.0 10.70.37.254 0.0.0.0 UG 0 0 0 rhevm However, there was a crash report generated after adding RHSS Node to gluster enabled cluster and this defect is reported as a new bug - https://bugzilla.redhat.com/show_bug.cgi?id=1054759 Since the cause for this defect(this bug) is fixed, moving this bug to VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-0208.html