Bug 1013611 - [RHSC] "connect: Network is unreachable" on RHS 2.1 nodes connected to RHSC: 2.1.1-0.0.1.master.el6ev
[RHSC] "connect: Network is unreachable" on RHS 2.1 nodes connected to RHSC: ...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: vdsm (Show other bugs)
2.1
Unspecified Unspecified
high Severity high
: ---
: RHGS 2.1.2
Assigned To: Bala.FA
Shruti Sampat
: ZStream
Depends On: 1015009 1068929
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-30 08:54 EDT by Shruti Sampat
Modified: 2015-05-13 12:28 EDT (History)
9 users (show)

See Also:
Fixed In Version: vdsm-4.13.0-17.gitdbbbacd.el6_4
Doc Type: Bug Fix
Doc Text:
Previously, the default gateway was removed after adding the host and the node was not able to reach outside of network due to missing gateway. Now, after adding host, default gateway is set properly and the host is able to reach outside of network.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-02-25 02:46:14 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm logs (3.31 MB, text/x-log)
2013-09-30 08:54 EDT, Shruti Sampat
no flags Details
engine logs (3.74 MB, text/x-log)
2013-09-30 08:58 EDT, Shruti Sampat
no flags Details
host-deploy logs (65.08 KB, text/x-log)
2013-09-30 09:00 EDT, Shruti Sampat
no flags Details
vdsm logs (487.00 KB, text/x-log)
2013-09-30 09:04 EDT, Shruti Sampat
no flags Details
/var/log/messages (1.32 MB, text/plain)
2013-09-30 09:34 EDT, Shruti Sampat
no flags Details

  None (edit)
Description Shruti Sampat 2013-09-30 08:54:59 EDT
Created attachment 805182 [details]
vdsm logs

Description of problem:
-------------------------
When a RHS 2.1 server (updated with latest glusterfs-3.4.1rc1-1.el6.x86_64) is added to a cluster managed by RHSC: 2.1.1-0.0.1.master.el6ev, the message is seen on the storage server when the user tries to ping an IP address that is of a machine other than VMs on the same hypervisor that this storage server resides on.

[root@rhs ~]# ping 10.70.6.202
connect: Network is unreachable

The following is seen in /var/log/messages - 

Sep 30 17:56:04 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:56:04 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:56:04 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:56:16 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:56:16 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:56:16 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:56:33 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:56:33 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:56:33 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:56:47 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:56:47 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:56:47 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:00 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:00 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:00 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:10 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:10 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:10 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:22 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:22 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:22 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:30 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:30 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:30 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:39 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:39 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:39 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:53 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:53 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:53 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:58:12 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:58:12 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:58:12 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.


Version-Release number of selected component (if applicable):
Red Hat Storage Console Version: 2.1.1-0.0.1.master.el6ev 
glusterfs-3.4.1rc1-1.el6.x86_64
vdsm-4.12.0-143.gitee97932.el6.x86_64

How reproducible:
Always.

Steps to Reproduce:
1. Install RHS 2.1 on a VM and update to latest glusterfs and vdsm as mentioned in the versions above.
2. Add it to a cluster via the Console.
3. Try to ping a VM that does not exist on the same hypervisor.

Actual results:
The following message is seen - 

connect: Network is unreachable

Expected results:
The network should work fine on the RHS server.

Additional info:
Comment 1 Shruti Sampat 2013-09-30 08:58:59 EDT
Created attachment 805188 [details]
engine logs
Comment 2 Shruti Sampat 2013-09-30 09:00:54 EDT
Created attachment 805189 [details]
host-deploy logs
Comment 3 Shruti Sampat 2013-09-30 09:04:41 EDT
Created attachment 805194 [details]
vdsm logs
Comment 5 Shruti Sampat 2013-09-30 09:34:23 EDT
Created attachment 805212 [details]
/var/log/messages
Comment 6 Bala.FA 2013-10-03 03:35:54 EDT
I see 'DEFROUTE=no' is set for ifcfg-ovirtmgmt bridge.  This means default route is not set when dhcp is done on the bridge.
Comment 7 Bala.FA 2013-10-03 04:25:32 EDT
Workaround is to comment/remove 'DEROUTE=no' line in ifcfg-ovirtmgmt and do 'service network restart'
Comment 8 Bala.FA 2013-10-03 06:11:56 EDT
The problem is found at vdsm.  Branding downstream vdsm sets 'rhevm' as bridge for the check to set default route.
Comment 9 Bala.FA 2013-10-21 02:35:50 EDT
The fix is merged in downstream and available in cb4 build
Comment 10 Shruti Sampat 2013-10-25 05:34:15 EDT
Bala, could you please explain what the fix is and what can be done to verify it?
Comment 11 Bala.FA 2013-10-25 06:12:21 EDT
Problem: as per comment #8, if other than 'rhevm' bridge name, vdsm doesn't set gateway.

Fix: bridge name is reverted back to ovirtmgmt and gateway is set accordingly
Comment 12 Shruti Sampat 2013-10-28 08:19:06 EDT
Added a RHS server to the engine. Able to ping the server from other machines and able to ping other machines from the server. Marking as VERIFIED.
Comment 13 Shalaka 2014-02-05 06:30:19 EST
Please review the edited doc text and sign off.
Comment 14 Bala.FA 2014-02-06 00:18:35 EST
I updated the doc text.  I feel, it looks good.
Comment 15 Shalaka 2014-02-06 04:33:30 EST
Please review the edited doc text and sign off.
Comment 16 Bala.FA 2014-02-06 04:52:43 EST
Looks good to me
Comment 18 errata-xmlrpc 2014-02-25 02:46:14 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0208.html

Note You need to log in before you can comment on or make changes to this bug.