Bug 1013611 - [RHSC] "connect: Network is unreachable" on RHS 2.1 nodes connected to RHSC: 2.1.1-0.0.1.master.el6ev
Summary: [RHSC] "connect: Network is unreachable" on RHS 2.1 nodes connected to RHSC: ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: vdsm
Version: 2.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: RHGS 2.1.2
Assignee: Bala.FA
QA Contact: Shruti Sampat
URL:
Whiteboard:
Depends On: 1015009 1068929
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-30 12:54 UTC by Shruti Sampat
Modified: 2015-05-13 16:28 UTC (History)
9 users (show)

Fixed In Version: vdsm-4.13.0-17.gitdbbbacd.el6_4
Doc Type: Bug Fix
Doc Text:
Previously, the default gateway was removed after adding the host and the node was not able to reach outside of network due to missing gateway. Now, after adding host, default gateway is set properly and the host is able to reach outside of network.
Clone Of:
Environment:
Last Closed: 2014-02-25 07:46:14 UTC
Embargoed:


Attachments (Terms of Use)
vdsm logs (3.31 MB, text/x-log)
2013-09-30 12:54 UTC, Shruti Sampat
no flags Details
engine logs (3.74 MB, text/x-log)
2013-09-30 12:58 UTC, Shruti Sampat
no flags Details
host-deploy logs (65.08 KB, text/x-log)
2013-09-30 13:00 UTC, Shruti Sampat
no flags Details
vdsm logs (487.00 KB, text/x-log)
2013-09-30 13:04 UTC, Shruti Sampat
no flags Details
/var/log/messages (1.32 MB, text/plain)
2013-09-30 13:34 UTC, Shruti Sampat
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2014:0208 0 normal SHIPPED_LIVE Red Hat Storage 2.1 enhancement and bug fix update #2 2014-02-25 12:20:30 UTC

Description Shruti Sampat 2013-09-30 12:54:59 UTC
Created attachment 805182 [details]
vdsm logs

Description of problem:
-------------------------
When a RHS 2.1 server (updated with latest glusterfs-3.4.1rc1-1.el6.x86_64) is added to a cluster managed by RHSC: 2.1.1-0.0.1.master.el6ev, the message is seen on the storage server when the user tries to ping an IP address that is of a machine other than VMs on the same hypervisor that this storage server resides on.

[root@rhs ~]# ping 10.70.6.202
connect: Network is unreachable

The following is seen in /var/log/messages - 

Sep 30 17:56:04 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:56:04 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:56:04 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:56:16 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:56:16 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:56:16 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:56:33 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:56:33 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:56:33 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:56:47 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:56:47 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:56:47 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:00 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:00 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:00 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:10 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:10 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:10 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:22 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:22 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:22 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:30 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:30 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:30 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:39 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:39 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:39 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:57:53 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:57:53 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:57:53 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.
Sep 30 17:58:12 rhs dhclient[21701]: DHCPREQUEST on ovirtmgmt to 10.70.34.2 port 67 (xid=0x624825bc)
Sep 30 17:58:12 rhs dhclient[21701]: send_packet: Network is unreachable
Sep 30 17:58:12 rhs dhclient[21701]: send_packet: please consult README file regarding broadcast address.


Version-Release number of selected component (if applicable):
Red Hat Storage Console Version: 2.1.1-0.0.1.master.el6ev 
glusterfs-3.4.1rc1-1.el6.x86_64
vdsm-4.12.0-143.gitee97932.el6.x86_64

How reproducible:
Always.

Steps to Reproduce:
1. Install RHS 2.1 on a VM and update to latest glusterfs and vdsm as mentioned in the versions above.
2. Add it to a cluster via the Console.
3. Try to ping a VM that does not exist on the same hypervisor.

Actual results:
The following message is seen - 

connect: Network is unreachable

Expected results:
The network should work fine on the RHS server.

Additional info:

Comment 1 Shruti Sampat 2013-09-30 12:58:59 UTC
Created attachment 805188 [details]
engine logs

Comment 2 Shruti Sampat 2013-09-30 13:00:54 UTC
Created attachment 805189 [details]
host-deploy logs

Comment 3 Shruti Sampat 2013-09-30 13:04:41 UTC
Created attachment 805194 [details]
vdsm logs

Comment 5 Shruti Sampat 2013-09-30 13:34:23 UTC
Created attachment 805212 [details]
/var/log/messages

Comment 6 Bala.FA 2013-10-03 07:35:54 UTC
I see 'DEFROUTE=no' is set for ifcfg-ovirtmgmt bridge.  This means default route is not set when dhcp is done on the bridge.

Comment 7 Bala.FA 2013-10-03 08:25:32 UTC
Workaround is to comment/remove 'DEROUTE=no' line in ifcfg-ovirtmgmt and do 'service network restart'

Comment 8 Bala.FA 2013-10-03 10:11:56 UTC
The problem is found at vdsm.  Branding downstream vdsm sets 'rhevm' as bridge for the check to set default route.

Comment 9 Bala.FA 2013-10-21 06:35:50 UTC
The fix is merged in downstream and available in cb4 build

Comment 10 Shruti Sampat 2013-10-25 09:34:15 UTC
Bala, could you please explain what the fix is and what can be done to verify it?

Comment 11 Bala.FA 2013-10-25 10:12:21 UTC
Problem: as per comment #8, if other than 'rhevm' bridge name, vdsm doesn't set gateway.

Fix: bridge name is reverted back to ovirtmgmt and gateway is set accordingly

Comment 12 Shruti Sampat 2013-10-28 12:19:06 UTC
Added a RHS server to the engine. Able to ping the server from other machines and able to ping other machines from the server. Marking as VERIFIED.

Comment 13 Shalaka 2014-02-05 11:30:19 UTC
Please review the edited doc text and sign off.

Comment 14 Bala.FA 2014-02-06 05:18:35 UTC
I updated the doc text.  I feel, it looks good.

Comment 15 Shalaka 2014-02-06 09:33:30 UTC
Please review the edited doc text and sign off.

Comment 16 Bala.FA 2014-02-06 09:52:43 UTC
Looks good to me

Comment 18 errata-xmlrpc 2014-02-25 07:46:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0208.html


Note You need to log in before you can comment on or make changes to this bug.