Bug 1728472 - Engine reports network out of sync due to ipv6 default gateway via ND RA on a non default route network.
Summary: Engine reports network out of sync due to ipv6 default gateway via ND RA on a...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.3.4
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ovirt-4.4.0
: ---
Assignee: Dominik Holler
QA Contact: Michael Burman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-10 04:22 UTC by Germano Veit Michel
Modified: 2020-10-29 15:13 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, the RHV Manager reported network out of sync because the Linux kernel applied the default gateway IPv6 router advertisements, and the IPv6 routing table was not configured by RHV. The current release fixes this issue. The IPv6 routing table is configured by RHV. NetworkManager manages the default gateway IPv6 router advertisements.
Clone Of:
Environment:
Last Closed: 2020-08-04 13:19:49 UTC
oVirt Team: Network
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2020:3247 0 None None None 2020-08-04 13:20:17 UTC

Description Germano Veit Michel 2019-07-10 04:22:00 UTC
Description of problem:

Engine is reporting ovirtmgmt out of sync due to default route, but it is not out of sync.

Comment 7 Donald Berry 2019-07-10 12:41:50 UTC
Hi Michael, IPV6_AUTOCONF is enabled on three networks on two of the hosts (I did not set it):

[root@dell-r640-01 network-scripts]# grep IPV6_AUTOCONF ifcfg*
ifcfg-bond0.1191:IPV6_AUTOCONF=yes
ifcfg-em1:IPV6_AUTOCONF=yes
ifcfg-ovirtmgmt:IPV6_AUTOCONF=yes

[root@dell-r640-03 network-scripts]# grep IPV6_AUTOCONF ifcfg*
ifcfg-bond0.1191:IPV6_AUTOCONF=yes
ifcfg-em1:IPV6_AUTOCONF=yes
ifcfg-ovirtmgmt:IPV6_AUTOCONF=yes

On the other host it is only set on two networks (it is not set on the gluster network bond0.1191):

[root@dell-r640-02 network-scripts]# grep IPV6_AUTOCONF ifcfg*
ifcfg-em1:IPV6_AUTOCONF=yes
ifcfg-ovirtmgmt:IPV6_AUTOCONF=yes

So ovirtmgmt has IPV6_AUTOCONF=yes on all three hosts. None have IPV6_DEFAULTGW set:

[root@dell-r640-01 network-scripts]# grep IPV6_DEFAULTGW ifcfg*
[root@dell-r640-01 network-scripts]# 

[root@dell-r640-02 network-scripts]# grep IPV6_DEFAULTGW ifcfg*
[root@dell-r640-02 network-scripts]# 

[root@dell-r640-03 network-scripts]# grep IPV6_DEFAULTGW ifcfg*
[root@dell-r640-03 network-scripts]# 

Don

Comment 9 Donald Berry 2019-07-10 14:05:19 UTC
Hi Michael, yes ovirtmgmt has an IPv6 address and default route on all three hosts:

[root@dell-r640-01 ~]# ip -6 addr show dev ovirtmgmt
58: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2620:52:0:875:e643:4bff:fe2b:a8c1/64 scope global mngtmpaddr dynamic 
       valid_lft 2591622sec preferred_lft 604422sec
    inet6 fe80::e643:4bff:fe2b:a8c1/64 scope link 
       valid_lft forever preferred_lft forever

[root@dell-r640-02 ~]# ip -6 addr show dev ovirtmgmt
54: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2620:52:0:875:e643:4bff:fe2b:a821/64 scope global mngtmpaddr dynamic 
       valid_lft 2591516sec preferred_lft 604316sec
    inet6 fe80::e643:4bff:fe2b:a821/64 scope link 
       valid_lft forever preferred_lft forever

[root@dell-r640-03 ~]# ip -6 addr show dev ovirtmgmt
53: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2620:52:0:875:e643:4bff:fe2b:a7d1/64 scope global mngtmpaddr dynamic 
       valid_lft 2591942sec preferred_lft 604742sec
    inet6 fe80::e643:4bff:fe2b:a7d1/64 scope link 
       valid_lft forever preferred_lft forever

[root@dell-r640-01 ~]# ip -6 route show dev ovirtmgmt
2620:52:0:875::/64 proto kernel metric 256 expires 2591594sec pref medium
fe80::/64 proto kernel metric 256 pref medium
default via fe80::3e8a:b0ff:fe10:43c1 proto ra metric 1024 expires 1394sec hoplimit 64 pref medium

[root@dell-r640-02 ~]# ip -6 route show dev ovirtmgmt
2620:52:0:875::/64 proto kernel metric 256 expires 2591504sec pref medium
fe80::/64 proto kernel metric 256 pref medium
default via fe80::3e8a:b0ff:fe10:43c1 proto ra metric 1024 expires 1304sec hoplimit 64 pref medium

[root@dell-r640-03 ~]# ip -6 route show dev ovirtmgmt
2620:52:0:875::/64 proto kernel metric 256 expires 2591681sec pref medium
fe80::/64 proto kernel metric 256 pref medium
default via fe80::3e8a:b0ff:fe10:43c1 proto ra metric 1024 expires 1481sec hoplimit 64 pref medium

I did try refreshing capabilities on all hosts, but it made no difference.

I disabled IPv6 on ovirtmgmt on all three hosts via the RHV GUI (Compute/Hosts/<host>/Network Interfaces/Setup Host Networks). I changed ovirtmgmt IPv6 from Stateless Address Autoconfiguration to None. I had to migrate the engine off the HE host.

The networks are now in sync.

Is it recommended to disable IPv6 on the hosts prior to installing RHV?

Thanks.

Don

Comment 10 Dominik Holler 2019-07-10 14:23:11 UTC
(In reply to Donald Berry from comment #9)

> 
> Is it recommended to disable IPv6 on the hosts prior to installing RHV?
> 

IPv6 Autoconf is not supported.
Only all hosts in a cluster with
 - IPv6 static without IPv4 or
 - IPv4 and IPv6 disabled
is supported.


> Thanks.
> 
> Don

Comment 12 Dominik Holler 2019-07-10 14:45:59 UTC
The configuration of IPv6 Stateless Address Autoconfiguration and IPv4 static is the default configuration on ovirtmgmt.
This default configuration should be
- IPv4 something and IPv6 none or
- IPv4 none and IPv6 something 

Germano, Don and Michael would this solve this issue?

Comment 13 Michael Burman 2019-07-10 14:49:55 UTC
(In reply to Dominik Holler from comment #12)
> The configuration of IPv6 Stateless Address Autoconfiguration and IPv4
> static is the default configuration on ovirtmgmt.
> This default configuration should be
> - IPv4 something and IPv6 none or
> - IPv4 none and IPv6 something 
> 
> Germano, Don and Michael would this solve this issue?

Looks like Don confirmed that setting IPv6 to none on ovirtmgmt indeed fix the issue

Comment 14 Germano Veit Michel 2019-07-11 00:25:22 UTC
Hi Dominik and Michael, thanks for quickly jumping in!

(In reply to Dominik Holler from comment #12)
> The configuration of IPv6 Stateless Address Autoconfiguration and IPv4
> static is the default configuration on ovirtmgmt.
> This default configuration should be
> - IPv4 something and IPv6 none or
> - IPv4 none and IPv6 something 
> 
> Germano, Don and Michael would this solve this issue?

Yeah, the problem is solved, but I don't see why one would not have both IPv4 and IPv6 at the same time. In fact, that may be the most common config, no?

It seems to me that VDSM should disable accepting ND router advertisements on interfaces assigned to networks that do not have the default route property set:
/proc/sys/net/ipv6/conf/<netdev>/accept_ra = 0

Wouldn't the correct behavior of a network that has default_route set to NO be:
- ignore default routes via DHCP/DHCPv6
- ignore ND router advertisements

(In reply to Michael Burman from comment #11)
> (In reply to Dominik Holler from comment #10)
> It's not only about supported or not. We have a known issue with default
> route role + IPv6 gateway. 
> If setting a non-mgmt network with the default route(ipv4) role and
> ovirtmgmt has IPv6 gateway, engine think it's out-of-sync becasue of the
> default route role, which is indeed misleading. Engine expect that if the
> network has an IPv6 gateway it is also must have the default route role. As
> i said, a known issue. 
> BTW, although we don't support autoconf officially it is working. 
> The problem in this case is that if network has IPv6 gateway then engine
> think it is also the default route role.

Do we have any other BZ to track this? Or at least to make the error clearer if you do not agree with the above?

Comment 20 Dominik Holler 2019-07-30 13:40:38 UTC
Seems like apply the default route role only on IPv4, but check on creating the icon on IPv4 and IPv6.
Eitan, do you think it would be a good idea to ignore the IPv6 default gateway on calculating the out-of-sync state?

Comment 21 Dominik Holler 2019-08-01 21:45:33 UTC
The proper fix would be to set /proc/sys/net/ipv6/conf/<interface_name>/accept_ra_defrtr, but this seems to be not possible by ifcfg-files.

Comment 22 Dominik Holler 2019-08-01 22:08:08 UTC
(In reply to Dominik Holler from comment #20)
> Eitan, do you think it would be a good idea to ignore the IPv6 default
> gateway on calculating the out-of-sync state?

More precise:
If the interface is the DefaultRouteNetwork and has no IPv4 config, an IPv6 gateway is required on the interface.
But if the interface is not the DefaultRouteNetwork and has an IPv6 gateway, this interface would be in sync.

Comment 23 Dominik Holler 2019-08-05 11:16:49 UTC
Let's ensure that this issue is closed in 4.4.

Comment 24 Dominik Holler 2020-01-17 15:04:14 UTC
simple reproduction scenario:
first NIC: IPv4: dhcp, IPv6: none, default route role
second NIC: IPv4: none, IPv6: auto

on oVirt-4.3: second NIC is out-of-sync because of default route, "ip -6 r | grep default | wc -l" shows 1
on oVirt-4.4: no NIC out-of-sync, "ip -6 r | grep default | wc -l" shows 0

Comment 25 Michael Burman 2020-01-19 09:38:39 UTC
(In reply to Dominik Holler from comment #24)
> simple reproduction scenario:
> first NIC: IPv4: dhcp, IPv6: none, default route role
> second NIC: IPv4: none, IPv6: auto
> 
> on oVirt-4.3: second NIC is out-of-sync because of default route, "ip -6 r |
> grep default | wc -l" shows 1
> on oVirt-4.4: no NIC out-of-sync, "ip -6 r | grep default | wc -l" shows 0

Based on comment 24, moving to verified with nmstate.
Tested u/s with vdsm-4.40.0-1518.gitfde24a6b5.el8.x86_64 and nmstate-0.1.3-1.el8.noarch

Comment 26 RHV bug bot 2020-01-24 19:51:20 UTC
WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 33 errata-xmlrpc 2020-08-04 13:19:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: RHV Manager (ovirt-engine) 4.4 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3247


Note You need to log in before you can comment on or make changes to this bug.