Bug 1350883

Summary:	vdscli.connect's heuristic ends up reading the local server address from vdsm config, where it finds the default ipv6-local address of "::".
Product:	[oVirt] vdsm	Reporter:	HWSD <s.danzi>
Component:	General	Assignee:	Edward Haas <edwardh>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Michael Burman <mburman>
Severity:	medium	Docs Contact:
Priority:	high
Version:	4.18.4	CC:	bugs, danken, gklein, mburman, sbonazzo, s.danzi, ylavi
Target Milestone:	ovirt-4.0.2	Flags:	rule-engine: ovirt-4.0.z+ rule-engine: exception+ rule-engine: planning_ack+ rule-engine: devel_ack+ rule-engine: testing_ack+
Target Release:	4.18.9
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-08-12 14:25:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Network	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description HWSD 2016-06-28 15:16:41 UTC

After upgrade from oVirt 3.6 to 4.0 ha broker is not able to connect to storage backend.

vdscli.connect's heuristic ends up reading the local
server address from vdsm config, where it finds the default ipv6-local
address of "::" but IPV6 is disabled.

More details on ovirt-users ml:

http://lists.ovirt.org/pipermail/users/2016-June/040578.html

Comment 1 HWSD 2016-06-28 15:51:35 UTC

[root@ovirt01 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
;vdsmdummy;             8000.000000000000       no
DMZ             8000.40167e3684f1       no              bond0.3
                                                        vnet1
                                                        vnet2
LAN_HAW         8000.40167e3684f1       no              bond0.1
ovirtmgmt               8000.00a1b08b2298       no              enp6s1
                                                        vnet0

[root@ovirt01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 40:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
3: enp6s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovirtmgmt state UP qlen 1000
    link/ether 00:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
4: enp4s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 40:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
5: enp5s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 40:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
6: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.50/24 brd 192.168.1.255 scope global ovirtmgmt
       valid_lft forever preferred_lft forever
7: bond0.1@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master LAN_HAW state UP
    link/ether 40:**:**:**:**:** f1 brd ff:ff:ff:ff:ff:ff
8: LAN_HAW: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 40:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
9: bond0.3@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master DMZ state UP
    link/ether 40:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
10: DMZ: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 40:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
11: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether ca:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
12: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovirtmgmt state UNKNOWN qlen 500
    link/ether fe:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
13: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master DMZ state UNKNOWN qlen 500
    link/ether fe:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
14: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master DMZ state UNKNOWN qlen 500
    link/ether fe:**:**:**:**:** brd ff:ff:ff:ff:ff:ff


[root@ovirt01 ~]# cat /var/lib/vdsm/persistence/netconf/bonds/bond0
{
    "nics": [
        "enp4s0",
        "enp5s0"
    ],
    "options": "mode=2 miimon=100"


[root@ovirt01 ~]# cat /var/lib/vdsm/persistence/netconf/nets/DMZ
{
    "bonding": "bond0",
    "vlan": "3",
    "STP": "no",
    "bridged": "true",
    "mtu": "1500"
}

[root@ovirt01 ~]# cat /var/lib/vdsm/persistence/netconf/nets/LAN_HAW
{
    "bonding": "bond0",
    "vlan": "1",
    "STP": "no",
    "bridged": "true",
    "mtu": "1500"
}

[root@ovirt01 ~]# cat /var/lib/vdsm/persistence/netconf/nets/ovirtmgmt
{
    "nic": "enp6s1",
    "netmask": "255.255.255.0",
    "bootproto": "none",
    "ipaddr": "192.168.1.50",
    "gateway": "192.168.1.254"
}

Comment 2 Edward Haas 2016-06-29 10:45:17 UTC

The loopback interface has IPv6 disabled.
The client uses the address mentioned in vdsm.conf (by default the '::'), but no ipv6 addresses are available on the host, therefore it will fail to establish a connection.

Enabling IPv6 on the lo iface should resolve the issue.

Comment 3 Dan Kenigsberg 2016-06-29 11:36:03 UTC

HWSD, have you done something in particular to disable inet6 on the loopback device? What's your /proc/sys/net/ipv6/conf/default/disable_ipv6 ?

Comment 4 HWSD 2016-06-29 11:56:20 UTC

I don't remember if I did something to disable IPV6 during host setup (last year...).

Enabling inet6 ONLY on lo interface don't solve.
I have to enable inet6 on lo and ovirtmgmt.

[root@ovirt01 ~]# cat /proc/sys/net/ipv6/conf/default/disable_ipv6
1

Comment 5 Edward Haas 2016-06-29 12:05:02 UTC

Have you restarted the vdsmd service? The server needs to listen on IPv6.

Comment 6 HWSD 2016-06-29 12:18:30 UTC

I restarted vdsmd and the host.... I have to manually enable ipv6 on lo and ovirtmgmt.
oVirt manage the network configuration.

I have a crontab hack:

[root@ovirt01 etc]# crontab -l
@reboot echo 'echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6' | /usr/bin/at now+1 minutes
@reboot echo 'echo 0 > /proc/sys/net/ipv6/conf/ovirtmgmt/disable_ipv6' | /usr/bin/at now+1 minutes
@reboot echo '/usr/sbin/route add default gw 192.168.1.254'  | /usr/bin/at now+1 minutes


When atd execute the command all start to work. Whitout any restart.

Comment 7 Michael Burman 2016-07-28 10:31:20 UTC

Verified on - vdsm-4.18.9-1.el7ev.x86_64