Bug 1120823

Summary: Network assignment to dummy interface lost after vdsm restart
Product: [Retired] oVirt Reporter: Markus Stockhausen <mst>
Component: vdsmAssignee: Dan Kenigsberg <danken>
Status: CLOSED WONTFIX QA Contact: Gil Klein <gklein>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.4CC: acathrow, bazulay, bugs, danken, ecohen, gklein, iheim, lvernia, mgoldboi, mst, myakove, yeylon
Target Milestone: ---Keywords: Triaged
Target Release: 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: network
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-08-03 13:00:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine-log
none
vdsm start log
none
supervdsm log
none
vdscaps
none
dummy interface webadmin none

Description Markus Stockhausen 2014-07-17 19:14:25 UTC
Description of problem:

Building virtual interfaces for test cases based on dummy networks get lost in the engine after host reboot.


Version-Release number of selected component (if applicable):

Ovirt-engine 3.4.2
vdsm 4.14.9


How reproducible:
100%

Steps to Reproduce:

1. activate dummy kernel module on hypervisor node

2. create hook /usr/libexec/vdsm/hooks/before_vdsm_start/network.sh on Hypervisor node. This ensures the interface is activated, always comes up with the same mac address and is active.

#!/bin/sh
ip link add dummy0 type dummy 2>/dev/null
/usr/sbin/ifconfig dummy0 hw ether 62:dc:0e:86:dd:47
/usr/sbin/ifconfig dummy0 up

3. Activate node in ovirt webadmin

4. Assing Network to dummy0 interface on node. Save config

5. Switch node to maintenance mode

6. reboot node

7. Activate node

Actual results:

Assignment Network/dummy is lost


Expected results:

assignment should still be active 

Additional info:

engine-log attached

Comment 1 Markus Stockhausen 2014-07-17 19:16:55 UTC
Created attachment 918799 [details]
engine-log

Comment 2 Markus Stockhausen 2014-07-17 19:20:21 UTC
Extra info. Although the network assignment gets lost in the engine, it is still visible in the node configuration files:

root@node# cd /etc/sysconfig/network-scripts

root@node# cat ifcfg-VTrust
# Generated by VDSM version 4.14.9-0.fc20
DEVICE=VTrust
ONBOOT=yes
TYPE=Bridge
DELAY=0
DEFROUTE=no
NM_CONTROLLED=no
STP=no
HOTPLUG=no

root@node# cat ifcfg-dummy0
# Generated by VDSM version 4.14.9-0.fc20
DEVICE=dummy0
ONBOOT=yes
HWADDR=62:dc:0e:86:dd:47
BRIDGE=VTrust
NM_CONTROLLED=no
STP=no

Comment 3 Lior Vernia 2014-07-20 08:57:03 UTC
Any chance to get vdsm and supervdsm logs?

Comment 4 Markus Stockhausen 2014-07-20 12:25:08 UTC
Testcase coul dbe narrowed down to vdsmd restart. To reproduce the error I executed the following steps.

1. Install vdsm hook /usr/libexec/vdsm/hooks/before_vdsm_start/network.sh

ip link add dummy0 type dummy 2>/dev/null
/usr/sbin/ifconfig dummy0 hw ether 62:dc:0e:86:dd:47
/usr/sbin/ifconfig dummy0 up

2. Activate node

3. assign network VTrust (self choosen name) to dummy0

4. Put node to maintenance mode

5. Stop vdsmd 

systemctl stop vdsmd

6. reinstall kernel dummy module

rmmod dummy
modprobe dummy

-> dummy0 interface has some random MAC address

ifconfig dummy0
dummy0: flags=130<BROADCAST,NOARP>  mtu 1500
        ether 0e:87:24:5b:bc:49  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


7. start vdsmd

systemctl start vdsmd

-> dummy0 interface has well defined MAC address and is up

ifconfig dummy0
dummy0: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet6 fe80::60dc:eff:fe86:dd47  prefixlen 64  scopeid 0x20<link>
        ether 62:dc:0e:86:dd:47  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 210 (210.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


8. activate host

assignment dummy0 -> VTrust is lost


I have attached the required logs.

Comment 5 Markus Stockhausen 2014-07-20 12:26:22 UTC
Created attachment 919387 [details]
vdsm start log

Comment 6 Markus Stockhausen 2014-07-20 12:26:51 UTC
Created attachment 919388 [details]
supervdsm log

Comment 7 Markus Stockhausen 2014-07-20 12:29:20 UTC
vdsm log contains only the startup routine (step 7 and on).

supervdsm is complete log.

Comment 8 Dan Kenigsberg 2014-07-23 08:41:53 UTC
Could the problem be that nothing re-creates the dummy devices post boot?

You should do 
  ip link add dummy0 type dummy
before vdsm starts (possibly in the before_vdsm_start hook).

If not, could you report the output of `vdsClient -s 0 getVdsCaps`?

Comment 9 Markus Stockhausen 2014-07-23 10:20:14 UTC
Shouldn't this question be answered by Comment 4?

Nevertheless I attached the requested output to this bug. While the getVdsCaps command was executed the engined had no network assignment. Screenshot attached too.

Comment 10 Markus Stockhausen 2014-07-23 10:21:01 UTC
Created attachment 920192 [details]
vdscaps

Comment 11 Markus Stockhausen 2014-07-23 10:22:12 UTC
Created attachment 920193 [details]
dummy interface webadmin

Comment 12 Dan Kenigsberg 2014-07-23 12:34:58 UTC
Yes, you were very specific in comment 4; minor dyslexia on my part part, sorry.

I do notice empty ports list on the bridge

                   'VTrust': {'addr': '',
                              'cfg': {'DEFROUTE': 'no',
                                      'DELAY': '0',
                                      'DEVICE': 'VTrust',
                                      'HOTPLUG': 'no',
                                      'NM_CONTROLLED': 'no',
                                      'ONBOOT': 'yes',
                                      'STP': 'no',
                                      'TYPE': 'Bridge'},
                              'gateway': '',
                              'ipv6addrs': ['fe80::60dc:eff:fe86:dd47/64'],
                              'ipv6gateway': '::',
                              'mtu': '1500',
                              'netmask': '',
                              'ports': [],
                              'stp': 'off'},

apparently because you do not run `ifup dummy0` in your hook. You can also add it to the bridge manually with `brctl addif VTrust dummy0`.

Does this solve the issue for you?

Comment 13 Markus Stockhausen 2014-07-23 17:53:31 UTC
Thanks for that hint.

Replacing "iconfig dummy0 up" with "ifup dummy0" lets the network associations appear after node restart. Difference is that the second one starts the bridge.

So we come to the conclusion of it all.

What will change if we jump into the new 3.5 network configuration logic? Will  VDSM handle those associations itself or should I file a RFE?

Comment 14 Dan Kenigsberg 2014-07-23 21:44:07 UTC
I must admit that I do not understand what you would like vdsm to do in 3.5; dummies are not supported by Vdsm, they are just tolerated, and this is not going to change in 3.5.

Comment 15 Markus Stockhausen 2014-08-03 13:00:02 UTC
online again.

Thanks for yout help and that clarification. I asked myself if a tighter integration would be useful. If "tolerated" dummy interfaces will still work in 3.5 I'm fine with that.

So closing this bug.