Bug 1256252 - Vdsm should recover ifcfg files in case they are no longer exist and recover all networks on the server
Summary: Vdsm should recover ifcfg files in case they are no longer exist and recover ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.6.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-3.6.0-rc
: 3.6.0
Assignee: Ondřej Svoboda
QA Contact: Michael Burman
URL:
Whiteboard:
Depends On:
Blocks: 1263979
TreeView+ depends on / blocked
 
Reported: 2015-08-24 07:48 UTC by Michael Burman
Modified: 2016-03-09 19:43 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1263979 (view as bug list)
Environment:
Last Closed: 2016-03-09 19:43:50 UTC
oVirt Team: Network
Target Upstream Version:
ylavi: Triaged+


Attachments (Terms of Use)
Logs (1.25 MB, application/x-gzip)
2015-08-24 07:48 UTC, Michael Burman
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0362 normal SHIPPED_LIVE vdsm 3.6.0 bug fix and enhancement update 2016-03-09 23:49:32 UTC
oVirt gerrit 45893 master MERGED ifcfg: make removeNic cope with a missing ifcfg file Never
oVirt gerrit 45932 ovirt-3.6 MERGED ifcfg: make removeNic cope with a missing ifcfg file Never
oVirt gerrit 45933 ovirt-3.5 MERGED ifcfg: make removeNic cope with a missing ifcfg file Never
oVirt gerrit 45983 master MERGED ifcfg: only determine and write HWADDR in removeNic if enabled Never
oVirt gerrit 46197 master MERGED ifcfg: only deal gracefully with ENOENT in removeNic Never
oVirt gerrit 46280 ovirt-3.6 MERGED ifcfg: only deal gracefully with ENOENT in removeNic Never
oVirt gerrit 46281 ovirt-3.6 MERGED ifcfg: only determine and write HWADDR in removeNic if enabled Never
oVirt gerrit 46302 ovirt-3.5 MERGED ifcfg: only deal gracefully with ENOENT in removeNic Never
oVirt gerrit 46303 ovirt-3.5 MERGED ifcfg: only determine and write HWADDR in removeNic if enabled Never

Description Michael Burman 2015-08-24 07:48:40 UTC
Created attachment 1066236 [details]
Logs

Description of problem:
Vdsm should recover ifcfg files in case there are no longer exist and recover all networks on the server.

In case that ifcfg files are removed/missing/deleted for any reason, vdsm should recover them on boot and restore all networks.


Version-Release number of selected component (if applicable):
3.6.0-0.12.master.el6
vdsm-4.17.3-1.el7ev.noarch

Steps to Reproduce:
1. Install clean rhel 7.2 in rhev-m 3.6 latest
2. Configure some networks on host via Setup Networks
3. Delete all ifcfg files, except the ifcfg-lo, ifcfg-ovirtmgmt and the NICs ifcfg that the management network attached to
4. Reboot server

Actual results:
All networks are missing, except the management network, host got ip, but is in non-responsive state. 
Vdsm didn't recovered the ifcfg files on boot and couldn't restore networks on server.

Expected results:
Vdsm should recover ifcfg files in case they are missing and restore all networks.

Comment 1 Ido Barkan 2015-08-31 06:49:35 UTC
This happens since the network is still stored in libvirt and the ifcfg file is missing. the ifcfg configurator can't handle a missing nic ifcfg file when it tries to delete it:

Traceback (most recent call last):
  File "/usr/share/vdsm/network/api.py", line 890, in setupNetworks
    configurator=configurator)
  File "/usr/share/vdsm/network/api.py", line 397, in _delBrokenNetwork
    implicitBonding=False, _netinfo=_netinfo)
  File "/usr/share/vdsm/network/api.py", line 219, in wrapped
    ret = func(**attrs)
  File "/usr/share/vdsm/network/api.py", line 486, in _delNetwork
    net_ent_to_remove.remove()
  File "/usr/share/vdsm/network/models.py", line 188, in remove
    self.configurator.removeBridge(self)
  File "/usr/share/vdsm/network/configurators/ifcfg.py", line 182, in removeBridge
    bridge.port.remove()
  File "/usr/share/vdsm/network/models.py", line 100, in remove
    self.configurator.removeNic(self)
  File "/usr/share/vdsm/network/configurators/ifcfg.py", line 231, in removeNic
    self.configApplier.removeNic(nic.name)
  File "/usr/share/vdsm/network/configurators/ifcfg.py", line 644, in removeNic
    with open(cf) as nicFile:
IOError: [Errno 2] No such file or directory: u'/etc/sysconfig/network-scripts/ifcfg-ens1f1'

I assume this is not a 3.6 regression, and the configurator never had the ability to handle such an evil situation where someone tries to remove a physical nic ifcfg.
Lowering urgency.

Comment 2 Ondřej Svoboda 2015-09-09 10:58:38 UTC
A similar outcome happens when a placeholder file with "# original file did not exist" is placed instead of e.g. ifcfg-eth0 into /var/lib/vdsm/netconfback/ and a host is rebooted.

I have created a patch that lets a configurator continue gracefully (and even read HWADDR from the system).

Comment 4 Michael Burman 2015-10-06 14:10:07 UTC
Verified on - 3.6.0-0.18.el6 with vdsm-4.17.8-1.el7ev.noarch

Comment 7 errata-xmlrpc 2016-03-09 19:43:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0362.html


Note You need to log in before you can comment on or make changes to this bug.