When a virtual device such as a VLAN was created with a pre-existing associated ifcfg file, Udev would execute ifup on it. If the ifcfg file was configuring the VLAN device for DHCP network address management, when VDSM executed ifup for the device, dhclient would fail as another dhclient (the one started by the ifup Udev performed) would already be controlling the device. Now, hotplug=no has been added to the ifcfg file for virtual devices, so configuring VLANs with DHCP works properly.
Created attachment 898352[details]
logs_from_host_with_failed_install
Description of problem:
if engine uses VLAN tagged management network (rhevm) and host is being added over VLAN tagged interface using DHCP to obtain its IP, host install fails
Reproducer:
1) data center with bridgeless, VLAN tagged (I used VLANID=172) rhevm
2)
host:
install rhel6.5
set up interface em1.172 with DHCP
cat /etc/sysconfig/network-scripts/ifcfg-em1
DEVICE="em1"
BOOTPROTO="none"
IPV6INIT="no"
MTU="1500"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
cat /etc/sysconfig/network-scripts/ifcfg-em1.172
DEVICE=em1.172
ONBOOT=yes
VLAN=yes
BOOTPROTO=dhcp
DEFROUTE=yes
NM_CONTROLLED=no
3)
add host to rhevm
result: engine loses connectivity to host since the host loses IP
Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Manager Version: 3.4.0-0.21.el6ev
How reproducible:
100%
Actual results:
install fails
Expected results:
install successful
Additional info:
there seem to be 2 problems
1) first is udev spawning dhclient for newly defined em1.172, this can be solved by adding HOTPLUG=no in ifcfg file apuimendo already posted a patch for this http://gerrit.ovirt.org/27980
cat /etc/sysconfig/network-scripts/ifcfg-em1.172
DEVICE=em1.172
ONBOOT=yes
VLAN=yes
BOOTPROTO=dhcp
DEFROUTE=yes
NM_CONTROLLED=no
HOTPLUG=no
2) second issue seems to be that if dhclient is already running on the host, host fails to obtain address (see logs bellow)
logs:
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,105::ifcfg::537::root::(writeConfFile) Writing to file /etc/sysconfig/network-scripts/ifcfg-em1.172 configuration:
# Generated by VDSM version 4.14.7-2.el6ev
DEVICE=em1.172
ONBOOT=yes
VLAN=yes
BOOTPROTO=dhcp
DEFROUTE=yes
NM_CONTROLLED=no
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,169::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,171::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,175::utils::642::root::(execCmd) '/sbin/ip -d link show dev em1.172' (cwd None)
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,177::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,179::utils::642::root::(execCmd) '/sbin/ifup em1.172' (cwd None)
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,207::utils::662::root::(execCmd) FAILED: <err> = 'dhclient(7333) is already running - exiting. \n\nThis version of ISC DHCP is based on the release available\non ftp.isc.org. Features have been added and other changes\nhave been made to the base software release in order to make\nit work better with this distribution.\n\nPlease report for this software via the Red Hat Bugzilla site:\n http://bugzilla.redhat.com\n\nexiting.\n'; <rc> = 1
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,207::utils::642::root::(execCmd) '/sbin/ifdown em1.172' (cwd None)
sourceRoute::DEBUG::2014-05-21 13:13:16,315::sourceRouteThread::20::root::(process_IN_CLOSE_WRITE_filePath) Responding to DHCP response in /var/run/vdsm/sourceRoutes/1400670796
sourceRoute::INFO::2014-05-21 13:13:16,316::sourceRoute::175::root::(remove) Removing gateway - device: em1.172
sourceRoute::DEBUG::2014-05-21 13:13:16,316::utils::642::root::(execCmd) '/sbin/ip rule' (cwd None)
sourceRoute::DEBUG::2014-05-21 13:13:16,326::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
sourceRoute::ERROR::2014-05-21 13:13:16,326::sourceRoute::162::root::(_getRules) Routing rules not found for device em1.172
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,394::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-13::INFO::2014-05-21 13:13:16,395::ifcfg::339::root::(restoreAtomicNetworkBackup) Rolling back logical networks configuration (restoring atomic logical networks backup)
MainProcess|Thread-13::INFO::2014-05-21 13:13:16,395::ifcfg::379::root::(restoreAtomicBackup) Rolling back configuration (restoring atomic backup)
MainProcess|Thread-13::INFO::2014-05-21 13:13:16,395::ifcfg::388::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/ifcfg-em1.172
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,395::utils::642::root::(execCmd) '/sbin/ifup em1.172' (cwd None)
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,510::utils::662::root::(execCmd) FAILED: <err> = '/etc/sysconfig/network-scripts/ifup-eth: line 282: 9426 Terminated /sbin/dhclient ${DHCLIENTARGS} ${DEVICE}\n'; <rc> = 1
MainProcess|Thread-13::ERROR::2014-05-21 13:13:16,511::ifcfg::431::root::(_startDevices) Failed to ifup device em1.172 during rollback.
Traceback (most recent call last):
File "/usr/share/vdsm/netconf/ifcfg.py", line 428, in _startDevices
ifup(dev)
File "/usr/share/vdsm/netconf/ifcfg.py", line 789, in ifup
rc, out, err = _ifup(iface)
File "/usr/share/vdsm/netconf/ifcfg.py", line 778, in _ifup
out[-1] if out else '')
ConfigNetworkError: (29, 'Determining IP information for em1.172... failed.')
MainProcess|Thread-13::ERROR::2014-05-21 13:13:16,511::supervdsmServer::100::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 98, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 202, in setupNetworks
return configNetwork.setupNetworks(networks, bondings, **options)
File "/usr/share/vdsm/configNetwork.py", line 651, in setupNetworks
implicitBonding=True, **d)
File "/usr/share/vdsm/configNetwork.py", line 186, in wrapped
return func(*args, **kwargs)
File "/usr/share/vdsm/configNetwork.py", line 288, in addNetwork
netEnt.configure(**options)
File "/usr/share/vdsm/netmodels.py", line 123, in configure
self.configurator.configureVlan(self, **opts)
File "/usr/share/vdsm/netconf/ifcfg.py", line 89, in configureVlan
ifup(vlan.name, vlan.ipConfig.async)
File "/usr/share/vdsm/netconf/ifcfg.py", line 789, in ifup
rc, out, err = _ifup(iface)
File "/usr/share/vdsm/netconf/ifcfg.py", line 778, in _ifup
out[-1] if out else '')
ConfigNetworkError: (29, 'Determining IP information for em1.172... failed.')
sourceRoute::DEBUG::2014-05-21 13:13:16,522::sourceRouteThread::20::root::(process_IN_CLOSE_WRITE_filePath) Responding to DHCP response in /var/run/vdsm/sourceRoutes/1400670796
sourceRoute::INFO::2014-05-21 13:13:16,526::sourceRouteThread::37::root::(process_IN_CLOSE_WRITE_filePath) interface em1.172 is not a libvirt interface
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHBA-2015-0159.html