Description of problem: Red Hat Enterprise Virtualization Hypervisor 7.1 (20150304.0.el7ev) Install RHEV-H 20150304.0.el7 as autoinstall with the following params: firstboot storage_init=/dev/sda adminpw=RHhwCLrQXB8zE management_server=192.168.122.166 BOOTIF=01-52-54-00-73-3e-08 BOOT_IMAGE=vmlinuz0 After completed install rhevm interface is created but no ip address is assigned and /etc/ifcfg-rhevm is removed as well. If I run dhclient manually and host get the ip, the host immediately appears in RHEV-M for approval. This behavior is only happening when using BOOTIF=Macaddress as showed above if used BOOTIF=interface it works nice. Data ============ vdsm-4.16.8.1-7.el7ev.x86_64 ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::5054:ff:fe73:3e08 prefixlen 64 scopeid 0x20<link> ether 52:54:00:73:3e:08 txqueuelen 1000 (Ethernet) RX packets 1784 bytes 82659 (80.7 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 32 bytes 2974 (2.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 11613 bytes 1701327 (1.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 11613 bytes 1701327 (1.6 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 rhevm: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::5054:ff:fe73:3e08 prefixlen 64 scopeid 0x20<link> ether 52:54:00:73:3e:08 txqueuelen 0 (Ethernet) RX packets 1744 bytes 80224 (78.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 # cat /etc/sysconfig/network-scripts/ifcfg-ens3 # Generated by VDSM version 4.16.8.1-7.el7ev DEVICE=ens3 HWADDR=52:54:00:73:3e:08 BRIDGE=rhevm ONBOOT=yes NM_CONTROLLED=no IPV6_AUTOCONF=no PEERNTP=yes IPV6INIT=no # ls /etc/sysconfig/network-scripts/ifcfg-* /etc/sysconfig/network-scripts/ifcfg-ens3 /etc/sysconfig/network-scripts/ifcfg-lo # ls /config/etc/sysconfig/network-scripts/ifcfg-* /config/etc/sysconfig/network-scripts/ifcfg-ens3 /config/etc/sysconfig/network-scripts/ifcfg-lo supervdsm.log ================= <snip> MainThread::DEBUG::2015-03-10 00:18:32,152::network::55::root.ovirt.node.utils.network::(<module>) NetworkManager support disabled: NM Client not found (could not import gobject (error was: ImportError('When using gi.repository you must not import static modules like "gobject". Please change all occurrences of "import gobject" to "from gi.repository import GObject".',))) MainThread::DEBUG::2015-03-10 00:18:32,632::ifcfg::281::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/ifcfg-rhevm MainThread::DEBUG::2015-03-10 00:18:32,632::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2015-03-10 00:18:32,632::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2015-03-10 00:18:32,632::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2015-03-10 00:18:32,632::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2015-03-10 00:18:32,634::vdsm-restore-net-config::72::root::(unified_restoration) Calling setupNetworks with networks ({}) and bond ({}). </snip>
Created attachment 999708 [details] logs
Douglas, I don't really understand what BOOTIF is expected to do. Is there any difference the generated ifcfg-rhevm of the two cases? Anything else? What has created ifcfg-rhevm? Somehow, not trace of it is to be found in the attached supervdsm.log. Was it ever ifup`ed by the node?
Hi Dan, (In reply to Dan Kenigsberg from comment #3) > Douglas, I don't really understand what BOOTIF is expected to do. Is there > any difference the generated ifcfg-rhevm of the two cases? Anything else? Should not have any difference, the problem is that there is no ifcfg-rhevm in /etc/sysconfig/network-scripts, just the interface up without the ip address. > > What has created ifcfg-rhevm? Somehow, not trace of it is to be found in the > attached supervdsm.log. Was it ever ifup`ed by the node? Unfortunately, no ifcfg-rhevm created. I see it removed in supervdsm.log but it was not created again. MainThread::DEBUG::2015-03-10 00:18:32,632::ifcfg::281::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/ifcfg-rhevm # ls /etc/sysconfig/network-scripts/ifcfg-* /etc/sysconfig/network-scripts/ifcfg-ens3 /etc/sysconfig/network-scripts/ifcfg-lo # ls /config/etc/sysconfig/network-scripts/ifcfg-* /config/etc/sysconfig/network-scripts/ifcfg-ens3 /config/etc/sysconfig/network-scripts/ifcfg-lo Just to add more data to this bugzilla, I have tested upstream oVirt Node (EL6) with the below vdsm and it worked the BOOIF=Mac out of box, the node appeared to be approved. From /var/log/messages ======================== Mar 12 01:11:19 localhost kernel: Command line: initrd=initrd0.img root=live:CDLABEL=node7bytesvalidation.iso.edited rootfstype=auto ro liveimg check RD_NO_LVM crashkernel=128M rd_NO_MULTIPATH rootflags=ro elevator=deadline install quiet max_loop=256 rhgb rd_NO_LUKS rd_NO_MD rd_NO_DM firstboot storage_init=/dev/sda adminpw=RHhwCLrQXB8zE management_server=192.168.122.166 BOOTIF=01-52-54-00-c3-38-2d BOOT_IMAGE=vmlinuz0 # rpm -qa | grep -i vdsm vdsm-yajsonrpc-4.16.10-8.gitc937927.el6.noarch vdsm-xmlrpc-4.16.10-8.gitc937927.el6.noarch vdsm-4.16.10-8.gitc937927.el6.x86_64 vdsm-gluster-4.16.10-8.gitc937927.el6.noarch vdsm-python-zombiereaper-4.16.10-8.gitc937927.el6.noarch vdsm-python-4.16.10-8.gitc937927.el6.noarch vdsm-cli-4.16.10-8.gitc937927.el6.noarch vdsm-hook-ethtool-options-4.16.10-8.gitc937927.el6.noarch ovirt-node-plugin-vdsm-0.4.3-0.0.master.el6.noarch vdsm-jsonrpc-4.16.10-8.gitc937927.el6.noarch # cat /etc/redhat-release oVirt Node Hypervisor 3.5(0.999.201503111303.el6) (Edited)
Currently I do not see a connection between the use of the mac addr and an nic for BOOTIF= and vdsm failing to come up with networking. Ying, can you reproduce the problem described in the description?
Hi, I cannot reproduce this issue with rhev-hypervisor7-7.1-20150312.0 with vdsm-4.16.12-2. # cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor 7.1 (20150312.0.el7ev) # rpm -qa | grep -i vdsm vdsm-python-4.16.12-2.el7ev.noarch ovirt-node-plugin-vdsm-0.2.0-20.el7ev.noarch vdsm-hook-vhostmd-4.16.12-2.el7ev.noarch vdsm-python-zombiereaper-4.16.12-2.el7ev.noarch vdsm-xmlrpc-4.16.12-2.el7ev.noarch vdsm-jsonrpc-4.16.12-2.el7ev.noarch vdsm-4.16.12-2.el7ev.x86_64 vdsm-hook-ethtool-options-4.16.12-2.el7ev.noarch vdsm-cli-4.16.12-2.el7ev.noarch vdsm-yajsonrpc-4.16.12-2.el7ev.noarch vdsm-reg-4.16.12-2.el7ev.noarch /var/log/messages ====================== Mar 19 01:16:37 localhost kernel: Command line: initrd=initrd0.img root=live:CDLABEL=rhev-hypervisor7-7.1-20150312.0 rootfstype=auto ro rd.live.image rd.live.check crashkernel=256M rd_NO_MULTIPATH rootflags=ro elevator=deadline install quiet max_loop=256 rhgb rd.luks=0 rd.md=0 rd.dm=0 firstboot storage_init=/dev/sda adminpw=RHhwCLrQXB8zE management_server=192.168.122.166 BOOTIF=01-52-54-00-8b-34-a3 BOOT_IMAGE=vmlinuz0 Ying, could you please re-try using this last image available? From my side, I would close this one as next release or current release.
This bug is split from this bug https://bugzilla.redhat.com/show_bug.cgi?id=1199035#c12 haiyang, could you please to answer comment 5 and comment 6, thanks.
(In reply to Douglas Schilling Landgraf from comment #6) > Hi, > > I cannot reproduce this issue with rhev-hypervisor7-7.1-20150312.0 with > vdsm-4.16.12-2. > > # cat /etc/redhat-release > Red Hat Enterprise Virtualization Hypervisor 7.1 (20150312.0.el7ev) > > # rpm -qa | grep -i vdsm > vdsm-python-4.16.12-2.el7ev.noarch > ovirt-node-plugin-vdsm-0.2.0-20.el7ev.noarch > vdsm-hook-vhostmd-4.16.12-2.el7ev.noarch > vdsm-python-zombiereaper-4.16.12-2.el7ev.noarch > vdsm-xmlrpc-4.16.12-2.el7ev.noarch > vdsm-jsonrpc-4.16.12-2.el7ev.noarch > vdsm-4.16.12-2.el7ev.x86_64 > vdsm-hook-ethtool-options-4.16.12-2.el7ev.noarch > vdsm-cli-4.16.12-2.el7ev.noarch > vdsm-yajsonrpc-4.16.12-2.el7ev.noarch > vdsm-reg-4.16.12-2.el7ev.noarch > > /var/log/messages > ====================== > Mar 19 01:16:37 localhost kernel: Command line: initrd=initrd0.img > root=live:CDLABEL=rhev-hypervisor7-7.1-20150312.0 rootfstype=auto ro > rd.live.image rd.live.check crashkernel=256M rd_NO_MULTIPATH rootflags=ro > elevator=deadline install quiet max_loop=256 rhgb rd.luks=0 rd.md=0 rd.dm=0 > firstboot storage_init=/dev/sda adminpw=RHhwCLrQXB8zE > management_server=192.168.122.166 BOOTIF=01-52-54-00-8b-34-a3 > BOOT_IMAGE=vmlinuz0 > > Ying, could you please re-try using this last image available? > From my side, I would close this one as next release or current release. Test version: rhev-hypervisor7-7.1-20150312.0.iso vdsm-4.16.12-2.el7ev.x86_64 ovirt-node-3.2.1-10.el7.noarch Red Hat Enterprise Virtualization Manager Version: 3.5.1-0.1.el6ev Test steps: 1. Auto install RHEV-H with "BOOTIF=$MACAddress storage_init=/dev/sda management_server=$RHEV-M_IP firstboot" parameters. /var/log/messages ====================== Mar 19 04:00:25 dhcp-10-30 kernel: Kernel command line: initrd=initrd0.img root=live:CDLABEL=rhev-hypervisor7-7.1-20150312.0 rootfstype=auto ro rd.live.image rd.live.check crashkernel=256M rd_NO_MULTIPATH rootflags=ro elevator=deadline install quiet max_loop=256 rhgb rd.luks=0 rd.md=0 rd.dm=0 BOOTIF=01-78-2b-cb-9a-72-a3 storage_init=/dev/sda management_server=10.66.72.35 firstboot BOOT_IMAGE=vmlinuz0 2. Check rhevh on rhevm3.5 side 3. Check in rhevh side Test results: 1. After step2, rhevh7.1 is not registered on rhevm3.5 side. 2. After step3, the rhevh will display managed by : RHEV-M http://10.66.72.35. and network was unconfigured. No “ifcfg-em1” and "ifcfg-rhevm" files in "/etc/sysconfig/network-scripts". Additional info: No this issue by using "BOOTIF=em1 storage_init=/dev/sda management_server=$RHEV-M_IP firstboot" in rhev-hypervisor7-7.1-20150312.0.iso build.
Created attachment 1003584 [details] attached var logs for rhev-hypervisor7-7.1-20150312.0.iso
Created attachment 1003587 [details] attached sosreport logs for rhev-hypervisor7-7.1-20150312.0.iso
Haiyang, can you please also provide the same clean logs for the successfull run with BOOTIF=em1?
Some sightings from the logs: ovirt-node.log: … (installation logs) 2015-03-19 04:06:21,124 INFO Transaction '<Transaction elements='[<UpdateResolvConf 'Updating resolv.conf'>, <UpdatePeerDNS 'Update PEERDNS statement in ifcfg-* files'>]' title='Configuring DNS' at 0x376aa90>' succeeded 2015-03-19 04:06:22,070 INFO Won't update /etc/hosts, it's not empty. Node stuff above, vdsm stuff below. 2015-03-19 04:06:36,143 WARNING options BOOTPROTO is deprecated. Use bootproto instead 2015-03-19 04:06:36,174 INFO Adding network rhevm with vlan=None, bonding=None, nics=['em1'], bondingOptions=None, mtu=None, bridged=True, defaultRoute=True,options={'IPV6_AUTOCONF': 'no', 'PEERNTP': 'yes', 'bootproto': 'dhcp', 'IPV6INIT': 'no', 'ONBOOT': 'yes'} 2015-03-19 04:06:36,427 INFO Successfully unpersisted file "/etc/sysconfig/network-scripts/ifcfg-em1" 2015-03-19 04:06:38,555 DEBUG Running upgrade upgrade-unified-persistence 2015-03-19 04:06:38,555 DEBUG /sbin/ip route show to 0.0.0.0/0 table all (cwd None) 2015-03-19 04:06:38,558 DEBUG SUCCESS: <err> = ''; <rc> = 0 2015-03-19 04:06:38,560 DEBUG trying to connect libvirt 2015-03-19 04:06:38,569 DEBUG /sbin/ip route show to 0.0.0.0/0 table main (cwd None) 2015-03-19 04:06:38,572 DEBUG SUCCESS: <err> = ''; <rc> = 0 2015-03-19 04:06:38,572 DEBUG upgrade-unified-persistence upgrade persisting networks {} and bondings {} 2015-03-19 04:06:38,572 DEBUG Non-existing config set. 2015-03-19 04:06:38,572 DEBUG Non-existing config set. 2015-03-19 04:06:38,572 INFO Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ 2015-03-19 04:06:38,573 DEBUG No existent config to clear. 2015-03-19 04:06:38,573 INFO Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ 2015-03-19 04:06:38,573 DEBUG No existent config to clear. 2015-03-19 04:06:38,573 INFO Saved new config RunningConfig({}, {}) to /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ 2015-03-19 04:06:38,573 DEBUG /usr/share/vdsm/vdsm-store-net-config unified (cwd None) 2015-03-19 04:06:39,586 DEBUG SUCCESS: <err> = 'cp: cannot stat \xe2\x80\x98/var/run/vdsm/netconf\xe2\x80\x99: No such file or directory\n'; <rc> = 0 The line above is weird. It evaluates to: $ echo -e "\xe2\x80\x98/var/run/vdsm/netconf\xe2\x80\x99" ‘/var/run/vdsm/netconf’ 2015-03-19 04:06:39,586 DEBUG Non-existing config set. 2015-03-19 04:06:39,586 DEBUG Non-existing config set. 2015-03-19 04:06:39,587 INFO File "/var/lib/vdsm/upgrade/upgrade-unified-persistence" successfully persisted 2015-03-19 04:06:39,590 DEBUG Upgrade upgrade-unified-persistence successfully performed And continuing with ovirt.log: 2015-03-19 04:06:39,513 - DEBUG - ovirtfunctions - mkdir -p /config//var/lib/vdsm 2015-03-19 04:06:39,514 - DEBUG - ovirtfunctions - 2015-03-19 04:06:39,523 - DEBUG - ovirtfunctions - cp -a /var/lib/vdsm/persistence /config/var/lib/vdsm/persistence 2015-03-19 04:06:39,523 - DEBUG - ovirtfunctions - 2015-03-19 04:06:39,533 - DEBUG - ovirtfunctions - umount /var/lib/vdsm/persistence 2015-03-19 04:06:39,534 - DEBUG - ovirtfunctions - 2015-03-19 04:06:39,534 - DEBUG - ovirtfunctions - Command 'umount /var/lib/vdsm/persistence' failed with return code 32 STDERR: umount: /var/lib/vdsm/persistence: not mounted 2015-03-19 04:06:39,543 - DEBUG - ovirtfunctions - mount -n --bind /config/var/lib/vdsm/persistence /var/lib/vdsm/persistence 2015-03-19 04:06:39,543 - DEBUG - ovirtfunctions - 2015-03-19 04:06:39,543 - INFO - ovirtfunctions - File: /var/lib/vdsm/persistence persisted 2015-03-19 04:06:39,561 - INFO - ovirtfunctions - Successfully persisted: /var/lib/vdsm/persistence 2015-03-19 04:06:41,407 - DEBUG - ovirtfunctions - sed --copy -i "\|/etc/sysconfig/network-scripts/ifcfg-rhevm$|d" /config/files 2015-03-19 04:06:41,407 - DEBUG - ovirtfunctions - 2015-03-19 04:06:41,416 - DEBUG - ovirtfunctions - shred -u /config'/etc/sysconfig/network-scripts/ifcfg-rhevm' 2015-03-19 04:06:41,417 - DEBUG - ovirtfunctions - 2015-03-19 04:06:41,417 - DEBUG - ovirtfunctions - Command 'shred -u /config'/etc/sysconfig/network-scripts/ifcfg-rhevm'' failed with return code 1 STDERR: shred: /config/etc/sysconfig/network-scripts/ifcfg-rhevm: failed to open for writing: No such file or directory 2015-03-19 04:06:41,426 - DEBUG - ovirtfunctions - shred -u '/etc/sysconfig/network-scripts/ifcfg-rhevm' 2015-03-19 04:06:41,426 - DEBUG - ovirtfunctions - 2015-03-19 04:06:41,446 - DEBUG - ovirtfunctions - sed --copy -i "\|/etc/sysconfig/network-scripts/ifcfg-em1$|d" /config/files 2015-03-19 04:06:41,446 - DEBUG - ovirtfunctions - 2015-03-19 04:06:41,456 - DEBUG - ovirtfunctions - shred -u /config'/etc/sysconfig/network-scripts/ifcfg-em1' 2015-03-19 04:06:41,456 - DEBUG - ovirtfunctions - 2015-03-19 04:06:41,456 - DEBUG - ovirtfunctions - Command 'shred -u /config'/etc/sysconfig/network-scripts/ifcfg-em1'' failed with return code 1 STDERR: shred: /config/etc/sysconfig/network-scripts/ifcfg-em1: failed to open for writing: No such file or directory 2015-03-19 04:06:41,465 - DEBUG - ovirtfunctions - shred -u '/etc/sysconfig/network-scripts/ifcfg-em1' 2015-03-19 04:06:41,465 - DEBUG - ovirtfunctions - In the lines above vdsm seems to be removing the configuration files which are needed for networking. What I do not understand is, why it works when BOOTIF=em1 Dan, do you identify a known problem above? It looks to be related to the unified persistence.
Created attachment 1003846 [details] attached var logs for run with BOOTIF=em1 works in rhev-hypervisor7-7.1-20150312.0.iso (In reply to Fabian Deutsch from comment #12) > Haiyang, can you please also provide the same clean logs for the successfull > run with BOOTIF=em1? Attached var logs for run with BOOTIF=em1 works
Created attachment 1003847 [details] attached sosreport logs for running with BOOTIF=em1 in rhev-hypervisor7-7.1-20150312.0.iso (In reply to Fabian Deutsch from comment #12) > Haiyang, can you please also provide the same clean logs for the successfull > run with BOOTIF=em1? Attached sosreport logs for running with BOOTIF=em1 works
Can you please recreate this on RHEV-H build for 3.5.1?
Assigning to me to keep it on my radar.
(In reply to Yaniv Dary from comment #16) > Can you please recreate this on RHEV-H build for 3.5.1? Hey ydary@ Just want to double check that you hope i make new bug on node side as testonly? thanks hadong
(In reply to haiyang,dong from comment #18) > (In reply to Yaniv Dary from comment #16) > > Can you please recreate this on RHEV-H build for 3.5.1? > > Hey ydary@ > > Just want to double check that you hope i make new bug on node side as > testonly? > > thanks > hadong We had so many network issues resolved in 3.5.1, I want to make sure this is still a real issue.
Indeed, Hiyang, can you please check if this bug still exists on 3.5.1?
Hi, I cannot reproduce this error anymore. As soon as vdsm-reg-setup trigger the registration the rhevm interface is created with ip address. Tested with: # cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor 7.1 (20150512.0.el7ev)