Bug 1612961
| Summary: | Guest interface in status DOWN after hot-unplug-plug | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | msheena | ||||
| Component: | libvirt | Assignee: | Laine Stump <laine> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | yalzhang <yalzhang> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.5 | CC: | dcbw, laine, lmen, mburman, msheena, tburke, xuzhang | ||||
| Target Milestone: | rc | Keywords: | Regression | ||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-09-16 02:15:14 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1611539 | ||||||
| Attachments: |
|
||||||
I suspect this bug blocks https://bugzilla.redhat.com/show_bug.cgi?id=1611539, since on rhel 7.5 this bug causes the vm to report the link DOWN after a hot-unplug and then a hot-plug, so it's very likely that this bug blocks bug 1611539 because if bug 1612961 exists then bug 1611539 will can't be verified since the link simply won't go UP. Hi msheena, I can not reproduce the bug on pure libvirt-3.9.0-14.el7_5.6.x86_64, either on below packages: ovirt-engine-4.2.6.3-0.1.el7ev.noarch libvirt-4.4.0-2.el7.x86_64 qemu-kvm-rhev-2.12.0-6.el7.x86_64 vdsm-4.20.33-1.el7ev.x86_64 I'm trying to prepare the env. as what you described in comment 0 and try again. BTW, Could you please help to confirm when you do step 7 on the GUI page, after the edit, it saved as "up" and "plugged", right? 7. Edit the vnic while the vm is running to 'plugged' (In reply to yalzhang from comment #3) > Hi msheena, I can not reproduce the bug on pure > libvirt-3.9.0-14.el7_5.6.x86_64, either on below packages: > ovirt-engine-4.2.6.3-0.1.el7ev.noarch > libvirt-4.4.0-2.el7.x86_64 > qemu-kvm-rhev-2.12.0-6.el7.x86_64 > vdsm-4.20.33-1.el7ev.x86_64 > > I'm trying to prepare the env. as what you described in comment 0 and try > again. > BTW, Could you please help to confirm when you do step 7 on the GUI page, > after the edit, it saved as "up" and "plugged", right? > > 7. Edit the vnic while the vm is running to 'plugged' Hi yalzhang, I have an env in which I can reproduce, with the following versions: ovirt-engine-4.2.6.3-0.1.el7ev.noarch vdsm-4.20.37-1.el7ev.x86_64 libvirt-3.9.0-14.el7_5.7.x86_64 * Guest os is 'Red Hat Enterprise Linux Server release 7.5 (Maipo)' Please contact me for the env This Bug seems to be a Regression from rhel 7.5, since this flow is verified to work on 7.4 Hi Dan, I saw you are expert at NM. Could you please help to look into this? I have checked on the env, the interface is set to not controlled by NM: # cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE='eth0' ONBOOT="yes" IPV6INIT="yes" BOOTPROTO="dhcp" NM_CONTROLLED='no' <==== after hot-unplug -> hot-plug eth0, the link can not be detected on vm: # ethtool eth0 Settings for eth0: Link detected: no It seems like it is NM's responsibility to bring up the hot-plugged device. But should the iface be brought up by udev? And when I set 'Name="eth0"', there will be a warning says: "Aug 22 19:42:50 localhost NetworkManager[596]: <warn> [1534938170.8135] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-eth0" fails: NM_CONTROLLED was false but device was not uniquely identified; device will be managed" And it will still be brought up by NM. Thank you very much! I am also unable to reproduce this on libvirt 3.9.0-14.el7_5.7 and doing the hotplug/unplug with virsh attach-device / virsh detach-device. My suspicion is that this is either a problem at a level higher than libvirt, or that it is an issue with something in the guest OS. (In reply to msheena from comment #4) > Hi yalzhang, > I have an env in which I can reproduce, with the following versions: > ovirt-engine-4.2.6.3-0.1.el7ev.noarch > vdsm-4.20.37-1.el7ev.x86_64 > libvirt-3.9.0-14.el7_5.7.x86_64 > * Guest os is 'Red Hat Enterprise Linux Server release 7.5 (Maipo)' 1) Does the guest show the device as offline after doing hotplug with the "virsh attach-device" command directly, rather than via the RHV console? This is done by creating a temporary file (e.g. /tmp/if.xml) containing the <interface> section of the guest's libvirt XML, e.g.: <interface type="bridge"> <model type="virtio"/> <link state="up"/ <source bridge="on5041fcaec9944"/> <alias name="ua-d2c20b8f-4efa-4434-bd79-8b27ec2086d8"/> <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/> <mac address="00:1a:4a:16:20:b5"/> <mtu size="1500"/> <filterref filter="vdsm-no-mac-spoofing"/> </interface> then running the virsh commands: virsh detach-device $guestname /tmp/if.xml (wait a few seconds) virsh attach-device $guestname /tmp/if.xml (in the example shown in your logs, I think $guestname is "golden_env_mixed_virtio_2") 2) Does that behavior change if you leave *everything* in the host *and* guest systems the same, but install the older libvirt (from 7.4) or newer libvirt (from 7.6)? 3) does the behavior change with a different guest OS? Hi Laine, I have checked on their env, the guest with such issue both has this configuration file(as described in comment 7): # cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE='eth0' ... NM_CONTROLLED='no' I can reproduce with this config file, too. I think it is because of the configuration about NetworkManager and it is not a bug. After hotunplug and hotplug: [root@host ~]# virsh domif-getlink rhel vnet0 vnet0 up [root@guest ~]# ethtool eth0 Settings for eth0: Link detected: no IMO, as eth0 is configured not managed by NM, when guest boot, network service will set the interface up. But after hotunplug and hotplug, as the device will still be recognized as "eth0", NM will not set it up, nor the network service as it is not event driven. So the interface is down unless you set it up manually. Reference: https://access.redhat.com/solutions/429653 Before I observed the configuration file, I have tried on their original env with below scenarios: 1. start guest with 2 virtio interfaces, then hotunplug -> hotplug the 'eth1', eth1 will be up; 2. start guest with 1 rtl8139 interface, then hotunplug -> hotplug the 'ens*',the 'ens*' will be up; > Reference: https://access.redhat.com/solutions/429653 Based on Comment 9 and the explanation in the above Knowledge Base article, I'm closing this as NOTABUG. (In reply to Laine Stump from comment #10) > > Reference: https://access.redhat.com/solutions/429653 > > Based on Comment 9 and the explanation in the above Knowledge Base article, > I'm closing this as NOTABUG. Hi Laine , I'm sorry but i don't understand why this is not a bug if it used to work on guests with rhel 7.4? This is clearly a regression and this affect our tests in RHV. The interface state after hot-unplug-plug should be UP, not DOWN. In RHV we never let NM to control our network and this is expected that we set NM_CONTROLLED=no. Why we need manually to set the NIC as UP after hot-plug? if on previous versions we didn't set it manually , then why now we need? I just realized part of this conversation was conducted in an email exchange, so the question in Comment 11 appears unanswered from the POV of this BZ. The answer is that according to https://access.redhat.com/solutions/429653 the problem is caused by the *guest* OS, not the host OS, and is due to a change that was made (purposefully) prior to the release of RHEL7.0 - if the guest OS is RHEL7 or later and has NM disabled, hotplug of a network device will not automatically bring the device online. My personal opinion is that this behavior shouldn't have been removed, but there are likely details I don't know about. At any rate, there is nothing here that can be remedied in any way by libvirt, qemu, or the host kernel. If the behavior really did change when updating the *guest* OS from some earlier RHEL7 release, then you may be able to reopen this BZ (or open a new one) against whichever packet modifies the udev rules file referenced in the knowledge base article. |
Created attachment 1473656 [details] vdsm and libvirt logs Description of problem: When attaching a network to a vm vnic, and then starting the vm, it will power on with the corresponding interface in state UP. When performing a hot-unplug the guest will rightly not show the interface, however when hot-plugging the vnic back, the guest reports the interface state is DOWN, although libvirt reports the interface is UP. Version-Release number of selected component (if applicable): libvirt-3.9.0-14.el7_5.6.x86_64 ovirt-engine-4.2.6-0.1.el7ev.noarch vdsm-4.20.36-1.el7ev.x86_64 qemu-kvm-rhev-2.10.0-21.el7_5.5.x86_64 qemu-kvm-common-rhev-2.10.0-21.el7_5.5.x86_64 Red Hat Enterprise Linux Server release 7.5 (Maipo) How reproducible: 100% Steps to Reproduce: 1. Create a vm network called 'red' 2. Attach the network to a host 3. Attach the corresponding vnic profile to a vm that is powered down 4. Power on the vm. verify the interface link state is UP using 'ip link' command / 'ip a' command 5. Edit the vnic while the vm is running to 'unplugged' 6. Verify the vm does not report the interface 7. Edit the vnic while the vm is running to 'plugged' Actual results: The vm should report the interface state is UP Expected results: The vm reports the interface state is DOWN Additional info: Libvirt xml dump shows that the interface state is UP, although the guest reports it is DOWN.