Bug 1762906
| Summary: | Intermittent error 'Hotplug nic can not get IP address' when executing test case nic_hotplug.vhost_nic.nic_virtio | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Karen Mezick <kmezick> |
| Component: | qemu-kvm | Assignee: | Juan Quintela <quintela> |
| qemu-kvm sub component: | Networking | QA Contact: | Lei Yang <leiyang> |
| Status: | CLOSED WONTFIX | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | aadam, ailan, chayang, ehabkost, juzhang, ldoktor, leiyang, ngu, pezhang, qzhang, rbalakri, virt-maint, wainersm, wyu, yihyu |
| Version: | 8.1 | Keywords: | Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-01-26 09:57:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Attachments: | |||
|
Description
Karen Mezick
2019-10-17 18:56:26 UTC
Created attachment 1626926 [details]
Log file showing nic_hotplug test case passing (attached for comparison purposes)
(In reply to Karen Mezick from comment #0) > enp0s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > ether 9a:b5:b6:b7:b8:b9 txqueuelen 1000 (Ethernet) > RX packets 393 bytes 20728 (20.2 KiB) > RX errors 0 dropped 54 overruns 0 frame 0 > TX packets 1 bytes 325 (325.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > Hello Karen, Sometimes we may encounter mac address conflicts due to the way the MAC address generation is implemented (avocado-vt). For more details, you can see this issue: https://github.com/avocado-framework/avocado-vt/issues/2121 Fortunately, it has been fixed at avocado-vt 71. Can you update your test environment and retest it? If it still fails, please let me know. Created attachment 1627242 [details]
Log file showing nic_virtio test case error with Avocado 71.0
Hi Yihuang Thank you for your quick response. I tried running this with Avocado 71.0 and got the same error. Please see attached log. # avocado -v Avocado 71.0 ppc64le Red Hat Enterprise Linux release 8.1 Beta (Ootpa) qemu-kvm-2.12.0-88.module+el8.1.0+4233+bc44be3f.ppc64le kernel-4.18.0-141.el8.ppc64le Thanks, Karen One clarification: In this latest test with Avocado 71.0, the error is slightly different: avocado.core.exceptions.TestFail: Hotplug nic can't get ip after reboot vm I have also seen this error previously when running with Avocado 69.0. This is some result of the log analysis:
1. Hot plug a nic device with "9a:2e:0b:20:95:0c"
2019-10-18 13:13:12,692 qemu_monitor L1920 DEBUG| Send command: {'execute': 'device_add', 'arguments': {'driver': 'virtio-net-pci', 'netdev': 'id1hXOy1', 'mac': '9a:2e:0b:20:95:0c', 'id': 'hotplug_nic1', 'vectors': 4}, 'id': 'kxorZOVp'}
2. Guest get ip address "192.168.122.76"
2019-10-18 13:13:20,747 nic_hotplug L0189 INFO | Got the ip address of new nic: 192.168.122.76
3. Avocado-vt will drop the address cache during reboot
2019-10-18 13:15:45,686 ip_sniffing L0068 DEBUG| Dropped the address cache of HWADDR (9a:2e:0b:20:95:0c)
4. When guest can not get ip address, we can see from arp table, it has cleaned up the address cache but does not refresh
2019-10-18 13:22:58,924 process L0437 DEBUG| [stdout] 10.0.1.254 ether e8:b6:c2:10:dc:81 C enP68p1s0f2
2019-10-18 13:22:58,924 process L0437 DEBUG| [stdout] 192.168.122.186 ether 9a:16:f1:d7:a6:ea C virbr0
2019-10-18 13:22:58,924 process L0437 DEBUG| [stdout] 192.168.122.76 (incomplete) virbr0 <--------------------------
Actually, the latest code remove this step, I don't know why, will cc the author(@wyu) here.
Please not that this not only happens on ppc, I can also reproduce it on x86 after add this step in the code.
(1/5) repeat1.Host_RHEL.m8.u1.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.1.0.x86_64.io-github-autotest-qemu.nic_hotplug.vhost_nic.nic_virtio.q35: PASS (200.10 s)
(2/5) repeat2.Host_RHEL.m8.u1.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.1.0.x86_64.io-github-autotest-qemu.nic_hotplug.vhost_nic.nic_virtio.q35: FAIL: Hotplug nic can't get ip after reboot vm (718.15 s)
(3/5) repeat3.Host_RHEL.m8.u1.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.1.0.x86_64.io-github-autotest-qemu.nic_hotplug.vhost_nic.nic_virtio.q35: FAIL: Hotplug nic can not get ip address (610.05 s)
(4/5) repeat4.Host_RHEL.m8.u1.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.1.0.x86_64.io-github-autotest-qemu.nic_hotplug.vhost_nic.nic_virtio.q35: PASS (199.41 s)
(5/5) repeat5.Host_RHEL.m8.u1.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.1.0.x86_64.io-github-autotest-qemu.nic_hotplug.vhost_nic.nic_virtio.q35: FAIL: Hotplug nic can not get ip address (580.89 s)
It's hard to say whether it's an issue with automation or kvm(network?). Need some time for further analysis.
RHEL7 cannot reproduce this issue, so it may be a regression.
Pei and Lei, I am CC'ing you because this is not a ppc-specific problem.
You can modify the automatic script to reproduce this problem in RHEL8 host as follows:
diff --git a/qemu/tests/nic_hotplug.py b/qemu/tests/nic_hotplug.py
index 00b3609..5156f79 100644
--- a/qemu/tests/nic_hotplug.py
+++ b/qemu/tests/nic_hotplug.py
@@ -221,8 +221,11 @@ def run(test, params, env):
err_msg = "New nic failed ping test, error info: '%s'"
test.fail(err_msg % output)
+ logging.info("Reboot vm after hotplug nic")
# reboot vm via serial port since some guest can't auto up
# hotplug nic and next step will check is hotplug nic works.
+ s_session = vm.reboot(session=s_session, serial=True)
+ vm.verify_alive()
hotnic_ip = get_hotplug_nic_ip(
vm,
After that, repeat this case some times, and you will see this case reports "FAIL: Hotplug nic can not get ip address" or "FAIL: Hotplug nic can't get ip after reboot vm"
I can see that ip-sniffer.log has different points, hopeful you can help to debug it.
Thanks,
Yihuang
Lei, could you help trying to reproduce the bug on x86 according to comment 8? (In reply to Gu Nini from comment #9) > Lei, could you help trying to reproduce the bug on x86 according to comment > 8? Hi,Nini I have reproduced this bug on x86_64 server with automation method.But i can not reproduce this bug from manually. Test result: (21/24) Host_RHEL.m8.u2.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.0.x86_64.io-github-autotest-qemu.nic_hotplug.vhost_nic.nic_virtio.q35: FAIL: Hotplug nic can not get ip address Best regards, LeiYang (In reply to Lei Yang from comment #10) > (In reply to Gu Nini from comment #9) > > Lei, could you help trying to reproduce the bug on x86 according to comment > > 8? > > Hi,Nini > > I have reproduced this bug on x86_64 server with automation method.But i can > not reproduce this bug from manually. > > Test result: > (21/24) > Host_RHEL.m8.u2.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.0.x86_64.io- > github-autotest-qemu.nic_hotplug.vhost_nic.nic_virtio.q35: FAIL: Hotplug nic > can not get ip address > > Best regards, > LeiYang Thanks Lei. Move the bug to be shared arch. QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks Hit same issue: Test Version: qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a.x86_64 kernel-4.18.0-262.el8.dt3.x86_64 |