Description of problem: When boot windows guest with hugepage, q35, vhost=on and cpu flags "hv_stimer,hv_time,hv_synic,hv_vpindex", guest can't get IP address, and qemu print many lines of message "qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000". Version-Release number of selected component (if applicable): qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb kernel-4.18.0-158.el8.x86_64 virtio-win-prewhql-0.1-172 How reproducible: always Steps to Reproduce: 1. Set up hugepage on host # echo 4096 > /proc/sys/vm/nr_hugepages # mount -t hugetlbfs none /mnt/kvm_hugepage 2. Boot windows guest with qemu cli[1] Actual results: Guest can't get IP address, and QEMU print many lines of same message. (qemu) qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000 qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000 qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000 qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000 qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000 qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000 ... Expected results: Guest network work well. Additional info: [1] Full qemu cli: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine q35 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x1 \ -m 4096 \ -object memory-backend-file,size=1024M,prealloc=no,mem-path=/mnt/kvm_hugepage,policy=default,id=mem-mem0 \ -object memory-backend-file,size=3072M,prealloc=no,mem-path=/mnt/kvm_hugepage,policy=default,id=mem-mem1 \ -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 \ -numa node,memdev=mem-mem0 \ -numa node,memdev=mem-mem1 \ -cpu SandyBridge,hv_stimer,hv_time,hv_synic,hv_vpindex \ -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \ -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \ -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \ -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/yuhuang/win2019-64-virtio-scsi.qcow2 \ -device scsi-hd,id=image1,drive=drive_image1 \ -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \ -device virtio-net-pci,mac=9a:5c:d7:9f:cd:48,id=id7ex9m8,netdev=idim5Sro,bus=pcie.0-root-port-4,addr=0x0 \ -netdev tap,id=idim5Sro,vhost=on \ -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso \ -device scsi-cd,id=cd1,drive=drive_cd1 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :1 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \ -monitor stdio -qmp tcp:0:4444,server,nowait [2] Reproduced with win2019, win2016 and win10.x86_64 guest. [3] Reproduced on rhel8.1.1-av, qemu-kvm-4.1.0-15.module+el8.1.1+4700+209eec8f, kernel-4.18.0-147.3.1.el8_1.x86_64 .
Changed name of the BZ to refelect exact problem. I'd suggest to use more specific names in future. The same problem was described in: https://bugs.launchpad.net/qemu/+bug/1811533 https://github.com/virtio-win/kvm-guest-drivers-windows/issues/402 1. Please check whether the problem happens with: addition of the enlightenment "+x-hv-synic-kvm-only" to the cpu command line
(In reply to ybendito from comment #2) > Changed name of the BZ to refelect exact problem. > I'd suggest to use more specific names in future. > The same problem was described in: > https://bugs.launchpad.net/qemu/+bug/1811533 > https://github.com/virtio-win/kvm-guest-drivers-windows/issues/402 > > 1. Please check whether the problem happens with: > addition of the enlightenment "+x-hv-synic-kvm-only" to the cpu command line The issue is gone with "+x-hv-synic-kvm-only". Thanks.
Vitaly, do you have some comments about possible root cause?
Unfortunately I didn't get a chance to do any real debugging here but at first glance the issue seems to be related to memory regions internals in QEMU. For SynIC, Windows guest picks two 4k pages (per vCPU) somewhere in its memory and writes their PFNs to synthetic MSRs (HV_X64_MSR_SIMP/HV_X64_MSR_SIEFP) and without "+x-hv-synic-kvm-only" these addresses are passed down to in-QEMU synic implementation, which does memory_region_init_ram()/memory_region_add_subregion() for these two pages (per vCPU). Without VMBus devices (not yet upstream) this is not being used but this seems to be enough to break something inside QEMU's (vhost?) logic. I'm not at all familiar with this code, hope David has some ideas.
So, IUUC the problem started from 3.1 configuration which prefers synic in qemu wherever it seems possible https://git.qemu.org/?p=qemu.git;a=commit;h=9b4cf107b09d18ac30f46fd1c4de8585ccba030c Can you please add 'need info' for relevant people?
Dave, I recall we exchanged a few words about this, do you have any suggestions on who would be the right person to take a look from QEMU side? (not sure if it's vhost or memory regions code actually)
info mtree shows: 0000000000080000-0000000000080fff (prio 0, ram): synic-0-msg-page 0000000000081000-0000000000081fff (prio 0, ram): synic-1-msg-page 0000000000082000-0000000000082fff (prio 0, ram): synic-2-msg-page 0000000000083000-0000000000083fff (prio 0, ram): synic-3-msg-page 0000000000084000-0000000000084fff (prio 0, ram): synic-4-msg-page 0000000000085000-0000000000085fff (prio 0, ram): synic-5-msg-page 0000000000086000-0000000000086fff (prio 0, ram): synic-6-msg-page 0000000000087000-0000000000087fff (prio 0, ram): synic-7-msg-page 0000000000088000-0000000000088fff (prio 0, ram): synic-8-msg-page 0000000000089000-0000000000089fff (prio 0, ram): synic-9-msg-page 000000000008a000-000000000008afff (prio 0, ram): synic-10-msg-page 000000000008b000-000000000008bfff (prio 0, ram): synic-11-msg-page 000000000008c000-000000000008cfff (prio 0, ram): synic-12-msg-page 000000000008d000-000000000008dfff (prio 0, ram): synic-13-msg-page 000000000008e000-000000000008efff (prio 0, ram): synic-14-msg-page 000000000008f000-000000000008ffff (prio 0, ram): synic-15-msg-page 80000=512k and each one is 4k; so those are all sitting over the lower RAM mapping which is why it's getting confused. Carrying on looking
Hi Yumei I've just installed a test package with a potential fix in; https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=25665929 it's installed on per730-39 and seems to work for me. Please let me know if it's OK. Dave
Posted upstream: https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg01429.html
(In reply to Dr. David Alan Gilbert from comment #15) > Hi Yumei > I've just installed a test package with a potential fix in; > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=25665929 > > it's installed on per730-39 and seems to work for me. > > Please let me know if it's OK. Yes, it works fine. Thanks. > > Dave
New version posted upstream vhost: Add names to section rounded warning memory: Allow a MemoryRegion to be marked no_vhost hyperv/synic: Mark regions as no vhost
Merged upstream: 76525114736e8f669766 - vhost: Only align sections for vhost-user
Tested with qemu-kvm-4.2.0-8.module+el8.2.0+5607+dc756904 / kernel-4.18.0-175.el8.x86_64, the issue is gone, win2019 guest network works well.
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks
==Steps 1. Set up hugepage on host # echo 4096 > /proc/sys/vm/nr_hugepages # mount -t hugetlbfs none /mnt/kvm_hugepage 2. Boot windows guest with qemu cli /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine q35 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x1 \ -m 4096 \ -object memory-backend-file,size=1024M,prealloc=no,mem-path=/mnt/kvm_hugepage,policy=default,id=mem-mem0 \ -object memory-backend-file,size=3072M,prealloc=no,mem-path=/mnt/kvm_hugepage,policy=default,id=mem-mem1 \ -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 \ -numa node,memdev=mem-mem0 \ -numa node,memdev=mem-mem1 \ -cpu SandyBridge,hv_stimer,hv_time,hv_synic,hv_vpindex \ -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \ -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \ -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \ -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2 \ -device scsi-hd,id=image1,drive=drive_image1 \ -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \ -device virtio-net-pci,mac=9a:5c:d7:9f:cd:48,id=id7ex9m8,netdev=idim5Sro,bus=pcie.0-root-port-4,addr=0x0 \ -netdev tap,id=idim5Sro,vhost=on \ -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso \ -device scsi-cd,id=cd1,drive=drive_cd1 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :1 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \ -monitor stdio -qmp tcp:0:4444,server,nowait \ 2. Guest can't get IP address, and QEMU print many lines of same message. # sh win2019.sh QEMU 4.2.0 monitor - type 'help' for more information (qemu) qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 80000 qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 80000 qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 80000 qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 80000 qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 80000 ==Reproduced with qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64 ==Verified with qemu-kvm-4.2.0-8.module+el8.2.0+5607+dc756904.x86_64 1.Boot windows guest,guest get IP and QEMU no message printed. # sh win2019.sh QEMU 4.2.0 monitor - type 'help' for more information (qemu) 2.So this bug has been fixed very well. Move to 'VERIFIED'. Test Version: kernel-4.18.0-179.el8.x86_64 qemu-kvm-4.2.0-8.module+el8.2.0+5607+dc756904.x86_64 virtio-win-prewhql-0.1-179.iso
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017