Bug 1779041 - netkvm: no connectivity Windows guest with q35 + hugepages + vhost + hv_synic
Summary: netkvm: no connectivity Windows guest with q35 + hugepages + vhost + hv_synic
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.2
Assignee: Dr. David Alan Gilbert
QA Contact: Lei Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-03 06:38 UTC by Yumei Huang
Modified: 2020-02-19 05:15 UTC (History)
12 users (show)

Fixed In Version: qemu-kvm-4.2.0-8.module+el8.2.0+5607+dc756904
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)

Description Yumei Huang 2019-12-03 06:38:44 UTC
Description of problem:
When boot windows guest with hugepage, q35, vhost=on and cpu flags "hv_stimer,hv_time,hv_synic,hv_vpindex", guest can't get IP address, and qemu print many lines of message "qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000".


Version-Release number of selected component (if applicable):
qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb
kernel-4.18.0-158.el8.x86_64
virtio-win-prewhql-0.1-172

How reproducible:
always

Steps to Reproduce:
1. Set up hugepage on host
# echo 4096 > /proc/sys/vm/nr_hugepages
# mount -t hugetlbfs none /mnt/kvm_hugepage 

2. Boot windows guest with qemu cli[1]


Actual results:
Guest can't get IP address, and QEMU print many lines of same message.
(qemu) qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000
qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000
qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000
qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000
qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000
qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 8f000
...



Expected results:
Guest network work well. 


Additional info:
[1] Full qemu cli:
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1 \
    -m 4096 \
    -object memory-backend-file,size=1024M,prealloc=no,mem-path=/mnt/kvm_hugepage,policy=default,id=mem-mem0 \
    -object memory-backend-file,size=3072M,prealloc=no,mem-path=/mnt/kvm_hugepage,policy=default,id=mem-mem1  \
    -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
    -numa node,memdev=mem-mem0  \
    -numa node,memdev=mem-mem1  \
    -cpu SandyBridge,hv_stimer,hv_time,hv_synic,hv_vpindex \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/yuhuang/win2019-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:5c:d7:9f:cd:48,id=id7ex9m8,netdev=idim5Sro,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idim5Sro,vhost=on \
    -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso \
    -device scsi-cd,id=cd1,drive=drive_cd1 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :1  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -monitor stdio -qmp tcp:0:4444,server,nowait

[2] Reproduced with win2019, win2016 and win10.x86_64 guest.

[3] Reproduced on rhel8.1.1-av, qemu-kvm-4.1.0-15.module+el8.1.1+4700+209eec8f, kernel-4.18.0-147.3.1.el8_1.x86_64 .

Comment 2 ybendito 2019-12-31 10:45:59 UTC
Changed name of the BZ to refelect exact problem.
I'd suggest to use more specific names in future.
The same problem was described in:
https://bugs.launchpad.net/qemu/+bug/1811533
https://github.com/virtio-win/kvm-guest-drivers-windows/issues/402

1. Please check whether the problem happens with:
addition of the enlightenment "+x-hv-synic-kvm-only" to the cpu command line

Comment 3 Yumei Huang 2020-01-02 03:10:01 UTC
(In reply to ybendito from comment #2)
> Changed name of the BZ to refelect exact problem.
> I'd suggest to use more specific names in future.
> The same problem was described in:
> https://bugs.launchpad.net/qemu/+bug/1811533
> https://github.com/virtio-win/kvm-guest-drivers-windows/issues/402
> 
> 1. Please check whether the problem happens with:
> addition of the enlightenment "+x-hv-synic-kvm-only" to the cpu command line

The issue is gone with "+x-hv-synic-kvm-only". Thanks.

Comment 4 ybendito 2020-01-03 06:47:46 UTC
Vitaly, do you have some comments about possible root cause?

Comment 5 Vitaly Kuznetsov 2020-01-03 15:08:37 UTC
Unfortunately I didn't get a chance to do any real debugging here but at first glance the issue seems to be related to memory regions internals in QEMU.

For SynIC, Windows guest picks two 4k pages (per vCPU) somewhere in its memory and writes their PFNs to synthetic MSRs (HV_X64_MSR_SIMP/HV_X64_MSR_SIEFP)
and without "+x-hv-synic-kvm-only" these addresses are passed down to in-QEMU synic implementation, which does memory_region_init_ram()/memory_region_add_subregion() for these two pages (per vCPU).
Without VMBus devices (not yet upstream) this is not being used but this seems to be enough to break something inside QEMU's (vhost?) logic. I'm not at all familiar with this code, hope David
has some ideas.

Comment 6 ybendito 2020-01-04 08:52:30 UTC
So, IUUC the problem started from 3.1 configuration which prefers synic in qemu wherever it seems possible
https://git.qemu.org/?p=qemu.git;a=commit;h=9b4cf107b09d18ac30f46fd1c4de8585ccba030c
Can you please add 'need info' for relevant people?

Comment 7 Vitaly Kuznetsov 2020-01-06 12:19:18 UTC
Dave,

I recall we exchanged a few words about this, do you have any suggestions on
who would be the right person to take a look from QEMU side?
(not sure if it's vhost or memory regions code actually)

Comment 13 Dr. David Alan Gilbert 2020-01-07 15:26:08 UTC
info mtree shows:

    0000000000080000-0000000000080fff (prio 0, ram): synic-0-msg-page
    0000000000081000-0000000000081fff (prio 0, ram): synic-1-msg-page
    0000000000082000-0000000000082fff (prio 0, ram): synic-2-msg-page
    0000000000083000-0000000000083fff (prio 0, ram): synic-3-msg-page
    0000000000084000-0000000000084fff (prio 0, ram): synic-4-msg-page
    0000000000085000-0000000000085fff (prio 0, ram): synic-5-msg-page
    0000000000086000-0000000000086fff (prio 0, ram): synic-6-msg-page
    0000000000087000-0000000000087fff (prio 0, ram): synic-7-msg-page
    0000000000088000-0000000000088fff (prio 0, ram): synic-8-msg-page
    0000000000089000-0000000000089fff (prio 0, ram): synic-9-msg-page
    000000000008a000-000000000008afff (prio 0, ram): synic-10-msg-page
    000000000008b000-000000000008bfff (prio 0, ram): synic-11-msg-page
    000000000008c000-000000000008cfff (prio 0, ram): synic-12-msg-page
    000000000008d000-000000000008dfff (prio 0, ram): synic-13-msg-page
    000000000008e000-000000000008efff (prio 0, ram): synic-14-msg-page
    000000000008f000-000000000008ffff (prio 0, ram): synic-15-msg-page
 

80000=512k and each one is 4k; so those are all sitting over the lower RAM mapping which is why it's getting confused.
Carrying on looking

Comment 15 Dr. David Alan Gilbert 2020-01-08 13:35:21 UTC
Hi Yumei
  I've just installed a test package with a potential fix in;
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=25665929

it's installed on per730-39  and seems to work for me.

Please let me know if it's OK.

Dave

Comment 16 Dr. David Alan Gilbert 2020-01-08 14:05:19 UTC
Posted upstream:
  https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg01429.html

Comment 19 Yumei Huang 2020-01-09 03:04:45 UTC
(In reply to Dr. David Alan Gilbert from comment #15)
> Hi Yumei
>   I've just installed a test package with a potential fix in;
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=25665929
> 
> it's installed on per730-39  and seems to work for me.
> 
> Please let me know if it's OK.

Yes, it works fine. Thanks. 

> 
> Dave

Comment 27 Dr. David Alan Gilbert 2020-01-13 17:47:29 UTC
New version posted upstream
  vhost: Add names to section rounded warning
  memory: Allow a MemoryRegion to be marked no_vhost
  hyperv/synic: Mark regions as no vhost

Comment 33 Dr. David Alan Gilbert 2020-01-24 18:18:08 UTC
Merged upstream:
  76525114736e8f669766 - vhost: Only align sections for vhost-user

Comment 37 Yumei Huang 2020-02-05 07:32:03 UTC
Tested with qemu-kvm-4.2.0-8.module+el8.2.0+5607+dc756904 / kernel-4.18.0-175.el8.x86_64, the issue is gone, win2019 guest network works well.

Comment 38 Ademar Reis 2020-02-05 23:09:23 UTC
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks

Comment 39 Lei Yang 2020-02-19 05:15:08 UTC
==Steps
1. Set up hugepage on host
# echo 4096 > /proc/sys/vm/nr_hugepages
# mount -t hugetlbfs none /mnt/kvm_hugepage 

2. Boot windows guest with qemu cli
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-machine q35  \
-nodefaults \
-device VGA,bus=pcie.0,addr=0x1 \
-m 4096 \
-object memory-backend-file,size=1024M,prealloc=no,mem-path=/mnt/kvm_hugepage,policy=default,id=mem-mem0 \
-object memory-backend-file,size=3072M,prealloc=no,mem-path=/mnt/kvm_hugepage,policy=default,id=mem-mem1  \
-smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
-numa node,memdev=mem-mem0  \
-numa node,memdev=mem-mem1  \
-cpu SandyBridge,hv_stimer,hv_time,hv_synic,hv_vpindex \
-device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
-device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
-device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1 \
-device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
-device virtio-net-pci,mac=9a:5c:d7:9f:cd:48,id=id7ex9m8,netdev=idim5Sro,bus=pcie.0-root-port-4,addr=0x0  \
-netdev tap,id=idim5Sro,vhost=on \
-drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso \
-device scsi-cd,id=cd1,drive=drive_cd1 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
-vnc :1  \
-rtc base=localtime,clock=host,driftfix=slew  \
-boot order=cdn,once=c,menu=off,strict=off \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
-monitor stdio -qmp tcp:0:4444,server,nowait \

2. Guest can't get IP address, and QEMU print many lines of same message.
# sh win2019.sh 
QEMU 4.2.0 monitor - type 'help' for more information
(qemu) qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 80000
qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 80000
qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 80000
qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 80000
qemu-kvm: vhost_region_add_section:Section rounded to 0 prior to previous 80000

==Reproduced with qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64

==Verified with qemu-kvm-4.2.0-8.module+el8.2.0+5607+dc756904.x86_64
1.Boot windows guest,guest get IP and QEMU no message printed.
# 
sh win2019.sh 
QEMU 4.2.0 monitor - type 'help' for more information
(qemu) 

2.So this bug has been fixed very well. Move to 'VERIFIED'.

Test Version:
kernel-4.18.0-179.el8.x86_64
qemu-kvm-4.2.0-8.module+el8.2.0+5607+dc756904.x86_64
virtio-win-prewhql-0.1-179.iso


Note You need to log in before you can comment on or make changes to this bug.