Bug 1448149
| Summary: | <memoryBacking> <access mode='shared'/> is ignored if no NUMA nodes are configured | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Bill Townsend <wtownsen> | ||||
| Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Jing Qi <jinqi> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.4 | CC: | dyuan, jdenemar, jtomko, mprivozn, wtownsen, xuzhang, yalzhang | ||||
| Target Milestone: | rc | Keywords: | Upstream | ||||
| Target Release: | --- | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | libvirt-4.3.0-1.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-10-30 09:49:58 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Could you please attach the full XML? I tried both snippets as you've provided them and they work. It's possible that you've placed them incorrectly. OK, I tried the below alone and hugepages were not shared.
<domain type='kvm'>
<name>rhel7.3-rhelsa</name>
<uuid>05a207a4-e6ef-4eac-b9c2-39db94d1939e</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<memoryBacking>
<hugepages>
<page size='524288' unit='KiB' nodeset='0'/>
</hugepages>
<access mode='shared'/>
</memoryBacking>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='aarch64' machine='virt-rhel7.4.0'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/rhel7.3-rhelsa_VARS.fd</nvram>
<boot dev='hd'/>
</os>
<features>
<gic version='3'/>
</features>
<cpu mode='host-passthrough'/>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/rhel7.3-rhelsa.qcow2'/>
<target dev='sda' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='scsi' index='0' model='virtio-scsi'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='1' port='0x8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='2' port='0x9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='3' port='0xa'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='4' port='0xb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<interface type='vhostuser'>
<mac address='00:be:ef:ed:00:01'/>
<source type='unix' path='/usr/local/var/run/openvswitch/guestdpdk1' mode='client'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
</devices>
</domain>
The numa version I tried where it rejected it:
<domain type='kvm'>
<name>rhel7.3-rhelsa-2</name>
<uuid>ee88d7c2-d7c0-4070-9234-f6267481a6c8</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<memoryBacking>
<hugepages>
<page size='524288' unit='KiB' nodeset='0'/>
</hugepages>
</memoryBacking>
<cpu>
<numa>
<cell id='0' cpus='0' memory='2097152' unit='KiB' memAccess='shared'/>
</numa>
</cpu>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='aarch64' machine='virt-rhel7.4.0'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/rhel7.3-rhelsa-2_VARS.fd</nvram>
<boot dev='hd'/>
</os>
<features>
<gic version='3'/>
</features>
<cpu mode='host-passthrough'/>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/rhel7.3-rhelsa-2.qcow2'/>
<target dev='sda' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='scsi' index='0' model='virtio-scsi'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='1' port='0x8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='2' port='0x9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='3' port='0xa'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='4' port='0xb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:63:31:bb'/>
<source network='default'/>
<model type='virtio'/>
<driver name='qemu'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:ba:be'/>
<source bridge='br1'/>
<virtualport type='openvswitch'>
<parameters interfaceid='fcad8498-17c9-4c9d-a148-891dc3af6f38'/>
</virtualport>
<model type='virtio'/>
<driver name='qemu'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='vhostuser'>
<mac address='00:be:ef:ed:00:07'/>
<source type='unix' path='/usr/local/var/run/openvswitch/guestdpdk2' mode='client'/>
<model type='virtio'/>
<driver name='qemu'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
</devices>
</domain>
Oh, here is the command line when using memoryBacking husgepages shared. I dont see something like this: -object memory-backend-file,id=mem,size=4G,mem-path=/dev/hugepages,share=on qemu 17251 89.1 0.2 2499328 187200 ? Sl 13:34 0:20 /usr/libexec/qemu-kvm -name guest=rhel7.3-rhelsa,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-rhel7.3-rhelsa/master-key.aes -machine virt-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -cpu host -drive file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/rhel7.3-rhelsa_VARS.fd,if=pflash,format=raw,unit=1 -m 2048 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/6-rhel7.3-rhelsa -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 05a207a4-e6ef-4eac-b9c2-39db94d1939e -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-rhel7.3-rhelsa/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 -device ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 -device ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 -device ioh3420,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x0 -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 -drive file=/var/lib/libvirt/images/rhel7.3-rhelsa.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -chardev socket,id=charnet0,path=/usr/local/var/run/openvswitch/guestdpdk1 -netdev vhost-user,chardev=charnet0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:be:ef:ed:00:01,bus=pcie.0,addr=0x9 -serial pty -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-6-rhel7.3-rhelsa/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -msg timestamp=on root 17263 0.0 0.0 110272 2496 ttyAMA0 S+ 13:34 0:00 grep --color=auto qemu (In reply to Bill Townsend from comment #4) > The numa version I tried where it rejected it: > > <domain type='kvm'> ... > </memoryBacking> > <cpu> > <numa> > <cell id='0' cpus='0' memory='2097152' unit='KiB' memAccess='shared'/> > </numa> > </cpu> > <vcpu placement='static'>1</vcpu> .. > </features> > <cpu mode='host-passthrough'/> > <clock offset='utc'/> You specified the <cpu> element twice. Looking at the code, specifying
<memoryBacking>
<access mode='shared'/>
is not enough, since the shared memory object is created only for a NUMA node. So going the NUMA way is desired.
Libvirt should not allow starting a VM if the config is not valid though.
(In reply to Ján Tomko from comment #6) > (In reply to Bill Townsend from comment #4) > > The numa version I tried where it rejected it: > > > > <domain type='kvm'> > ... > > </memoryBacking> > > <cpu> > > <numa> > > <cell id='0' cpus='0' memory='2097152' unit='KiB' memAccess='shared'/> > > </numa> > > </cpu> > > <vcpu placement='static'>1</vcpu> > .. > > </features> > > <cpu mode='host-passthrough'/> > > <clock offset='utc'/> > > You specified the <cpu> element twice. ACK, you are right. No one else caught that. I changed to: <cpu mode='host-passthrough'> <numa> <cell id='0' cpus='0' memory='1048576' unit='KiB' memAccess='shared'/> </numa> </cpu> And that was accepted. Also tested it and it does work. Thanks! (In reply to Peter Krempa from comment #7) > Looking at the code, specifying > <memoryBacking> > <access mode='shared'/> > > is not enough, since the shared memory object is created only for a NUMA > node. So going the NUMA way is desired. > > Libvirt should not allow starting a VM if the config is not valid though. OK, so what do you want to do with this part? Thanks. (In reply to Bill Townsend from comment #9) > (In reply to Peter Krempa from comment #7) > > Looking at the code, specifying > > <memoryBacking> > > <access mode='shared'/> > > > > is not enough, since the shared memory object is created only for a NUMA > > node. So going the NUMA way is desired. > > > > Libvirt should not allow starting a VM if the config is not valid though. > > > OK, so what do you want to do with this part? Thanks. Looking at the domain model documentation under memoryBacking it reads: access Specify if memory is shared or private. This can be overridden per numa node by memAccess Thanks for confirming this. I'll repurpose this bug to track that <access mode='shared'/> is ignored if no NUMA nodes are configured. This is already fixed upstream:
commit 671d18594f42f24f7cb3f1b2c574ff4ddba0b183
Author: Michal Privoznik <mprivozn>
Date: Tue Jun 6 10:58:37 2017 +0200
qemu: Allow memAccess for hugepages again
https://bugzilla.redhat.com/show_bug.cgi?id=1214369
https://bugzilla.redhat.com/show_bug.cgi?id=1458638
Historically, we've always supported memAccess for domains backed
by hugepages. However, somewhere along the way we've regressed
and stopped allowing such configuration. Fix it.
Signed-off-by: Michal Privoznik <mprivozn>
v3.4.0-102-g671d18594
Hi Michal, From comment 5, comment 7 and comment 11, i think this bug is track the issue when there is no numa node in guest xml, access='shared' is not working after guest start. And i have tried this config in latest libvirt, and found that libvirt still will ignore the access='shared': guest config xml: ... <memoryBacking> <hugepages> <page size='2048' unit='KiB' nodeset='0'/> </hugepages> <access mode='shared'/> </memoryBacking> ... <cpu mode='host-model' check='partial'> <model fallback='allow'/> </cpu> ... qemu cmdline after guest start: -m 1536 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/6-r7-mig There is no shared=on (since qemu does not have a option to make the mem-path shared, the only way is use memory-backend-file) And the expected result should be libvirt report an error when start guest (from comment 7) ? Could you please help to check if the test result on latest libvirt is expected ? and did i misunderstand what this bug trying to fix ? thanks in advance for your reply. Yes. This is a bug. I've proposed patch on the list: https://www.redhat.com/archives/libvir-list/2017-December/msg00385.html I've just pushed the patch upstream:
commit 6534b3c4bb146d0c2a8412bd032ce5f356af6e4a
Author: Michal Privoznik <mprivozn>
AuthorDate: Tue Dec 12 14:03:13 2017 +0100
Commit: Michal Privoznik <mprivozn>
CommitDate: Wed Jan 3 15:53:00 2018 +0100
qemuBuildMemPathStr: Forbid memoryBacking/access for non-numa case
https://bugzilla.redhat.com/show_bug.cgi?id=1448149
If a domain has no numa nodes, that means we don't put any
memory-backend-file onto the qemu command line. That in turn
means we can't set access='shared'. Therefore, we should produce
an error instead of ignoring the setting silently.
Signed-off-by: Michal Privoznik <mprivozn>
v3.10.0-95-g6534b3c4b
Verified with libvirt version:
libvirt-4.4.0-2.virtcov.el7.x86_64
If we continue to configure xml as below.
<memoryBacking>
<hugepages>
<page size='2048' unit='KiB' nodeset='0'/>
</hugepages>
<access mode='shared'/>
</memoryBacking>
and no cell is configured.
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
#virsh start avocado-vt-vm-numa
error: Failed to start domain avocado-vt-vm-numa
error: hugepages: node 0 not found
And if nodeset='0' is removed and the memorybacking is still configured as below (no nodeset='*').
<memoryBacking>
<hugepages>
<page size='2048' unit='KiB' />
</hugepages>
<access mode='shared'/>
</memoryBacking>
There is still no "share=on" in qemu command. And no "mem-path" existing in qemu the command. Is this as expected?
qemu 2065 1 39 17:31 ? 00:00:08 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm-numa,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-24-avocado-vt-vm-numa/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,vmport=on,dump-guest-core=off -cpu qemu64,+kvm_pv_unhalt,pmu=off -bios /usr/share/seabios/bios.bin -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid fef59087-9555-4b54-9198-6c644b7ce078 -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-24-avocado-vt-vm-numa/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=1 -boot menu=on,reboot-timeout=0,splash-time=3000,strict=on -device pci-bridge,chassis_nr=1,id=pci.1,bus=pci.0,addr=0x6 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x8 -device pxb,bus_nr=254,id=pci.3,bus=pci.0,addr=0xe -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device ich9-usb-ehci1,id=usb1,bus=pci.0,addr=0x9.0x7 -device ich9-usb-uhci1,masterbus=usb1.0,firstport=0,bus=pci.0,multifunction=on,addr=0x9 -device ich9-usb-uhci2,masterbus=usb1.0,firstport=2,bus=pci.0,addr=0x9.0x1 -device ich9-usb-uhci3,masterbus=usb1.0,firstport=4,bus=pci.0,addr=0x9.0x2 -device nec-usb-xhci,p2=4,p3=4,id=usb2,bus=pci.0,addr=0xa -device qemu-xhci,p2=15,p3=15,id=usb3,bus=pci.0,addr=0xd -device virtio-scsi-pci,id=scsi0,cmd_per_lun=5,max_sectors=10000,ioeventfd=on,bus=pci.0,addr=0x8 -device ahci,id=sata0,bus=pci.0,addr=0xf -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.1,addr=0x5 -device usb-ccid,id=ccid0,bus=usb.0,port=1 -device usb-hub,id=hub0,bus=usb1.0,port=3 -drive file=/var/lib/libvirt/images/demo1.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,serial=f65effa5-90a6-47f2-8487-a9f64c95d4f5,werror=stop,rerror=stop,cache=none,discard=unmap,aio=threads,throttling.bps-total=1000000000000,throttling.iops-read=400000000000,throttling.iops-write=100000000000,throttling.group=group1 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -drive file=/var/lib/libvirt/images/test.raw,format=raw,if=none,id=drive-sata0-0-0 -device ide-hd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0 -netdev tap,fds=30:32:33:34:35,id=hostnet0,vhost=on,vhostfds=36:37:38:39:40 -device virtio-net-pci,tx=bh,ioeventfd=on,event_idx=off,csum=off,gso=off,host_tso4=off,host_tso6=off,host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,mq=on,vectors=12,host_mtu=9000,netdev=hostnet0,id=net0,mac=52:54:00:c6:3b:95,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channel/target/domain-24-avocado-vt-vm-numa/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev pty,id=charchannel2 -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=org.qemu.guest_agent.1 -chardev pty,id=charconsole1 -device virtconsole,chardev=charconsole1,id=console1 -device usb-tablet,id=input0,bus=usb.0,port=2 -device usb-kbd,id=input2,bus=usb1.0,port=1 -spice port=5902,addr=192.168.122.1,agent-mouse=on,disable-ticketing,plaintext-channel=usbredir,image-compression=auto_glz,streaming-video=filter,seamless-migration=on -vnc 0.0.0.0:3 -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=128,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x7 -device qxl,id=video1,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device qxl,id=video2,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0xb -device AC97,id=sound0,bus=pci.0,addr=0x4 -device ib700,id=watchdog0 -watchdog-action none -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb1.0,port=4 -device usb-host,id=hostdev0,bus=usb1.0,port=2 -device virtio-balloon-pci,id=ua-c80aba6e-b6d8-448b-ab6e-8c7b5c29f353,bus=pci.0,addr=0xc,deflate-on-oom=on -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -device pvpanic,ioport=1285 -msg timestamp=on
(In reply to Jing Qi from comment #22) > > #virsh start avocado-vt-vm-numa > error: Failed to start domain avocado-vt-vm-numa > error: hugepages: node 0 not found Patch proposed upstream: https://www.redhat.com/archives/libvir-list/2018-July/msg00272.html > > And if nodeset='0' is removed and the memorybacking is still configured as > below (no nodeset='*'). > <memoryBacking> > <hugepages> > <page size='2048' unit='KiB' /> > </hugepages> > <access mode='shared'/> > </memoryBacking> > > There is still no "share=on" in qemu command. And no "mem-path" existing in > qemu the command. Is this as expected? Yes. Memory sharing can be done only for memory-backend-file. Michal,Thanks for your response. I'll set the bug to verified and the "hugepages: node 0 not found " issue is tracing by 1591235 > > And if nodeset='0' is removed and the memorybacking is still configured as
> > below (no nodeset='*').
> > <memoryBacking>
> > <hugepages>
> > <page size='2048' unit='KiB' />
> > </hugepages>
> > <access mode='shared'/>
> > </memoryBacking>
> >
> > There is still no "share=on" in qemu command. And no "mem-path" existing in
> > qemu the command. Is this as expected?
>
> Yes. Memory sharing can be done only for memory-backend-file.
Hi Michal, I tried this xml and it did use the memory-backend-file, but the "shared='on'" is missing in qemu command line.
So I think if "share='on'" is not supported on UMA guest with hugepages, it should not also report error and not start.
# virsh start rhel
Domain rhel started
# virsh dumpxml rhel
...
<memoryBacking>
<hugepages>
<page size='2048' unit='KiB'/>
</hugepages>
<access mode='shared'/>
</memoryBacking>
<vcpu placement='static'>2</vcpu>
...
qemu cmd:
-m 1024 -mem-prealloc -mem-path /dev/hugepages2M/libvirt/qemu/4-rhel
(In reply to yalzhang from comment #25) > > > And if nodeset='0' is removed and the memorybacking is still configured as > > > below (no nodeset='*'). > > > <memoryBacking> > > > <hugepages> > > > <page size='2048' unit='KiB' /> > > > </hugepages> > > > <access mode='shared'/> > > > </memoryBacking> > > > > > > There is still no "share=on" in qemu command. And no "mem-path" existing in > > > qemu the command. Is this as expected? > > > > Yes. Memory sharing can be done only for memory-backend-file. > > Hi Michal, I tried this xml and it did use the memory-backend-file, but the > "shared='on'" is missing in qemu command line. > So I think if "share='on'" is not supported on UMA guest with hugepages, it > should not also report error and not start. > > # virsh start rhel > Domain rhel started > > # virsh dumpxml rhel > ... > <memoryBacking> > <hugepages> > <page size='2048' unit='KiB'/> > </hugepages> > <access mode='shared'/> > </memoryBacking> > <vcpu placement='static'>2</vcpu> > ... > > qemu cmd: > -m 1024 -mem-prealloc -mem-path /dev/hugepages2M/libvirt/qemu/4-rhel This qemu cmd line says your guest is not using memory-backend-file. Libvirt does not mix -mem-path and memory-backend-file. Can you share the full cmd line and domain XML? Created attachment 1472840 [details]
live xml of the guest
Attached xml of the guest.
test on libvirt-4.5.0-6.virtcov.el7.x86_64
And the qemu cmd as below:
# ps aux | grep mem
qemu 12512 100 0.6 1910844 48360 ? Rl 08:21 0:09 /usr/libexec/qemu-kvm -name guest=rhel,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-rhel/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Penryn
-m 1024 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/1-rhel
-realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid bec6e7a8-872e-4309-a392-c0a1a6c591fc -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=27,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/RHEL-7.6-x86_64-latest.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=29,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:a7:87:59,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
(In reply to yalzhang from comment #27) > Created attachment 1472840 [details] > live xml of the guest > > Attached xml of the guest. > test on libvirt-4.5.0-6.virtcov.el7.x86_64 > And the qemu cmd as below: > # ps aux | grep mem > qemu 12512 100 0.6 1910844 48360 ? Rl 08:21 0:09 > /usr/libexec/qemu-kvm -name guest=rhel,debug-threads=on -S -object > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-rhel/ > master-key.aes -machine > pc-i440fx-rhel7.6.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu > Penryn > > -m 1024 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/1-rhel > > -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid > bec6e7a8-872e-4309-a392-c0a1a6c591fc -no-user-config -nodefaults -chardev > socket,id=charmonitor,fd=27,server,nowait -mon > chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew > -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global > PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device > ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device > ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on, > addr=0x5 -device > ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device > ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive > file=/var/lib/libvirt/images/RHEL-7.6-x86_64-latest.qcow2,format=qcow2, > if=none,id=drive-ide0-0-0 -device > ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev > tap,fd=29,id=hostnet0 -device > rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:a7:87:59,bus=pci.0,addr=0x3 > -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 -chardev > spicevmc,id=charchannel0,name=vdagent -device > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0, > name=com.redhat.spice.0 -spice > port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless- > migration=on -device > qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0, > vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device > intel-hda,id=sound0,bus=pci.0,addr=0x4 -device > hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev > spicevmc,id=charredir0,name=usbredir -device > usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev > spicevmc,id=charredir1,name=usbredir -device > usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -sandbox > on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg > timestamp=on So there is not memory-backend-file on the command line. As I said earlier, if that's the case, libvirt has no way of configuring memAccess. Therefore this is no bug and should be switched to VERIFIED back again. Verified in version libvirt-4.5.0-7.virtcov.el7.x86_64.
With domain config as below:
<memoryBacking>
<hugepages>
<page size='2048' unit='KiB' nodeset='0'/>
</hugepages>
<access mode='shared'/>
</memoryBacking>
....
<cpu>
....
<numa>
<cell id='0' cpus='0-1' memory='512000' unit='KiB'/>
</numa>
</cpu>
Or:
<memoryBacking>
<hugepages>
<page size='2048' unit='KiB' nodeset='0'/>
</hugepages>
</memoryBacking>
<numa>
<cell id='0' cpus='0-1' memory='512000' unit='KiB' memAccess='shared'/>
</numa>
</cpu>
qemu-command includes 'share=yes':
/usr/libexec/qemu-kvm -name guest=avocado-vt-vm-numa,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-avocado-vt-vm-numa/master-key.aes -machine pc-i440fx-rhel7.5.0,accel=kvm,usb=off,dump-guest-core=off -cpu SandyBridge-IBRS,vme=on,ds=on,acpi=on,ss=on,ht=on,tm=on,pbe=on,dtes64=on,monitor=on,ds_cpl=on,vmx=on,smx=on,est=on,tm2=on,xtpr=on,pdcm=on,pcid=on,arat=on,tsc-deadline=off,xsave=off,avx=off,+kvm_pv_unhalt -bios /usr/share/seabios/bios.bin -m size=512000k,slots=8,maxmem=8938496k -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/2-avocado-vt-vm-numa,share=yes,size=524288000 -numa node,nodeid=0,cpus=0-1,memdev=ram-node0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:3113 |
Description of problem: Using virsh edit, I have entered the below xml and the resulting qemu command line does not have hugepages shared. <memoryBacking> <hugepages> <page size='524288' unit='KiB' nodeset='0'/> </hugepages> <access mode='shared'/> </memoryBacking> Also tried to use the below and virsh edit would not accept it: <cpu> <numa> <cell id='0' cpus='0' memory='1048576' unit='KiB' memAccess='shared'/> </numa> </cpu> error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Extra element cpu in interleave Element domain failed to validate content Version-Release number of selected component (if applicable): Pegas 4.10.0-7.el7.aarch64 libvirt 3.1.0 qemu-kvm 2.8.0 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: