Bug 1448149

Summary: <memoryBacking> <access mode='shared'/> is ignored if no NUMA nodes are configured
Product: Red Hat Enterprise Linux 7 Reporter: Bill Townsend <wtownsen>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: Jing Qi <jinqi>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.4CC: dyuan, jdenemar, jtomko, mprivozn, wtownsen, xuzhang, yalzhang
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-4.3.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-30 09:49:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
live xml of the guest none

Description Bill Townsend 2017-05-04 15:52:13 UTC
Description of problem:
Using virsh edit, I have entered the below xml and the resulting qemu command line does not have hugepages shared.

  <memoryBacking>
    <hugepages>
      <page size='524288' unit='KiB' nodeset='0'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>

Also tried to use the below and virsh edit would not accept it:

  <cpu>
    <numa>
      <cell id='0' cpus='0' memory='1048576' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>

error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng
Extra element cpu in interleave
Element domain failed to validate content


Version-Release number of selected component (if applicable):
Pegas  4.10.0-7.el7.aarch64 
libvirt 3.1.0
qemu-kvm 2.8.0

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Peter Krempa 2017-05-04 16:00:47 UTC
Could you please attach the full XML? I tried both snippets as you've provided them and they work. It's possible that you've placed them incorrectly.

Comment 3 Bill Townsend 2017-05-04 16:52:53 UTC
OK, I tried the below alone and hugepages were not shared.


<domain type='kvm'>
  <name>rhel7.3-rhelsa</name>
  <uuid>05a207a4-e6ef-4eac-b9c2-39db94d1939e</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='524288' unit='KiB' nodeset='0'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='aarch64' machine='virt-rhel7.4.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/rhel7.3-rhelsa_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <gic version='3'/>
  </features>
  <cpu mode='host-passthrough'/>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel7.3-rhelsa.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='vhostuser'>
      <mac address='00:be:ef:ed:00:01'/>
      <source type='unix' path='/usr/local/var/run/openvswitch/guestdpdk1' mode='client'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
  </devices>
</domain>

Comment 4 Bill Townsend 2017-05-04 16:56:44 UTC
The numa version I tried where it rejected it:

<domain type='kvm'>
  <name>rhel7.3-rhelsa-2</name>
  <uuid>ee88d7c2-d7c0-4070-9234-f6267481a6c8</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='524288' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <cpu>
    <numa>
      <cell id='0' cpus='0' memory='2097152' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='aarch64' machine='virt-rhel7.4.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/rhel7.3-rhelsa-2_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <gic version='3'/>
  </features>
  <cpu mode='host-passthrough'/>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel7.3-rhelsa-2.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:63:31:bb'/>
      <source network='default'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:00:ba:be'/>
      <source bridge='br1'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='fcad8498-17c9-4c9d-a148-891dc3af6f38'/>
      </virtualport>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='00:be:ef:ed:00:07'/>
      <source type='unix' path='/usr/local/var/run/openvswitch/guestdpdk2' mode='client'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
  </devices>
</domain>

Comment 5 Bill Townsend 2017-05-04 17:44:03 UTC
Oh, here is the command line when using memoryBacking husgepages shared. I dont see something like this: 
 -object memory-backend-file,id=mem,size=4G,mem-path=/dev/hugepages,share=on 

qemu     17251 89.1  0.2 2499328 187200 ?      Sl   13:34   0:20 /usr/libexec/qemu-kvm -name guest=rhel7.3-rhelsa,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-rhel7.3-rhelsa/master-key.aes -machine virt-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -cpu host -drive file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/rhel7.3-rhelsa_VARS.fd,if=pflash,format=raw,unit=1 -m 2048 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/6-rhel7.3-rhelsa -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 05a207a4-e6ef-4eac-b9c2-39db94d1939e -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-rhel7.3-rhelsa/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 -device ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 -device ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 -device ioh3420,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x0 -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 -drive file=/var/lib/libvirt/images/rhel7.3-rhelsa.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -chardev socket,id=charnet0,path=/usr/local/var/run/openvswitch/guestdpdk1 -netdev vhost-user,chardev=charnet0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:be:ef:ed:00:01,bus=pcie.0,addr=0x9 -serial pty -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-6-rhel7.3-rhelsa/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -msg timestamp=on
root     17263  0.0  0.0 110272  2496 ttyAMA0  S+   13:34   0:00 grep --color=auto qemu

Comment 6 Ján Tomko 2017-05-05 07:34:07 UTC
(In reply to Bill Townsend from comment #4)
> The numa version I tried where it rejected it:
> 
> <domain type='kvm'>
...
>   </memoryBacking>
>   <cpu>
>     <numa>
>       <cell id='0' cpus='0' memory='2097152' unit='KiB' memAccess='shared'/>
>     </numa>
>   </cpu>
>   <vcpu placement='static'>1</vcpu>
..
>   </features>
>   <cpu mode='host-passthrough'/>
>   <clock offset='utc'/>

You specified the <cpu> element twice.

Comment 7 Peter Krempa 2017-05-05 12:25:37 UTC
Looking at the code, specifying
 <memoryBacking>
    <access mode='shared'/>

is not enough, since the shared memory object is created only for a NUMA node. So going the NUMA way is desired.

Libvirt should not allow starting a VM if the config is not valid though.

Comment 8 Bill Townsend 2017-05-05 14:26:28 UTC
(In reply to Ján Tomko from comment #6)
> (In reply to Bill Townsend from comment #4)
> > The numa version I tried where it rejected it:
> > 
> > <domain type='kvm'>
> ...
> >   </memoryBacking>
> >   <cpu>
> >     <numa>
> >       <cell id='0' cpus='0' memory='2097152' unit='KiB' memAccess='shared'/>
> >     </numa>
> >   </cpu>
> >   <vcpu placement='static'>1</vcpu>
> ..
> >   </features>
> >   <cpu mode='host-passthrough'/>
> >   <clock offset='utc'/>
> 
> You specified the <cpu> element twice.

ACK, you are right. No one else caught that. I changed to:
  <cpu mode='host-passthrough'>
    <numa>
      <cell id='0' cpus='0' memory='1048576' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
And that was accepted. Also tested it and it does work. Thanks!

Comment 9 Bill Townsend 2017-05-05 14:28:11 UTC
(In reply to Peter Krempa from comment #7)
> Looking at the code, specifying
>  <memoryBacking>
>     <access mode='shared'/>
> 
> is not enough, since the shared memory object is created only for a NUMA
> node. So going the NUMA way is desired.
> 
> Libvirt should not allow starting a VM if the config is not valid though.


OK, so what do you want to do with this part? Thanks.

Comment 10 Bill Townsend 2017-05-05 14:31:43 UTC
(In reply to Bill Townsend from comment #9)
> (In reply to Peter Krempa from comment #7)
> > Looking at the code, specifying
> >  <memoryBacking>
> >     <access mode='shared'/>
> > 
> > is not enough, since the shared memory object is created only for a NUMA
> > node. So going the NUMA way is desired.
> > 
> > Libvirt should not allow starting a VM if the config is not valid though.
> 
> 
> OK, so what do you want to do with this part? Thanks.

Looking at the domain model documentation under memoryBacking it reads:
access
    Specify if memory is shared or private. This can be overridden per numa node by memAccess

Comment 11 Peter Krempa 2017-05-05 14:41:12 UTC
Thanks for confirming this. I'll repurpose this bug to track that <access mode='shared'/> is ignored if no NUMA nodes are configured.

Comment 12 Michal Privoznik 2017-08-31 14:13:09 UTC
This is already fixed upstream:

commit 671d18594f42f24f7cb3f1b2c574ff4ddba0b183
Author: Michal Privoznik <mprivozn@redhat.com>
Date:   Tue Jun 6 10:58:37 2017 +0200

    qemu: Allow memAccess for hugepages again
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1214369
    https://bugzilla.redhat.com/show_bug.cgi?id=1458638
    
    Historically, we've always supported memAccess for domains backed
    by hugepages. However, somewhere along the way we've regressed
    and stopped allowing such configuration. Fix it.
    
    Signed-off-by: Michal Privoznik <mprivozn@redhat.com>

v3.4.0-102-g671d18594

Comment 14 Luyao Huang 2017-12-06 08:22:39 UTC
Hi Michal,

From comment 5, comment 7 and comment 11, i think this bug is track the issue when there is no numa node in guest xml, access='shared' is not working after guest start. And i have tried this config in latest libvirt, and found that libvirt still will ignore the access='shared':

guest config xml:

...
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
...
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
...

qemu cmdline after guest start:

 -m 1536 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/6-r7-mig

There is no shared=on (since qemu does not have a option to make the mem-path shared, the only way is use memory-backend-file)
And the expected result should be libvirt report an error when start guest (from comment 7) ?

Could you please help to check if the test result on latest libvirt is expected ? and did i misunderstand what this bug trying to fix ? thanks in advance for your reply.

Comment 15 Michal Privoznik 2017-12-12 13:40:01 UTC
Yes. This is a bug. I've proposed patch on the list:

https://www.redhat.com/archives/libvir-list/2017-December/msg00385.html

Comment 17 Michal Privoznik 2018-01-03 15:00:01 UTC
I've just pushed the patch upstream:

commit 6534b3c4bb146d0c2a8412bd032ce5f356af6e4a
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Tue Dec 12 14:03:13 2017 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Wed Jan 3 15:53:00 2018 +0100

    qemuBuildMemPathStr: Forbid memoryBacking/access for non-numa case
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1448149
    
    If a domain has no numa nodes, that means we don't put any
    memory-backend-file onto the qemu command line. That in turn
    means we can't set access='shared'. Therefore, we should produce
    an error instead of ignoring the setting silently.
    
    Signed-off-by: Michal Privoznik <mprivozn@redhat.com>



v3.10.0-95-g6534b3c4b

Comment 22 Jing Qi 2018-07-04 09:34:54 UTC
Verified with libvirt version:
libvirt-4.4.0-2.virtcov.el7.x86_64

If we continue to configure xml as below.
<memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>

and no cell is configured.
 <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

#virsh start avocado-vt-vm-numa
error: Failed to start domain avocado-vt-vm-numa
error: hugepages: node 0 not found

And if nodeset='0' is removed and the memorybacking is still configured as below (no nodeset='*').
<memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' />
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>

There is still no "share=on" in qemu command. And no "mem-path" existing in qemu the command. Is this as expected?

qemu      2065     1 39 17:31 ?        00:00:08 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm-numa,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-24-avocado-vt-vm-numa/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,vmport=on,dump-guest-core=off -cpu qemu64,+kvm_pv_unhalt,pmu=off -bios /usr/share/seabios/bios.bin -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid fef59087-9555-4b54-9198-6c644b7ce078 -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-24-avocado-vt-vm-numa/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=1 -boot menu=on,reboot-timeout=0,splash-time=3000,strict=on -device pci-bridge,chassis_nr=1,id=pci.1,bus=pci.0,addr=0x6 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x8 -device pxb,bus_nr=254,id=pci.3,bus=pci.0,addr=0xe -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device ich9-usb-ehci1,id=usb1,bus=pci.0,addr=0x9.0x7 -device ich9-usb-uhci1,masterbus=usb1.0,firstport=0,bus=pci.0,multifunction=on,addr=0x9 -device ich9-usb-uhci2,masterbus=usb1.0,firstport=2,bus=pci.0,addr=0x9.0x1 -device ich9-usb-uhci3,masterbus=usb1.0,firstport=4,bus=pci.0,addr=0x9.0x2 -device nec-usb-xhci,p2=4,p3=4,id=usb2,bus=pci.0,addr=0xa -device qemu-xhci,p2=15,p3=15,id=usb3,bus=pci.0,addr=0xd -device virtio-scsi-pci,id=scsi0,cmd_per_lun=5,max_sectors=10000,ioeventfd=on,bus=pci.0,addr=0x8 -device ahci,id=sata0,bus=pci.0,addr=0xf -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.1,addr=0x5 -device usb-ccid,id=ccid0,bus=usb.0,port=1 -device usb-hub,id=hub0,bus=usb1.0,port=3 -drive file=/var/lib/libvirt/images/demo1.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,serial=f65effa5-90a6-47f2-8487-a9f64c95d4f5,werror=stop,rerror=stop,cache=none,discard=unmap,aio=threads,throttling.bps-total=1000000000000,throttling.iops-read=400000000000,throttling.iops-write=100000000000,throttling.group=group1 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -drive file=/var/lib/libvirt/images/test.raw,format=raw,if=none,id=drive-sata0-0-0 -device ide-hd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0 -netdev tap,fds=30:32:33:34:35,id=hostnet0,vhost=on,vhostfds=36:37:38:39:40 -device virtio-net-pci,tx=bh,ioeventfd=on,event_idx=off,csum=off,gso=off,host_tso4=off,host_tso6=off,host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,mq=on,vectors=12,host_mtu=9000,netdev=hostnet0,id=net0,mac=52:54:00:c6:3b:95,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channel/target/domain-24-avocado-vt-vm-numa/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev pty,id=charchannel2 -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=org.qemu.guest_agent.1 -chardev pty,id=charconsole1 -device virtconsole,chardev=charconsole1,id=console1 -device usb-tablet,id=input0,bus=usb.0,port=2 -device usb-kbd,id=input2,bus=usb1.0,port=1 -spice port=5902,addr=192.168.122.1,agent-mouse=on,disable-ticketing,plaintext-channel=usbredir,image-compression=auto_glz,streaming-video=filter,seamless-migration=on -vnc 0.0.0.0:3 -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=128,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x7 -device qxl,id=video1,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device qxl,id=video2,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0xb -device AC97,id=sound0,bus=pci.0,addr=0x4 -device ib700,id=watchdog0 -watchdog-action none -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb1.0,port=4 -device usb-host,id=hostdev0,bus=usb1.0,port=2 -device virtio-balloon-pci,id=ua-c80aba6e-b6d8-448b-ab6e-8c7b5c29f353,bus=pci.0,addr=0xc,deflate-on-oom=on -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -device pvpanic,ioport=1285 -msg timestamp=on

Comment 23 Michal Privoznik 2018-07-04 11:55:03 UTC
(In reply to Jing Qi from comment #22)

> 
> #virsh start avocado-vt-vm-numa
> error: Failed to start domain avocado-vt-vm-numa
> error: hugepages: node 0 not found

Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2018-July/msg00272.html

> 
> And if nodeset='0' is removed and the memorybacking is still configured as
> below (no nodeset='*').
> <memoryBacking>
>     <hugepages>
>       <page size='2048' unit='KiB' />
>     </hugepages>
>     <access mode='shared'/>
>   </memoryBacking>
> 
> There is still no "share=on" in qemu command. And no "mem-path" existing in
> qemu the command. Is this as expected?

Yes. Memory sharing can be done only for memory-backend-file.

Comment 24 Jing Qi 2018-07-17 02:14:14 UTC
Michal,Thanks for your response. 
I'll set the bug to verified and the "hugepages: node 0 not found " issue is tracing by 1591235

Comment 25 yalzhang@redhat.com 2018-08-01 02:06:23 UTC
> > And if nodeset='0' is removed and the memorybacking is still configured as
> > below (no nodeset='*').
> > <memoryBacking>
> >     <hugepages>
> >       <page size='2048' unit='KiB' />
> >     </hugepages>
> >     <access mode='shared'/>
> >   </memoryBacking>
> > 
> > There is still no "share=on" in qemu command. And no "mem-path" existing in
> > qemu the command. Is this as expected?
> 
> Yes. Memory sharing can be done only for memory-backend-file.

Hi Michal, I tried this xml and it did use the memory-backend-file, but the "shared='on'" is missing in qemu command line. 
So I think if "share='on'" is not supported on UMA guest with hugepages, it should not also report error and not start.

# virsh start rhel
Domain rhel started

# virsh dumpxml rhel
...
 <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
...

qemu cmd:
-m 1024 -mem-prealloc -mem-path /dev/hugepages2M/libvirt/qemu/4-rhel

Comment 26 Michal Privoznik 2018-08-02 08:19:13 UTC
(In reply to yalzhang@redhat.com from comment #25)
> > > And if nodeset='0' is removed and the memorybacking is still configured as
> > > below (no nodeset='*').
> > > <memoryBacking>
> > >     <hugepages>
> > >       <page size='2048' unit='KiB' />
> > >     </hugepages>
> > >     <access mode='shared'/>
> > >   </memoryBacking>
> > > 
> > > There is still no "share=on" in qemu command. And no "mem-path" existing in
> > > qemu the command. Is this as expected?
> > 
> > Yes. Memory sharing can be done only for memory-backend-file.
> 
> Hi Michal, I tried this xml and it did use the memory-backend-file, but the
> "shared='on'" is missing in qemu command line. 
> So I think if "share='on'" is not supported on UMA guest with hugepages, it
> should not also report error and not start.
> 
> # virsh start rhel
> Domain rhel started
> 
> # virsh dumpxml rhel
> ...
>  <memoryBacking>
>     <hugepages>
>       <page size='2048' unit='KiB'/>
>     </hugepages>
>     <access mode='shared'/>
>   </memoryBacking>
>   <vcpu placement='static'>2</vcpu>
> ...
> 
> qemu cmd:
> -m 1024 -mem-prealloc -mem-path /dev/hugepages2M/libvirt/qemu/4-rhel

This qemu cmd line says your guest is not using memory-backend-file. Libvirt does not mix -mem-path and memory-backend-file. Can you share the full cmd line and domain XML?

Comment 27 yalzhang@redhat.com 2018-08-03 00:28:50 UTC
Created attachment 1472840 [details]
live xml of the guest

Attached xml of the guest.
test on libvirt-4.5.0-6.virtcov.el7.x86_64
And the qemu cmd as below:
# ps aux | grep mem
qemu     12512  100  0.6 1910844 48360 ?       Rl   08:21   0:09 /usr/libexec/qemu-kvm -name guest=rhel,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-rhel/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Penryn 

-m 1024 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/1-rhel 

-realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid bec6e7a8-872e-4309-a392-c0a1a6c591fc -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=27,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/RHEL-7.6-x86_64-latest.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=29,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:a7:87:59,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

Comment 28 Michal Privoznik 2018-08-03 08:54:44 UTC
(In reply to yalzhang@redhat.com from comment #27)
> Created attachment 1472840 [details]
> live xml of the guest
> 
> Attached xml of the guest.
> test on libvirt-4.5.0-6.virtcov.el7.x86_64
> And the qemu cmd as below:
> # ps aux | grep mem
> qemu     12512  100  0.6 1910844 48360 ?       Rl   08:21   0:09
> /usr/libexec/qemu-kvm -name guest=rhel,debug-threads=on -S -object
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-rhel/
> master-key.aes -machine
> pc-i440fx-rhel7.6.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu
> Penryn 
> 
> -m 1024 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/1-rhel 
> 
> -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
> bec6e7a8-872e-4309-a392-c0a1a6c591fc -no-user-config -nodefaults -chardev
> socket,id=charmonitor,fd=27,server,nowait -mon
> chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
> -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global
> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device
> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device
> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,
> addr=0x5 -device
> ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device
> ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
> file=/var/lib/libvirt/images/RHEL-7.6-x86_64-latest.qcow2,format=qcow2,
> if=none,id=drive-ide0-0-0 -device
> ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev
> tap,fd=29,id=hostnet0 -device
> rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:a7:87:59,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -chardev
> spicevmc,id=charchannel0,name=vdagent -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,
> name=com.redhat.spice.0 -spice
> port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-
> migration=on -device
> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,
> vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device
> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
> spicevmc,id=charredir0,name=usbredir -device
> usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev
> spicevmc,id=charredir1,name=usbredir -device
> usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -sandbox
> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg
> timestamp=on

So there is not memory-backend-file on the command line. As I said earlier, if that's the case, libvirt has no way of configuring memAccess. Therefore this is no bug and should be switched to VERIFIED back again.

Comment 32 Jing Qi 2018-09-18 03:58:40 UTC
Verified in version libvirt-4.5.0-7.virtcov.el7.x86_64.

With domain config as below:

 <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>

....
<cpu>
....
 <numa>
      <cell id='0' cpus='0-1' memory='512000' unit='KiB'/>
    </numa>
</cpu>


Or:
<memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
</memoryBacking>
 <numa>
      <cell id='0' cpus='0-1' memory='512000' unit='KiB' memAccess='shared'/>
    </numa>
</cpu>


qemu-command includes 'share=yes':
/usr/libexec/qemu-kvm -name guest=avocado-vt-vm-numa,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-avocado-vt-vm-numa/master-key.aes -machine pc-i440fx-rhel7.5.0,accel=kvm,usb=off,dump-guest-core=off -cpu SandyBridge-IBRS,vme=on,ds=on,acpi=on,ss=on,ht=on,tm=on,pbe=on,dtes64=on,monitor=on,ds_cpl=on,vmx=on,smx=on,est=on,tm2=on,xtpr=on,pdcm=on,pcid=on,arat=on,tsc-deadline=off,xsave=off,avx=off,+kvm_pv_unhalt -bios /usr/share/seabios/bios.bin -m size=512000k,slots=8,maxmem=8938496k -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/2-avocado-vt-vm-numa,share=yes,size=524288000 -numa node,nodeid=0,cpus=0-1,memdev=ram-node0

Comment 34 errata-xmlrpc 2018-10-30 09:49:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113