Bug 1256014 - qemu failed to hot-unplug memory
qemu failed to hot-unplug memory
Status: CLOSED DUPLICATE of bug 1245892
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
x86_64 Linux
unspecified Severity medium
: rc
: ---
Assigned To: Igor Mammedov
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-23 05:14 EDT by Xiaoqing Wei
Modified: 2015-08-28 07:41 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-08-28 07:41:49 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
serial log (43.34 KB, text/plain)
2015-08-23 05:18 EDT, Xiaoqing Wei
no flags Details

  None (edit)
Description Xiaoqing Wei 2015-08-23 05:14:46 EDT
Description of problem:
qemu failed to hot-unplug memory

Version-Release number of selected component (if applicable):
kernel-3.10.0-306.el7.x86_64   both host and guest
qemu-kvm-rhev-2.3.0-18.el7.x86_64
seabios-bin-1.7.5-11.el7.noarch


How reproducible:
5/5

Steps to Reproduce:
1. define a vm with MaxMemory slot to allow mem hot plug
xml snip, full xml would be in separated comment later.

<domain type='kvm' id='4'>
  <name>rhel72-w34</name>
  <uuid>5ace6117-a351-41ba-a91f-a5ba657a59cf</uuid>
  <maxMemory slots='16' unit='KiB'>9765888</maxMemory>
  <memory unit='KiB'>1536000</memory>
  <currentMemory unit='KiB'>1536000</currentMemory>

    <numa>
      <cell id='0' cpus='0-1' memory='512000' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='512000' unit='KiB'/>
    </numa>

2015-08-23 08:46:02.512+0000: starting up libvirt version: 1.2.17, package: 2.el7 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2015-07-10-07:33:51, x86-035.build.eng.bos.redhat.com), qemu version: 2.3.0 (qemu-kvm-rhev-2.3.0-18.el7)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name rhel72-w34 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off,vmport=off -cpu Opteron_G5,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -m size=1024000k,slots=16,maxmem=9765888k -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -numa node,nodeid=0,cpus=0-1,mem=500 -numa node,nodeid=1,cpus=2-3,mem=500 -uuid 5ace6117-a351-41ba-a91f-a5ba657a59cf -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel72-w34.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot order=c,menu=on,strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x8 -drive file=/home/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/RHEL-Server-7.2-64-virtio.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:90:b4:93,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/win2k12.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -spice port=5900,addr=0.0.0.0,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device ich9-intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa -msg timestamp=on



2. boot the vm, then check memory usage by top
[root@dhcp-11-50 staf-kvm-devel]# virsh start rhel72-w34
Domain rhel72-w34 started
top -Hp`pidof qemu-kvm` -n1 > top-just-boot.txt


3. hot plug mem dev and verify by top/monitor
[root@dhcp-11-50 staf-kvm-devel]# cat memdevice.xml 
    <memory model='dimm'>
      <target>
        <size unit='m'>500</size>
        <node>1</node>
      </target>
    </memory>
[root@dhcp-11-50 staf-kvm-devel]# virsh attach-device rhel72-w34 memdevice.xml
Device attached successfully

top -Hp`pidof qemu-kvm` -n1 > top-just-plugged.txt
virsh qemu-monitor-command rhel72-w34 --hmp --cmd 'info memory-devices' > info-memory-devices-just-plugged.txt

4. hot unplug mem dev and verify by top/monitor

[root@dhcp-11-50 staf-kvm-devel]# virsh detach-device rhel72-w34 memdevice.xml
Device detached successfully

[root@dhcp-11-50 staf-kvm-devel]# echo $?
0

top -Hp`pidof qemu-kvm` -n1 > top-unplugged.txt
virsh qemu-monitor-command rhel72-w34 --hmp --cmd 'info memory-devices' > info-memory-devices-unplugged.txt


Actual results:
virsh returned 0, means qemu cmd execution success, but it's actually failed, by checking top/monitor, and guest dmesg.

tail -20 memory-unplug-failed-dmesg-vm.log
[   42.078928] virbr0: port 1(virbr0-nic) entered disabled state
[   43.279463] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[   50.272999] fuse init (API version 7.22)
[   50.280258] SELinux: initialized (dev fuse, type fuse), uses genfs_contexts
[   50.298311] SELinux: initialized (dev fusectl, type fusectl), uses genfs_contexts
[   51.460459] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[   52.504535] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[   91.856775] ACPI: \_SB_.MP00: ACPI_NOTIFY_DEVICE_CHECK event
[   91.857950] init_memory_mapping: [mem 0x100000000-0x11f3fffff]
[   91.859646]  [mem 0x100000000-0x11f3fffff] page 2M
[   91.865410]  [ffffea0004000000-ffffea00043fffff] PMD -> [ffff880004000000-ffff8800043fffff] on node 1
[   91.885437] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 168818
[   91.886641] Policy zone: DMA32
[   91.902267] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 201586
[   91.903500] Policy zone: Normal
[  270.325421] ACPI: \_SB_.MP00: ACPI_NOTIFY_EJECT_REQUEST event
[  270.331220] Offlined Pages 32768
[  270.333991] Offlined Pages 32768
[  270.484802] memory offlining [mem 0x118000000-0x11fffffff] failed
[  270.486617] memory memory34: Offline failed.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expected results:
hot unplug should be either success, or return fail event to virsh

Additional info:

[root@dhcp-11-50 ~]# cat info-memory-devices-just-plugged.txt
Memory device [dimm]: "dimm0"
  addr: 0x100000000
  slot: 0
  node: 1
  size: 524288000
  memdev: /objects/memdimm0
  hotplugged: true
  hotpluggable: true




[root@dhcp-11-50 ~]# cat info-memory-devices-unplugged.txt
Memory device [dimm]: "dimm0"
  addr: 0x100000000
  slot: 0
  node: 1
  size: 524288000
  memdev: /objects/memdimm0
  hotplugged: true
  hotpluggable: true
Comment 2 Xiaoqing Wei 2015-08-23 05:17:16 EDT
cat top-just-boot.txt 
top - 16:47:25 up 32 min,  4 users,  load average: 1.88, 1.26, 1.02
Threads:  71 total,   0 running,  71 sleeping,   0 stopped,   0 zombie
%Cpu0  :  3.2 us,  0.7 sy,  0.0 ni, 90.9 id,  5.2 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  2.6 us,  0.6 sy,  0.0 ni, 93.9 id,  2.9 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  3.0 us,  0.6 sy,  0.0 ni, 91.9 id,  4.5 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  2.8 us,  0.8 sy,  0.0 ni, 93.0 id,  3.4 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7014592 total,  4973160 free,  1378808 used,   662624 buff/cache
KiB Swap:  7356412 total,  7356412 free,        0 used.  5362120 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                 
 4263 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:02.78 qemu-kvm                                
 4270 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.07 qemu-kvm                                
 4273 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.14 qemu-kvm                                
 4274 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:13.83 qemu-kvm                                
 4275 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:06.59 qemu-kvm                                
 4276 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:06.08 qemu-kvm                                
 4277 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:06.14 qemu-kvm                                
 4279 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:01.13 qemu-kvm                                
 4282 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.06 qemu-kvm                                
 4283 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.05 qemu-kvm                                
 4284 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.06 qemu-kvm                                
 4285 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.06 qemu-kvm                                
 4286 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.04 qemu-kvm                                
 4287 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.04 qemu-kvm                                
 4288 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.04 qemu-kvm                                
 4289 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.04 qemu-kvm                                
 4290 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.04 qemu-kvm                                
 4291 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.04 qemu-kvm                                
 4292 qemu      20   0 2190080 864072  11544 S  0.0 12.3   0:00.03 qemu-kvm                                



cat top-just-plugged.txt
top - 16:48:05 up 32 min,  4 users,  load average: 1.37, 1.21, 1.01
Threads:   8 total,   0 running,   8 sleeping,   0 stopped,   0 zombie
%Cpu0  :  3.2 us,  0.7 sy,  0.0 ni, 91.0 id,  5.1 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  2.6 us,  0.6 sy,  0.0 ni, 93.9 id,  2.9 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  2.9 us,  0.6 sy,  0.0 ni, 92.0 id,  4.5 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  2.8 us,  0.8 sy,  0.0 ni, 93.1 id,  3.4 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7014592 total,  4921632 free,  1430468 used,   662492 buff/cache
KiB Swap:  7356412 total,  7356412 free,        0 used.  5310636 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                 
 4263 qemu      20   0 2210320 915784  11552 S  0.0 13.1   0:02.98 qemu-kvm                                
 4270 qemu      20   0 2210320 915784  11552 S  0.0 13.1   0:00.07 qemu-kvm                                
 4274 qemu      20   0 2210320 915784  11552 S  0.0 13.1   0:14.10 qemu-kvm                                
 4275 qemu      20   0 2210320 915784  11552 S  0.0 13.1   0:07.31 qemu-kvm                                
 4276 qemu      20   0 2210320 915784  11552 S  0.0 13.1   0:06.74 qemu-kvm                                
 4277 qemu      20   0 2210320 915784  11552 S  0.0 13.1   0:06.86 qemu-kvm                                
 4279 qemu      20   0 2210320 915784  11552 S  0.0 13.1   0:01.13 qemu-kvm                                
 4375 qemu      20   0 2210320 915784  11552 S  0.0 13.1   0:00.00 qemu-kvm                                




cat top-unplugged.txt
top - 16:51:07 up 35 min,  4 users,  load average: 0.20, 0.74, 0.86
Threads:   7 total,   0 running,   7 sleeping,   0 stopped,   0 zombie
%Cpu0  :  3.1 us,  0.6 sy,  0.0 ni, 91.5 id,  4.8 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  2.6 us,  0.5 sy,  0.0 ni, 94.2 id,  2.6 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  2.9 us,  0.6 sy,  0.0 ni, 92.4 id,  4.1 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  2.7 us,  0.7 sy,  0.0 ni, 93.4 id,  3.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7014592 total,  4796344 free,  1555696 used,   662552 buff/cache
KiB Swap:  7356412 total,  7356412 free,        0 used.  5185396 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                 
 4274 qemu      20   0 2222624 0.992g  12128 S  6.7 14.8   0:17.04 qemu-kvm                                
 4263 qemu      20   0 2222624 0.992g  12128 S  0.0 14.8   0:04.40 qemu-kvm                                
 4270 qemu      20   0 2222624 0.992g  12128 S  0.0 14.8   0:00.07 qemu-kvm                                
 4275 qemu      20   0 2222624 0.992g  12128 S  0.0 14.8   0:11.31 qemu-kvm                                
 4276 qemu      20   0 2222624 0.992g  12128 S  0.0 14.8   0:09.96 qemu-kvm                                
 4277 qemu      20   0 2222624 0.992g  12128 S  0.0 14.8   0:10.71 qemu-kvm                                
 4279 qemu      20   0 2222624 0.992g  12128 S  0.0 14.8   0:01.67 qemu-kvm
Comment 3 Xiaoqing Wei 2015-08-23 05:18:16 EDT
Created attachment 1066020 [details]
serial log
Comment 4 Xiaoqing Wei 2015-08-23 05:29:39 EDT

virsh dumpxml rhel72-w34   # of the point after unplug operation

<domain type='kvm' id='4'>
  <name>rhel72-w34</name>
  <uuid>5ace6117-a351-41ba-a91f-a5ba657a59cf</uuid>
  <maxMemory slots='16' unit='KiB'>9765888</maxMemory>
  <memory unit='KiB'>1536000</memory>
  <currentMemory unit='KiB'>1536000</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Opteron_G5</model>
    <numa>
      <cell id='0' cpus='0-1' memory='512000' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='512000' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/home/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/RHEL-Server-7.2-64-virtio.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:90:b4:93'/>
      <source bridge='switch'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/3'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/3'>
      <source path='/dev/pts/3'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/win2k12.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <sound model='ich9'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </memballoon>
    <memory model='dimm'>
      <target>
        <size unit='KiB'>512000</size>
        <node>1</node>
      </target>
      <alias name='dimm0'/>
      <address type='dimm' slot='0' base='0x100000000'/>
    </memory>
  </devices>
</domain>
Comment 5 Pei Zhang 2015-08-23 22:12:35 EDT
Seems this issue is as same as https://bugzilla.redhat.com/show_bug.cgi?id=1245892#c6
Comment 6 Xiaoqing Wei 2015-08-23 22:32:42 EDT
(In reply to zhangpei from comment #5)
> Seems this issue is as same as
> https://bugzilla.redhat.com/show_bug.cgi?id=1245892#c6

From the xml definition, yes, looks alike to me,

also the Expected results:
hot unplug should be either success, or return fail event to virsh


Let's wait for someone in dev team to confirm
Comment 7 Igor Mammedov 2015-08-28 07:41:49 EDT

*** This bug has been marked as a duplicate of bug 1245892 ***

Note You need to log in before you can comment on or make changes to this bug.