Bug 1280420
Summary: | VM with attached VFIO device is powered off when trying to hotplug increase memory of VM. [7.2.z] | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Jan Kurik <jkurik> |
Component: | libvirt | Assignee: | Peter Krempa <pkrempa> |
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
Severity: | urgent | Docs Contact: | |
Priority: | high | ||
Version: | 7.2 | CC: | abologna, alex.williamson, bugs, dyuan, dzheng, gklein, gsun, hannsj_uhl, honzhang, istein, jherrman, jsuchane, laine, lhuang, michal.skrivanek, mpoledni, mzhan, nsimsolo, pkrempa, rbalakri, snagar, tlavigne, ylavi |
Target Milestone: | rc | Keywords: | ZStream |
Target Release: | --- | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | libvirt-1.2.17-13.el7_2.1 | Doc Type: | Bug Fix |
Doc Text: |
When using Virtual Function I/O (VFIO) passthrough devices, the memory lock limit failed to be modified during a memory hot-plug operation. As a consequence, the guest virtual machine terminated unexpectedly. Now, the memory lock limit modification is performed before the memory hot-plug, and the described crash no longer occurs.
|
Story Points: | --- |
Clone Of: | 1273491 | Environment: | |
Last Closed: | 2015-12-08 10:41:27 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1273491 | ||
Bug Blocks: | 1201513 |
Description
Jan Kurik
2015-11-11 17:17:22 UTC
I can reproduce this issue on libvirt-1.2.17-13.el7.x86_64: 1. prepare a guest with vf and maxmemory: # virsh dumpxml rhel7.0-rhel <maxMemory slots='32' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>2048896</memory> <currentMemory unit='KiB'>2048896</currentMemory> ... <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </hostdev> 2. start guest: # virsh start rhel7.0-rhel Domain rhel7.0-rhel started 3. hot-plug memory device: # cat memdevice.xml <memory model='dimm'> <target> <size unit='G'>1</size> <node>0</node> </target> </memory> # virsh attach-device rhel7.0-rhel memdevice.xml error: Failed to attach device from memdevice.xml error: Unable to read from monitor: Connection reset by peer And verify this bug with libvirt-1.2.17-13.el7_2.1.x86_64: 1. prepare a guest with vf and maxmemory: # virsh dumpxml rhel7.0-rhel <maxMemory slots='32' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>2048896</memory> <currentMemory unit='KiB'>2048896</currentMemory> ... <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </hostdev> 2. start guest: # virsh start rhel7.0-rhel Domain rhel7.0-rhel started 3. hot-plug memory device: # cat memdevice.xml <memory model='dimm'> <target> <size unit='G'>1</size> <node>0</node> </target> </memory> # virsh attach-device rhel7.0-rhel memdevice.xml Device attached successfully # virsh attach-device rhel7.0-rhel memdevice.xml Device attached successfully 4. check limit of qemu process: # prlimit -p `pidof qemu-kvm` RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size unlimited unlimited blocks CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited blocks LOCKS max number of file locks held unlimited unlimited MEMLOCK max locked-in-memory address space 5320474624 5320474624 bytes <-- MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 NPROC max number of processes 31092 31092 RSS max resident set size unlimited unlimited pages RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 31092 31092 STACK max stack size 8388608 unlimited bytes # virsh dumpxml rhel7.0-rhel <domain type='kvm' id='2'> <name>rhel7.0-rhel</name> <uuid>67c7a123-5415-4136-af62-a2ee098ba6cd</uuid> <maxMemory slots='32' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>4147200</memory> <currentMemory unit='KiB'>2048896</currentMemory> Test hot-plug pci device + memory device: 1. prepare a guest with maxmemory: # virsh dumpxml rhel7.0-rhel <domain type='kvm'> <name>rhel7.0-rhel</name> <uuid>67c7a123-5415-4136-af62-a2ee098ba6cd</uuid> <maxMemory slots='32' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>2048896</memory> <currentMemory unit='KiB'>2048896</currentMemory> 2. start guest: # virsh start rhel7.0-rhel Domain rhel7.0-rhel started 3. check limit: # prlimit -p `pidof qemu-kvm` RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size unlimited unlimited blocks CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited blocks LOCKS max number of file locks held unlimited unlimited MEMLOCK max locked-in-memory address space 65536 65536 bytes MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 NPROC max number of processes 31092 31092 RSS max resident set size unlimited unlimited pages RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 31092 31092 STACK max stack size 8388608 unlimited bytes 4. hot-plug a vf to guest: # cat hostdev.xml <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </hostdev> # virsh attach-device rhel7.0-rhel hostdev.xml Device attached successfully 5. # prlimit -p `pidof qemu-kvm` RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size unlimited unlimited blocks CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited blocks LOCKS max number of file locks held unlimited unlimited MEMLOCK max locked-in-memory address space 3172990976 3172990976 bytes MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 NPROC max number of processes 31092 31092 RSS max resident set size unlimited unlimited pages RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 31092 31092 STACK max stack size 8388608 unlimited bytes 6. hot-plug memory device: # cat memdevice.xml <memory model='dimm'> <target> <size unit='G'>1</size> <node>0</node> </target> </memory> # virsh attach-device rhel7.0-rhel memdevice.xml Device attached successfully # prlimit -p `pidof qemu-kvm` RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size unlimited unlimited blocks CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited blocks LOCKS max number of file locks held unlimited unlimited MEMLOCK max locked-in-memory address space 4246732800 4246732800 bytes MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 NPROC max number of processes 31092 31092 RSS max resident set size unlimited unlimited pages RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 31092 31092 STACK max stack size 8388608 unlimited bytes Test with locked xml element: 1. prepare a guest with maxmemory + locked: # virsh dumpxml rhel7.0-rhel <maxMemory slots='32' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>2048896</memory> <currentMemory unit='KiB'>2048896</currentMemory> <memoryBacking> <locked/> </memoryBacking> 2. start guest # virsh start rhel7.0-rhel Domain rhel7.0-rhel started 3. check limit: # prlimit -p `pidof qemu-kvm` RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size unlimited unlimited blocks CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited blocks LOCKS max number of file locks held unlimited unlimited MEMLOCK max locked-in-memory address space 3172990976 3172990976 bytes MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 NPROC max number of processes 31092 31092 RSS max resident set size unlimited unlimited pages RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 31092 31092 STACK max stack size 8388608 unlimited bytes 4. hot-plug memory: # virsh attach-device rhel7.0-rhel memdevice.xml Device attached successfully # prlimit -p `pidof qemu-kvm` RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size unlimited unlimited blocks CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited blocks LOCKS max number of file locks held unlimited unlimited MEMLOCK max locked-in-memory address space 4246732800 4246732800 bytes MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 NPROC max number of processes 31092 31092 RSS max resident set size unlimited unlimited pages RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 31092 31092 STACK max stack size 8388608 unlimited bytes Test memory hot-unplug: 1. start a guest with vf + maxmemory: # virsh dumpxml rhel7.0-rhel ... <maxMemory slots='32' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>2048896</memory> <currentMemory unit='KiB'>2048896</currentMemory> ... <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </hostdev> # virsh start rhel7.0-rhel Domain rhel7.0-rhel started 2. check limit: # prlimit -p `pidof qemu-kvm` RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size unlimited unlimited blocks CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited blocks LOCKS max number of file locks held unlimited unlimited MEMLOCK max locked-in-memory address space 3172990976 3172990976 bytes MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 NPROC max number of processes 31092 31092 RSS max resident set size unlimited unlimited pages RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 31092 31092 STACK max stack size 8388608 unlimited bytes 3. hot-plug memory device: # cat memdevice.xml <memory model='dimm'> <target> <size unit='m'>256</size> <node>0</node> </target> </memory> # virsh attach-device rhel7.0-rhel memdevice.xml Device attached successfully # prlimit -p `pidof qemu-kvm` RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size unlimited unlimited blocks CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited blocks LOCKS max number of file locks held unlimited unlimited MEMLOCK max locked-in-memory address space 3307208704 3307208704 bytes MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 NPROC max number of processes 31092 31092 RSS max resident set size unlimited unlimited pages RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 31092 31092 STACK max stack size 8388608 unlimited bytes 4. hot-unplug: # virsh detach-device rhel7.0-rhel memdevice.xml Device detached successfully # prlimit -p `pidof qemu-kvm` RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size unlimited unlimited blocks CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited blocks LOCKS max number of file locks held unlimited unlimited MEMLOCK max locked-in-memory address space 3172990976 3172990976 bytes MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 NPROC max number of processes 31092 31092 RSS max resident set size unlimited unlimited pages RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 31092 31092 STACK max stack size 8388608 unlimited bytes And verify the audit log part with libvirt-1.2.17-13.el7_2.1.x86_64: 1. prepare a guest with maxmemory: # virsh dumpxml rhel7.0-rhel ... <maxMemory slots='32' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>2048896</memory> <currentMemory unit='KiB'>2048896</currentMemory> 2. start guest : # virsh start rhel7.0-rhel Domain rhel7.0-rhel started 3. hot-plug a device: # cat memdevice.xml <memory model='dimm'> <target> <size unit='m'>128</size> <node>0</node> </target> </memory> # virsh attach-device rhel7.0-rhel memdevice.xml Device attached successfully 4. check log # ausearch -m VIRT_RESOURCE -ts recent |grep mem type=VIRT_RESOURCE msg=audit(1448269938.342:7625): pid=23449 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm resrc=mem reason=start vm="rhel7.0-rhel" uuid=67c7a123-5415-4136-af62-a2ee098ba6cd old-mem=0 new-mem=2050048 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' type=VIRT_RESOURCE msg=audit(1448269968.002:7628): pid=23449 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm resrc=mem reason=update vm="rhel7.0-rhel" uuid=67c7a123-5415-4136-af62-a2ee098ba6cd old-mem=2050048 new-mem=2181120 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-2567.html |