Bug 726015
| Summary: | Fix memleak on exit in virtio-blk | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Amit Shah <amit.shah> |
| Component: | qemu-kvm | Assignee: | Amit Shah <amit.shah> |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.2 | CC: | juzhang, mkenneth, qzhang, shu, tburke, virt-maint |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-0.12.1.2-2.179.el6 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2011-12-06 15:54:49 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Amit Shah
2011-07-27 10:28:13 UTC
Try to reproduce this bug on qemu-kvm-0.12.1.2-2.178.el6.x86_64 and verify this bug on qemu-kvm-0.12.1.2-2.183.el6.x86_64, according to Comment 1, test unplug, however, run into some strange results, if i did wrong, please correct me: Steps to reproduce: ---------------------------- 1.in the host: #while true;do ps --no-headers --format 'rss,vsz' `pidof qemu-kvm`; sleep 10; done 2.boot guest with: #/usr/libexec/qemu-kvm -enable-kvm -M rhel6.2.0 -smp 4 -m 4G -name rhel6.1-64 -uuid 3f2ea5cd-3d29-48ff-aab2-23df1b6ae213 -drive file=RHEL-Server-6.1-64-virtio.qcow2,cache=none,if=none,rerror=stop,werror=stop,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,drive=drive-virtio-disk0,id=device-virtio-disk0 -netdev tap,script=/etc/qemu-ifup,id=netdev0 -device virtio-net-pci,netdev=netdev0,id=device-net0 -boot order=dc -usb -device usb-tablet,id=input0 -monitor stdio -vnc :10 3.hotplug a virtio-blk (qemu) __com.redhat_drive_add file=test.img,cache=none,rerror=stop,werror=stop,id=drive-virtio-disk1,format=qcow2 (qemu) device_add virtio-blk-pci,drive=drive-virtio-disk1,id=device-virtio-disk1 4.hotunplug the disk (qemu) device_del device-virtio-disk1 Actual results: -------------------------- qemu-kvm-0.12.1.2-2.178 qemu-kvm-0.12.1.2-2.183 after step 2 548236 4784364 543964 4784376 after step 3 548528 4786724 544260 4786856 after step 4 548524 4784672 544256 4784804 ----------------------------------------------------------------------------------------------------- step4 - step2 288 308 292 428 Additional info: --------------------------- If repeat step 2-4, on qemu-kvm-0.12.1.2-2.183: 543964 4784376 <- initial status 544260 4786856 <- first plug 544256 4784804 <- first unplug 544384 4786564 <- second plug 550528 4786564 <- second unplug 550560 4786708 <- third plug 550576 4786708 <- third unplug RSS keeps increasing but there is no regular pattern, it's weird why VSZ doesn't reduce in second and third time. If this bug is refer to the situation when there is additional virtio-blk device in qemu-kvm command line then unplug it, following is the test:
Steps to reproduce:
----------------------------
1.in the host:
#while true;do ps --no-headers --format 'rss,vsz' `pidof qemu-kvm`; sleep 10; done
2.boot guest with:
#/usr/libexec/qemu-kvm -enable-kvm -M rhel6.2.0 -smp 4 -m 4G -name rhel6.1-64 -uuid 3f2ea5cd-3d29-48ff-aab2-23df1b6ae213 -drive file=RHEL-Server-6.1-64-virtio.qcow2,cache=none,if=none,rerror=stop,werror=stop,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,drive=drive-virtio-disk0,id=device-virtio-disk0 -netdev tap,script=/etc/qemu-ifup,id=netdev0 -device virtio-net-pci,netdev=netdev0,id=device-net0 -boot order=dc -usb -device usb-tablet,id=input0 -monitor stdio -vnc :10 -drive file=test.img,cache=none,if=none,rerror=stop,werror=stop,id=drive-virtio-disk1,format=qcow2 -device virtio-blk-pci,drive=drive-virtio-disk1,id=device-virtio-disk1
3.hotunplug the disk
(qemu) device_del device-virtio-disk1
Actual results:
--------------------------
0.12.1.2-2.178 0.12.1.2-2.183
after step 2 544384 4788372 543280 4787420
after step 3 544360 4786184 543212 4785232
-----------------------------------------------------------------------------------
step3 - step2 24 2188 68 2188
Adjust "Actual results" format in Comment 5: Actual results: -------------------- 0.12.1.2-2.178 0.12.1.2-2.183 after step 2 548236 4784364 543964 4784376 after step 3 548528 4786724 544260 4786856 after step 4 548524 4784672 544256 4784804 ------------------------------------------------------------------------------------ step4 - step2 288 308 292 428 It doesn't matter how the device is initialised -- on the command line or via hotplug. Similarly, it doesn't matter how the device is removed - via hot-unplug or via quitting qemu. What you are seeing in the repeated tests is that the memory allocated by '__com_redhat_drive_add' stays allocated even if you unplug the virtio-blk device. If you remove the drive as well, you will see that usage go down. Hi Amit,
As i know, the recommended way to unplug a blk is using "device_del" only, it will remove the device, as you said, there is some mem allocated with the drive, but after device_del, the $drive_id is invalid, so i can not use __com_redhat_drive_del. Using __com_redhat_drive_del first then device_del works too, but it's not recommended, right? Which way you want me use to test this bug? And which kind of memory i should focus, RSS or VSZ?
If the test is right, do you think the result can verify the bug? It's weird that results differ when i hotplug the device then do the test and when i add the the device on the command line then do the test, i use "device_del" only to unplug the device for all tests:
Actual results (hotplug then unplug):
-------------------------------------
0.12.1.2-2.178 0.12.1.2-2.183
RSS VSZ RSS VSZ
after boot 548236 4784364 543964 4784376
after plug 548528 4786724 544260 4786856
after unplug 548524 4784672 544256 4784804
--------------------------------------------------------------------
plug - unplug 4 2052 4 2052
Actual results (on command line then unplug):
---------------------------------------------
0.12.1.2-2.178 0.12.1.2-2.183
RSS VSZ RSS VSZ
after boot 544384 4788372 543280 4787420
after unplug 544360 4786184 543212 4785232
------------------------------------------------------------------
boot - unplug 24 2188 68 2188
The structure that is freed with the patch here is 208 bytes long. It's entirely possible that the memory that gets allocated isn't given back immediately in anticipation of more mallocs, so you don't really see the benefit of the memfree. I suppose if you run valgrind through qemu, you will notice some difference here. However, it's surprising you see different results for the hotplug case and command line case. If this is repeatable over a number of runs, it means there are some additional mem leaks, or just that the freeing happens at different times. Overall, valgrind can confirm all the cases for us. Steps to reproduce and verify:
------------------------------------
1.Disable as more services as possible in level 3 and init 3:
#ls /etc/rc.d/rc3.d/
K01libvirt-guests K05atd K16ksm K50dnsmasq K60nfs K75netfs K83nfslock K85ebtables K87restorecond K89rdisc K99cpuspeed
K01smartd K10psacct K25sshd K50netconsole K69rpcsvcgssd K75ntpdate K83rpcgssd K85mdmonitor K88auditd K90network K99lvm2-monitor
K02avahi-daemon K10saslauthd K30postfix K50snmpd K74acpid K75quota_nld K83rpcidmapd K85messagebus K88iscsi K92ip6tables K99sysstat
K02rhsmcertd K15ksmtuned K30spice-vdagentd K50snmptrapd K74haldaemon K75udev-post K84NetworkManager K86cgred K88rsyslog K92iptables S13rpcbind
K03rhnsd K16abrtd K46radvd K60crond K74ntpd K80kdump K84wpa_supplicant K87irqbalance K89iscsid K95cgconfig S99local
2.Make sure top suggests there are not other memory consuming processes:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 19204 1488 1208 S 0.0 0.0 0:00.62 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.19 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
3.Boot guest, do 5000 times hotplug/unplug, use "free -m" when:
1) before start qemu
2) before hotplug/unplug
3) after hotplug/unplug
4) after exit qemu
each time before "free -m", do "echo 3 > /proc/sys/vm/drop_caches" first.
Actual results:
---------------------
reproduced on qemu-kvm-0.12.1.2-2.178.el6.x86_64:
used free buffers/cached
180 7281 0/6 <---before start qemu
811 6651 0/9 <---start qemu
1148 6314 0/9 <---after hotplug/unplug 5000 times
181 7280 0/6 <---after exit qemu
verified on qemu-kvm-0.12.1.2-2.183.el6.x86_64:
used free buffers/cached
181 7280 0/6 <---before start qemu
741 6720 0/9 <---start qemu
898 6563 0/9 <---after hotplug/unplug 5000 times
182 7279 0/6 <---after exit qemu
Conclusion:
--------------
The patch release the memory when unplug, given the condition that two versions share the similar memory usage before qemu start and after exit, test suggests after hotplug/unplug 5000 times but not exit qemu yet, free memory with qemu 183 is much more than 178, based on this, the patch works well, verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2011-1531.html |