Try to reproduce this bug on qemu-kvm-0.12.1.2-2.178.el6.x86_64 and verify this bug on qemu-kvm-0.12.1.2-2.183.el6.x86_64, according to Comment 1, test unplug, however, run into some strange results, if i did wrong, please correct me:

Steps to reproduce:
----------------------------
1.in the host:
#while true;do ps --no-headers --format 'rss,vsz' `pidof qemu-kvm`; sleep 10; done

2.boot guest with:
#/usr/libexec/qemu-kvm -enable-kvm -M rhel6.2.0 -smp 4 -m 4G -name rhel6.1-64 -uuid 3f2ea5cd-3d29-48ff-aab2-23df1b6ae213 -drive file=RHEL-Server-6.1-64-virtio.qcow2,cache=none,if=none,rerror=stop,werror=stop,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,drive=drive-virtio-disk0,id=device-virtio-disk0 -netdev tap,script=/etc/qemu-ifup,id=netdev0 -device virtio-net-pci,netdev=netdev0,id=device-net0 -boot order=dc -usb -device usb-tablet,id=input0 -monitor stdio -vnc :10

3.hotplug a virtio-blk
(qemu) __com.redhat_drive_add file=test.img,cache=none,rerror=stop,werror=stop,id=drive-virtio-disk1,format=qcow2
(qemu) device_add virtio-blk-pci,drive=drive-virtio-disk1,id=device-virtio-disk1

4.hotunplug the disk
(qemu) device_del device-virtio-disk1


Actual results:
--------------------------
                                qemu-kvm-0.12.1.2-2.178        qemu-kvm-0.12.1.2-2.183
after step 2            548236     4784364                   543964          4784376
after step 3            548528     4786724                   544260          4786856
after step 4            548524     4784672                   544256          4784804
-----------------------------------------------------------------------------------------------------
step4 - step2                288              308                          292                   428


Additional info:
---------------------------
If repeat step 2-4, on qemu-kvm-0.12.1.2-2.183:

543964          4784376       <- initial status

544260          4786856       <- first plug
544256          4784804       <- first unplug

544384          4786564       <- second plug
550528          4786564       <- second unplug

550560          4786708       <- third plug
550576          4786708       <- third unplug

RSS keeps increasing but there is no regular pattern, it's weird why VSZ doesn't reduce in second and third time.

If this bug is refer to the situation when there is additional virtio-blk device in qemu-kvm command line then unplug it, following is the test:


Steps to reproduce:
----------------------------
1.in the host:
#while true;do ps --no-headers --format 'rss,vsz' `pidof qemu-kvm`; sleep 10; done

2.boot guest with:
#/usr/libexec/qemu-kvm -enable-kvm -M rhel6.2.0 -smp 4 -m 4G -name rhel6.1-64 -uuid 3f2ea5cd-3d29-48ff-aab2-23df1b6ae213 -drive file=RHEL-Server-6.1-64-virtio.qcow2,cache=none,if=none,rerror=stop,werror=stop,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,drive=drive-virtio-disk0,id=device-virtio-disk0 -netdev tap,script=/etc/qemu-ifup,id=netdev0 -device virtio-net-pci,netdev=netdev0,id=device-net0 -boot order=dc -usb -device usb-tablet,id=input0 -monitor stdio -vnc :10 -drive file=test.img,cache=none,if=none,rerror=stop,werror=stop,id=drive-virtio-disk1,format=qcow2 -device virtio-blk-pci,drive=drive-virtio-disk1,id=device-virtio-disk1

3.hotunplug the disk
(qemu) device_del device-virtio-disk1


Actual results:
--------------------------
                           0.12.1.2-2.178                0.12.1.2-2.183
after step 2       544384    4788372        543280         4787420
after step 3       544360    4786184        543212         4785232
-----------------------------------------------------------------------------------
step3 - step2            24            2188                 68                2188

Adjust "Actual results" format in Comment 5:

Actual results:
--------------------
                           0.12.1.2-2.178                  0.12.1.2-2.183
after step 2       548236     4784364         543964         4784376
after step 3       548528     4786724         544260         4786856
after step 4       548524     4784672         544256         4784804
------------------------------------------------------------------------------------
step4 - step2          288              308                  292                 428

It doesn't matter how the device is initialised -- on the command line or via hotplug.  Similarly, it doesn't matter how the device is removed - via hot-unplug or via quitting qemu.

What you are seeing in the repeated tests is that the memory allocated by '__com_redhat_drive_add' stays allocated even if you unplug the virtio-blk device.  If you remove the drive as well, you will see that usage go down.

Hi Amit,

As i know, the recommended way to unplug a blk is using "device_del" only, it will remove the device, as you said, there is some mem allocated with the drive, but after device_del, the $drive_id is invalid, so i can not use __com_redhat_drive_del. Using __com_redhat_drive_del first then device_del works too, but it's not recommended, right? Which way you want me use to test this bug? And which kind of memory i should focus, RSS or VSZ?

If the test is right, do you think the result can verify the bug? It's weird that results differ when i hotplug the device then do the test and when i add the the device on the command line then do the test, i use "device_del" only to unplug the device for all tests:


Actual results (hotplug then unplug):
-------------------------------------
                   0.12.1.2-2.178             0.12.1.2-2.183
                   RSS        VSZ             RSS            VSZ
after boot         548236     4784364         543964         4784376
after plug         548528     4786724         544260         4786856
after unplug       548524     4784672         544256         4784804
--------------------------------------------------------------------
plug - unplug           4        2052              4            2052



Actual results (on command line then unplug):
---------------------------------------------
                   0.12.1.2-2.178           0.12.1.2-2.183
                   RSS       VSZ            RSS            VSZ
after boot         544384    4788372        543280         4787420
after unplug       544360    4786184        543212         4785232
------------------------------------------------------------------
boot - unplug          24       2188            68            2188

The structure that is freed with the patch here is 208 bytes long.

It's entirely possible that the memory that gets allocated isn't given back immediately in anticipation of more mallocs, so you don't really see the benefit of the memfree.  I suppose if you run valgrind through qemu, you will notice some difference here.

However, it's surprising you see different results for the hotplug case and command line case.  If this is repeatable over a number of runs, it means there are some additional mem leaks, or just that the freeing happens at different times.

Overall, valgrind can confirm all the cases for us.

Steps to reproduce and verify:
------------------------------------
1.Disable as more services as possible in level 3 and init 3:
#ls /etc/rc.d/rc3.d/
K01libvirt-guests  K05atd        K16ksm             K50dnsmasq     K60nfs         K75netfs      K83nfslock         K85ebtables    K87restorecond  K89rdisc      K99cpuspeed
K01smartd          K10psacct     K25sshd            K50netconsole  K69rpcsvcgssd  K75ntpdate    K83rpcgssd         K85mdmonitor   K88auditd       K90network    K99lvm2-monitor
K02avahi-daemon    K10saslauthd  K30postfix         K50snmpd       K74acpid       K75quota_nld  K83rpcidmapd       K85messagebus  K88iscsi        K92ip6tables  K99sysstat
K02rhsmcertd       K15ksmtuned   K30spice-vdagentd  K50snmptrapd   K74haldaemon   K75udev-post  K84NetworkManager  K86cgred       K88rsyslog      K92iptables   S13rpcbind
K03rhnsd           K16abrtd      K46radvd           K60crond       K74ntpd        K80kdump      K84wpa_supplicant  K87irqbalance  K89iscsid       K95cgconfig   S99local

2.Make sure top suggests there are not other memory consuming processes:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+    COMMAND 
    1 root      20   0 19204 1488 1208 S  0.0  0.0   0:00.62   init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00   kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.19   migration/0  
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.00   ksoftirqd/0
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00   migration/0

3.Boot guest, do 5000 times hotplug/unplug, use "free -m" when:
  1) before start qemu
  2) before hotplug/unplug
  3) after hotplug/unplug
  4) after exit qemu
  each time before "free -m", do "echo 3 > /proc/sys/vm/drop_caches" first.


Actual results:
---------------------
reproduced on qemu-kvm-0.12.1.2-2.178.el6.x86_64:
                               
used    free    buffers/cached             
180     7281          0/6             <---before start qemu
811     6651          0/9             <---start qemu
1148    6314          0/9             <---after hotplug/unplug 5000 times
181     7280          0/6             <---after exit qemu

verified on qemu-kvm-0.12.1.2-2.183.el6.x86_64:

used    free    buffers/cached             
181     7280          0/6             <---before start qemu
741     6720          0/9             <---start qemu
898     6563          0/9             <---after hotplug/unplug 5000 times
182     7279          0/6             <---after exit qemu


Conclusion:
--------------
The patch release the memory when unplug, given the condition that two versions share the similar memory usage before qemu start and after exit, test suggests after hotplug/unplug 5000 times but not exit qemu yet, free memory with qemu 183 is much more than 178, based on this, the patch works well, verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2011-1531.html