Bug 1730166

QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks

Can we retest this with RHEL-AV 8.3.0 - trying to reduce our backlog of old issues.  Thanks.

I've tested on rhel 8.3 and I was able to reproduce the issue:

Package version:
libvirt-6.6.0-6.module+el8.3.0+8125+aefcf088
qemu-kvm-5.1.0-10.module+el8.3.0+8254+568ca30d

The memory leak is still present.

Test with qemu-img-5.1.0-13.module+el8.3.0+8382+afc3bbea.x86_64, this issue still present, memory leakage is up to 53.5 MB every 1000 cycles.

Version:
qemu-img-5.1.0-13.module+el8.3.0+8382+afc3bbea.x86_64
libvirt-6.0.0-28.module+el8.3.0+7827+5e65edd7.x86_64
kernel-4.18.0-240.el8.x86_64

Steps:
1. prepare a copy.xml with an data disk.
2. Prepare a nbd server:
# qemu-nbd /tmp/a -x '32r5340034u34ui054iu*^&*_)$(@($*&#@&$&@#$&#&#)@*$()@#*$()#($*#@()$*#$*#*@*#@()(@#*$()*@#($*(@#*$(*@#*$#@*$*@#$@#*@#()*$#*@*$#@**][\]{}|{}{}{}|{{{|{{|{}|{}|{}|${|#{${#}{}${%}{$%{}|{%}{#$}|{%{#$}|%{}|${%}|{#$|%{|#$%|{#$|%{#$|' -t

3. Create the qcow2 backing file
# qemu-img create -b 'nbd:localhost:10809:exportname=32r5340034u34ui054iu*^&*_)$(@($*&#@&$&@#$&#&#)@*$()@#*$()#($*#@()$*#$*#*@*#@()(@#*$()*@#($*(@#*$(*@#*$#@*$*@#$@#*@#()*$#*@*$#@**][\]{}|{}{}{}|{{{|{{|{}|{}|{}|${|#{${#}{}${%}{$%{}|{%}{#$}|{%{#$}|%{}|${%}|{#$|%{|#$%|{#$|%{#$|' /var/lib/libvirt/images/n.qcow2 -f qcow2

4. Attach/detach the qcow2 disk for 1000 times. Just run the scripts as:
# ./main.py -e 'virsh attach-disk copy /var/lib/libvirt/images/n.qcow2 vdb --subdriver qcow2; sleep 1; virsh detach-disk copy vdb' -p "`pidof libvirtd qemu-kvm virtlogd qemu-nbd`" -c 1000

5. check svg file, indicates memory leakage is up to 53.5 MB

Actual result shows above.

Expected result:
no memory leakage.

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Reproduce this issue, it still exists, memory leak is up to 100MB, see attachment qemu-kvm-55972.svg.

Version:
qemu-kvm-5.2.0-14.module+el8.4.0+10425+ad586fa5.x86_64
kernel-4.18.0-301.el8.x86_64
libvirt-7.0.0-10.module+el8.4.0+10417+37f6984d.x86_64

Steps:
1.# qemu-img create -f qcow2  -o preallocation=falloc /home/nbd.qcow2 10G
Formatting '/home/nbd.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=falloc compression_type=zlib size=10737418240 lazy_refcounts=off refcount_bits=16

2. # qemu-nbd -t --fork -f raw /home/nbd.qcow2
3. # qemu-img create -b 'nbd:localhost:10809' /var/lib/libvirt/images/n.qcow2
qemu-img: /var/lib/libvirt/images/n.qcow2: Backing file not supported for file format 'raw'

4. # qemu-img info /var/lib/libvirt/images/n.qcow2
image: /var/lib/libvirt/images/n.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 196 KiB
cluster_size: 65536
backing file: nbd:localhost:10809
backing file format: qcow2
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false
5. Check qemu-kvm-55972.svg, every 1000 attach/detach, memory leaks around 100MB.


Since it can be reproduced b libvirt, I'd like to reopen this bug.

Created attachment 1767238 [details]
qemu-kvm-55972.svg

Eric - reassigning this to you since it is an NBD related bug. Alice has investigated using valgrand, but wasn't able to uncover the leak.

Whether it helps or not, who knows, but StefanoG recently fixed two rbd/block leaks in qemu commits b084b420d9d6347dede328fbcf18c8e4c695f7e8 and c1c1f6cf511496b985cb9a1c536d59c9be7b9317

Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Created attachment 1831096 [details]
qemu-kvm.svg

Hi Hanhan,

I tried in latest rhel9, there seems no ram usage increased. Could you please help to check again in your environment? Thanks



Tested env:
qemu-kvm-6.1.0-4.el9
kernel-5.14.0-0.rc7.54.el9.x86_64


Steps:
1. Prepare snapshot based on image over nbd
# qemu-img create -f qcow2  -o preallocation=falloc nbd.qcow2 10G
# qemu-nbd -t --fork -f raw nbd.qcow2 &
# qemu-img create -f qcow2 -F qcow2 -b 'nbd:localhost:10809' sn.qcow2
# qemu-img info sn.qcow2 
image: sn.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 196 KiB
cluster_size: 65536
backing file: nbd:localhost:10809
backing file format: qcow2
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

2. Start a guest
# virsh define copy.xml
# virsh start copy
Note:
# cat copy.xml 
<domain type="kvm" id="2">
<name>copy</name>
<uuid>c3f1cdcb-cc26-462d-9e4a-fdb6f40402ca</uuid>
<memory unit="KiB">1048576</memory>
<currentMemory unit="KiB">1048576</currentMemory>
<vcpu placement="static">1</vcpu>
<iothreads>3</iothreads>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch="x86_64" machine="pc">hvm</type>
<boot dev="hd"/>
</os>
<features>
<acpi/>
<apic/>
<vmport state="off"/>
</features>
<clock offset="utc">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type="file" device="disk" model="virtio">
<driver name="qemu" type="qcow2" cache="none" io="native" copy_on_read="on" discard="ignore"/>
<source file="/home/timao/test/stale/RHEL-9.0-x86_64-latest.qcow2"/>
<backingStore/>
<target dev="vda" bus="virtio"/>
<alias name="virtio-disk0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x0c" function="0x0"/>
</disk>
<controller type="usb" index="0" model="ich9-ehci1">
<alias name="usb"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x7"/>
</controller>
<controller type="usb" index="0" model="ich9-uhci1">
<alias name="usb"/>
<master startport="0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0" multifunction="on"/>
</controller>
<controller type="usb" index="0" model="ich9-uhci2">
<alias name="usb"/>
<master startport="2"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x1"/>
</controller>
<controller type="usb" index="0" model="ich9-uhci3">
<alias name="usb"/>
<master startport="4"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x2"/>
</controller>
<controller type="pci" index="0" model="pci-root">
<alias name="pci.0"/>
</controller>
<controller type="virtio-serial" index="0">
<alias name="virtio-serial0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
</controller>
<controller type="ide" index="0" model="ich6">
<alias name="ide"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
</controller>
<controller type="scsi" index="0" model="virtio-scsi">
<alias name="scsi0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
</controller>
<controller type="sata" index="0">
<alias name="sata0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x09" function="0x0"/>
</controller>
<controller type="sata" index="1">
<alias name="sata1"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x0a" function="0x0"/>
</controller>
<controller type="sata" index="2">
<alias name="sata2"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x0b" function="0x0"/>
</controller>
<interface type="direct">
<mac address="52:54:00:d1:0d:97"/>
<source dev="switch" mode="bridge"/>
<backenddomain name="netvm"/>
<target dev="macvtap0"/>
<model type="rtl8139"/>
<alias name="net0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
</interface>
<serial type="pty">
<source path="/dev/pts/0"/>
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
<alias name="serial0"/>
</serial>
<console type="pty" tty="/dev/pts/0">
<source path="/dev/pts/0"/>
<target type="serial" port="0"/>
<alias name="serial0"/>
</console>
<channel type="unix">
<source mode="bind" path="/var/lib/libvirt/qemu/channel/target/domain-2-copy/org.qemu.guest_agent.0"/>
<target type="virtio" name="org.qemu.guest_agent.0" state="connected"/>
<alias name="channel0"/>
<address type="virtio-serial" controller="0" bus="0" port="2"/>
</channel>
<input type="mouse" bus="ps2">
<alias name="input0"/>
</input>
<input type="keyboard" bus="ps2">
<alias name="input1"/>
</input>
<graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0">
<listen type="address" address="0.0.0.0"/>
<image compression="off"/>
</graphics>
<sound model="ich6">
<alias name="sound0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
</sound>
<video>
<model type="vga"/>
<alias name="video0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
</video>
<hub type="usb">
<alias name="hub0"/>
<address type="usb" bus="0" port="3"/>
</hub>
<memballoon model="virtio">
<alias name="balloon0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
</memballoon>
</devices>
</domain>

3. Execute the script to get the mem leak
# pip3 install psutil matplotlib
# ./main.py -e 'virsh attach-disk copy /home/timao/test/stale/sn.qcow2 vdb --subdriver qcow2; sleep 1; virsh detach-disk copy vdb' -p "`pidof libvirtd qemu-kvm virtlogd qemu-nbd`" -c 1000
Disk attached successfully

Disk detached successfully

Disk attached successfully

.....
Disk attached successfully

Disk detached successfully


Results:
Check the attachment for the mem leak info.

Created attachment 1832884 [details]
The result on RHEL8.5-AV

Tested on libvirt-7.6.0-5.module+el8.5.0+12933+58cb48a1.x86_64 qemu-kvm-6.0.0-32.module+el8.5.0+12949+ac589a5c.x86_64

From the graph of the ram usage of qemu-kvm process, it looks like no memory leak.

Thanks Han's comment. Agree to close this bug then. Thanks.