Bug 1975640

Summary: Memory sudden change when attaching/detaching disk image repeatedly
Product: Red Hat Enterprise Linux 9 Reporter: zixchen
Component: qemu-kvmAssignee: Virtualization Maintenance <virt-maint>
qemu-kvm sub component: Storage QA Contact: Tingting Mao <timao>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: low    
Priority: low CC: coli, jinzhao, juzhang, kkiwi, sgarzare, timao, virt-maint
Version: 9.0Keywords: Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-29 06:28:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
qemu-kvm-ceph-qemu6.0.svg
none
qemu-kvm-local-image none

Description zixchen 2021-06-24 06:30:31 UTC 
Description of problem:
This bug is found in Bug https://bugzilla.redhat.com/show_bug.cgi?id=1731078 
when attaching/detaching rbd disk image for 1000 times, memory sudden change is found, reproduce with local disk image, it has memory sudden change as well. 

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
Copy from Bug 1731078 Comment #21:
> I did more tests on memory leak of rbd attach/detach, the memory leak issue
> has been improved a lot, from the graph(the attachment
> qemu-kvm-ceph-qemu6.0.svg), repeatedly attach/detach rbd disk doesn't has
> clear memory leak, but a sudden change of 10 MB memory is observed, this
> could be another issue. Besides, I test 1000 attaching/detaching with local
> disk image, the graph is very steady but also a sudden change is
> observed,please check attachment qemu-kvm-local-image.svg Therefore, I am ok
> to close this bug, and we could track the sudden change issue in another
> bug. Thanks.
> 
> Version:
> qemu-kvm-6.0.0-19.module+el8.5.0+11385+6e7d542e.x86_64
> kernel-4.18.0-314.el8.x86_64
> libvirt-7.4.0-1.module+el8.5.0+11218+83343022.x86_64
> 
> Steps:
> 1. Prepare local image xml:
> <disk type='file' device='disk'>
>   <driver name='qemu' type='raw' cache='none'/>
>   <source file='/home/data.raw'/>
>   <target dev='vdb' bus='scsi'/>
> </disk>
> 2. Attach/detach 1000 times:
> # ./main.py -e 'virsh attach-device rhel85 file.xml;virsh detach-device
> rhel85 file.xml;sleep 2' -p "`pidof qemu-kvm`" -c 1000 -i 0.5
> 
> Actual result:
> Overall, the memory graph is steady, but a sudden change is observed.
> 
> Expected result:
> should not has the sudden change based on the actual result.


Additional info:
Comment 1 zixchen 2021-06-24 06:35:00 UTC 
Created attachment 1793752 [details]
qemu-kvm-ceph-qemu6.0.svg
Comment 2 zixchen 2021-06-24 06:36:05 UTC 
Created attachment 1793753 [details]
qemu-kvm-local-image
Comment 5 John Ferlan 2021-09-08 21:38:47 UTC 
Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.
Comment 7 Tingting Mao 2022-07-29 06:25:03 UTC 
Tried in latest rhel9.1, there is no the issue.

Tested with:
qemu-kvm-7.0.0-9.el9
kernel-5.14.0-130.kpq0.el9.x86_64


Steps:
1. Boot up a guest with below xml
<domain type='kvm' id='3'>
  <name>q35</name>
  <uuid>b6bc9106-8342-4092-8eae-bfef61479c81</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel8.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G3</model>
    <vendor>AMD</vendor>
    <model fallback='forbid'>Opteron_G3</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='disable' name='mmxext'/>
    <feature policy='disable' name='fxsr_opt'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='disable' name='3dnowext'/>
    <feature policy='disable' name='3dnow'/>
    <feature policy='disable' name='cmp_legacy'/>
    <feature policy='disable' name='cr8legacy'/>
    <feature policy='require' name='3dnowprefetch'/>
    <feature policy='disable' name='osvw'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='disable' name='rdtscp'/>
    <feature policy='disable' name='sse4a'/>
    <feature policy='disable' name='misalignsse'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native' copy_on_read='on' discard='ignore'/>
      <source file='/home/timao/leak/rhel870-64-virtio-scsi.qcow2'/>
      <backingStore/>
      <target dev='sda' bus='sata'/>
      <alias name='sata1-0-0'/>
      <address type='drive' controller='1' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='sata' index='1'>
      <alias name='sata1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x01' function='0x0'/>
    </controller>
    <serial type='pty'>
      <source path='/dev/pts/4'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/4'>
      <source path='/dev/pts/4'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c531,c764</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c531,c764</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

2. Run test
# ./main.py -e 'virsh attach-device q35 rbd.xml;virsh detach-device q35 rbd.xml;sleep 2' -p "`pidof qemu-kvm`" -c 1000 -i 0.5
Or
# ./main.py -e 'virsh attach-device q35 local_image.xml;virsh detach-device q35 local_image.xml;sleep 2' -p "`pidof qemu-kvm`" -c 1000 -i 0.5


Note:
# cat rbd.xml 
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <source protocol='rbd' name='rbd/image'>
        <host name='10.73.114.12' port='6789'/>
      </source>
      <target dev='vdc' bus='virtio'/>
    </disk>

# cat local_image.xml 
<disk type='file' device='disk'>
  <driver name='qemu' type='raw' cache='none'/>
  <source file='/home/data.raw'/>
  <target dev='vdb' bus='scsi'/>
</disk>

The main.py script is from comment3.


Results:
There is no memory sudden change, and check below attachments for details.