RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1879163 - [virtio-win][balloon]balloon value change is very slow inside win2016-64 guest
Summary: [virtio-win][balloon]balloon value change is very slow inside win2016-64 guest
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.5
Hardware: x86_64
OS: Windows
medium
medium
Target Milestone: rc
: ---
Assignee: Yvugenfi@redhat.com
QA Contact: xiagao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-15 14:57 UTC by FuXiangChun
Modified: 2023-02-13 06:25 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-08 07:31:04 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description FuXiangChun 2020-09-15 14:57:04 UTC 
Description of problem:

Try to change balloon value from 2G to 32G. Win2016 64bit guest takes an hour to complete. 

Version-Release number of selected component (if applicable):

qemu-kvm: qemu-kvm-rhev-2.10.0-21.el7_5.10.x86_64
kernel: kernel-3.10.0-862.14.4.el7.x86_64
spice: spice-server-0.14.0-2.el7_5.5.x86_64
seabios: seabios-bin-1.11.0-2.el7.noarch
seavgabios: seavgabios-bin-1.11.0-2.el7.noarch
sgabios: sgabios-bin-0.20110622svn-4.el7.noarch
ipxe: ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
virtio-win: virtio-win-1.9.12-4.el7.iso

How reproducible:
always

1./usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox off  \
-machine pc  \
-nodefaults \
-device VGA,bus=pci.0,addr=0x2 \
-m 30720  \
-smp 32,maxcpus=32,cores=16,threads=1,sockets=2  \
-cpu 'EPYC',+kvm_pv_unhalt \
-chardev socket,path=/var/tmp/avocado_ca05cK/monitor-qmpmonitor1-20200915-045748-SRg4PyT8,nowait,id=qmp_id_qmpmonitor1,server  \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,path=/var/tmp/avocado_ca05cK/monitor-catch_monitor-20200915-045748-SRg4PyT8,nowait,id=qmp_id_catch_monitor,server  \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=id2IPJqR \
-chardev socket,path=/var/tmp/avocado_ca05cK/serial-serial0-20200915-045748-SRg4PyT8,nowait,id=chardev_serial0,server \
-device isa-serial,id=serial0,chardev=chardev_serial0  \
-chardev socket,id=seabioslog_id_20200915-045748-SRg4PyT8,path=/var/tmp/avocado_ca05cK/seabios-20200915-045748-SRg4PyT8,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20200915-045748-SRg4PyT8,iobase=0x402 \
-device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/win2016-64-virtio-scsi.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1 \
-device virtio-net-pci,mac=9a:e9:3a:e6:08:cc,id=idjzr5z3,netdev=idA5Wxlw,bus=pci.0,addr=0x5  \
-netdev tap,id=idA5Wxlw,vhost=on \
-drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso \
-device scsi-cd,id=cd1,drive=drive_cd1  \
-vnc :0  \
-rtc base=localtime,clock=host,driftfix=slew  \
-boot menu=off,strict=off,order=cdn,once=c \
-enable-kvm \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-monitor stdio \

2. (qemu) balloon 2048
It takes about 2 minutes to complete inside guest.

3. (qemu) balloon 32000

Actual results:
It takes about an hour to complete inside guest.

Expected results:
soon

Additional info:
Comment 2 xiagao 2020-09-16 06:27:24 UTC 
Didn't hit this issue on RHEL8.3.0 with the same virtio-win driver, so it seems not the balloon driver's issue.

pkgs:
qemu-kvm-5.1.0-2.module+el8.3.0+7652+b30e6901.x86_64
seabios-1.14.0-1.module+el8.3.0+7638+07cf13d2.x86_64
kernel-4.18.0-229.el8.x86_64
virtio-win-1.9.12-4.el7.iso/virtio-win-1.9.12-2.el8.iso(balloon driver version: 177)


But it can be reproduced on the qemu of comment0.
@yuhuang could you have a check if it's qemu's bug?
Comment 3 Yumei Huang 2020-09-16 07:34:21 UTC 
Hi Xiaoling, 

I tried win2012 guest with the same packages in comment 0, and it works well. I guess it's win2016 OS or driver problem.
Comment 4 Yvugenfi@redhat.com 2020-09-17 07:09:02 UTC 
Is this even a valid scenrario?

VM was given 30720MB of memory and we are trying to inflate Balloon to 32G. At somepoint, all OS free memory will be consumed and it will be heavily using swap (thus the slowness).
Comment 5 xiagao 2020-09-18 02:56:40 UTC 
(In reply to Yan Vugenfirer from comment #4)
> Is this even a valid scenrario?
> 
> VM was given 30720MB of memory and we are trying to inflate Balloon to 32G.
> At somepoint, all OS free memory will be consumed and it will be heavily
> using swap (thus the slowness).

Hi Yan,
Balloon memory to 30720MB, still has this issue.

Thanks,
Xiaoling
Comment 6 xiagao 2020-09-28 10:49:49 UTC 
Win2019,win2016 and win10-64 hit this issue, it needs about 20 mins to balloon memory from 2G to 30G.
Test win2016 with libvirt, it needs about 13 mins.

test pkgs:
qemu-kvm-rhev-2.10.0-21.el7_5.10.x86_64
kernel-3.10.0-862.14.4.el7.x86_64
virtio-win-1.9.12-4.el7.iso/virtio-win-prewhql-189
Comment 7 Yvugenfi@redhat.com 2020-12-07 13:14:16 UTC 
(In reply to xiagao from comment #5)
> (In reply to Yan Vugenfirer from comment #4)
> > Is this even a valid scenrario?
> > 
> > VM was given 30720MB of memory and we are trying to inflate Balloon to 32G.
> > At somepoint, all OS free memory will be consumed and it will be heavily
> > using swap (thus the slowness).
> 
> Hi Yan,
> Balloon memory to 30720MB, still has this issue.
> 
> Thanks,
> Xiaoling

That's the problem. We cannot take all the memory from the guest. At some point, the guest will become very slow due to swapping.
At least 0.5-1G should be left for the OS to function.
So if you use -m 32G, then inflate ballon only up to 31G
Comment 9 lijin 2020-12-08 01:26:43 UTC 
(In reply to Yan Vugenfirer from comment #7)
> (In reply to xiagao from comment #5)
> > (In reply to Yan Vugenfirer from comment #4)
> > > Is this even a valid scenrario?
> > > 
> > > VM was given 30720MB of memory and we are trying to inflate Balloon to 32G.
> > > At somepoint, all OS free memory will be consumed and it will be heavily
> > > using swap (thus the slowness).
> > 
> > Hi Yan,
> > Balloon memory to 30720MB, still has this issue.
> > 
> > Thanks,
> > Xiaoling
> 
> That's the problem. We cannot take all the memory from the guest. At some
> point, the guest will become very slow due to swapping.
> At least 0.5-1G should be left for the OS to function.
> So if you use -m 32G, then inflate ballon only up to 31G

I don't think so.
Chang balloon from 2G to 32G is not "take memory from guest", we actually bring memory to guest.
The operation is to change guest memory from 2G to 32G, it is definitely a commnon scenario for balloon.

The issue here is that on some qemu versions(rhel7 qemu reproduced while rhel8 qemu not) and on win10+ guests, the process is quite slow.
rhel7 qemu issue, windows OS issue or drvier issue?
Comment 12 xiagao 2021-05-20 03:56:47 UTC 
Not reproduce this issue in RHEL7.9.z with latest pkg.

pkg:
qemu-kvm-rhev.x86_64 10:2.12.0-48.el7_9.3 
kernel-3.10.0-1160.29.1.el7.x86_64
virtio-win-prewhql-199
virtio-win-1.9.12-4.el7.iso
Win2019 guest

It took about 2mins to balloon memory from 2G to 32G which looks good.
Comment 13 Yvugenfi@redhat.com 2021-05-20 12:07:18 UTC 
(In reply to xiagao from comment #12)
> Not reproduce this issue in RHEL7.9.z with latest pkg.
> 
> pkg:
> qemu-kvm-rhev.x86_64 10:2.12.0-48.el7_9.3 
> kernel-3.10.0-1160.29.1.el7.x86_64
> virtio-win-prewhql-199
> virtio-win-1.9.12-4.el7.iso
> Win2019 guest
> 
> It took about 2mins to balloon memory from 2G to 32G which looks good.

Did you try with Win2016 guest as well (as the original BZ description)?
Comment 14 xiagao 2021-05-21 07:11:46 UTC 
(In reply to Yvugenfi from comment #13)
> (In reply to xiagao from comment #12)
> > Not reproduce this issue in RHEL7.9.z with latest pkg.
> > 
> > pkg:
> > qemu-kvm-rhev.x86_64 10:2.12.0-48.el7_9.3 
> > kernel-3.10.0-1160.29.1.el7.x86_64
> > virtio-win-prewhql-199
> > virtio-win-1.9.12-4.el7.iso
> > Win2019 guest
> > 
> > It took about 2mins to balloon memory from 2G to 32G which looks good.
> 
> Did you try with Win2016 guest as well (as the original BZ description)?

Win10+ hit this issue from comment 6, although, I tried with Win2016 and didn't reproduce with virtio-win-1.9.12-4.el7.iso and virtio-win-prewhql-199.
Comment 15 RHEL Program Management 2021-06-08 07:31:04 UTC 
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Note You need to log in before you can comment on or make changes to this bug.