Bug 1929144

recently hit a uncommon problem 'bsod' for testing guest-agent on win10-20H20_x86 guest, case is 42-Host_RHEL.m8.u4.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.Win10.x86_64.io-github-autotest-qemu.qemu_guest_agent.virtio_serial.check_os_info.q35, so I thought about this bz and I'm trying to collect dump file, but it's not easy to get, because I've been testing it 200 times, keep doing it. just update the situation here as tracker. feel free to correct me if I misunderstand something. 

complete log: http://fileshare.englab.nay.redhat.com/pub/logs/win10_20H2_32_guest-agent/test-results/42-Host_RHEL.m8.u4.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.Win10.x86_64.io-github-autotest-qemu.qemu_guest_agent.virtio_serial.check_os_info.q35/

packages version:
    kernel-4.18.0-278.el8.dt3.x86_64
    qemu-kvm-5.2.0-4.module+el8.4.0+9676+589043b9.x86_64
    virtio-win-prewhql-193
    seabios-bin-1.14.0-1.module+el8.4.0+8855+a9e237a9.noarch
    RHEL-8.4.0-20210128.d.2

(In reply to dehanmeng from comment #3)
> recently hit a uncommon problem 'bsod' for testing guest-agent on
> win10-20H20_x86 guest, case is
> 42-Host_RHEL.m8.u4.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.Win10.
> x86_64.io-github-autotest-qemu.qemu_guest_agent.virtio_serial.check_os_info.
> q35, so I thought about this bz and I'm trying to collect dump file, but
> it's not easy to get, because I've been testing it 200 times, keep doing it.
> just update the situation here as tracker. feel free to correct me if I
> misunderstand something. 

correct dump log link here: http://fileshare.englab.nay.redhat.com/pub/logs/win10_20H2_32_guest_agent/test-results/42-Host_RHEL.m8.u4.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.Win10.i386.io-github-autotest-qemu.qemu_guest_agent.virtio_serial.check_os_info.q35/

Hi Dehan,

Scratch build with changes:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=35146591
can this build be verified for this bug too?

Thanks in advance,
Basil

(In reply to Basil Salman from comment #10)
> Hi Dehan,
> 
> Scratch build with changes:
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=35146591
> can this build be verified for this bug too?
> 
> Thanks in advance,
> Basil

Hi Basil,
Okay, sure, Will use this new build to check and update result asap. thanks for your effort, Basil.

BR
Dehan

Hi Basil, qga command {"guest-get-disks"} of new build works well. 

reproduce with previous mingw-qemu-ga-win-101.2.0-1.el7ev
step to reproduce:
1. boot up win2019 guest with serial and qga.
2. connect with guest and execute qga command, run following command in terminal (host).
cat <(for i in {1..10000}; do echo '{"execute":"guest-get-osinfo"}'; sleep 2; echo '{"execute":"guest-get-devices"}'; sleep 2; done)  | nc -U /tmp/qga.sock > log
3.check the PID of QEMU-ga in cmd with following command (guest):
C:> handle.exe -s -p $qga_PID

Actual result:
handles raise up without limit, resource leak.
Expected result:
handles maintain a stable value.

Verify with this build qemu-ga-win-102.0.0-1.el8.noarch.rpm
step to verify:
as above

Actual result:
handles maintain a stable value.
Expected result:
as above.

(In reply to dehanmeng from comment #12) 
> reproduce with previous mingw-qemu-ga-win-101.2.0-1.el7ev
> step to reproduce:
> 1. boot up win2019 guest with serial and qga.
> 2. connect with guest and execute qga command, run following command in
> terminal (host).
> cat <(for i in {1..10000}; do echo '{"execute":"guest-get-osinfo"}'; sleep
> 2; echo '{"execute":"guest-get-devices"}'; sleep 2; done)  | nc -U
> /tmp/qga.sock > log
> 3.check the PID of QEMU-ga in cmd with following command (guest):
> C:> handle.exe -s -p $qga_PID
> 
> Actual result:
> handles raise up without limit, resource leak.
> Expected result:
> handles maintain a stable value.
> 
> Verify with this build qemu-ga-win-102.0.0-1.el8.noarch.rpm
> step to verify:
> as above
> 
> Actual result:
> handles maintain a stable value.
> Expected result:
> as above.

Hi Basil, qga command {"guest-get-osinfo"}, {"guest-get-devices"} of new build works well. didn't check out resource leak.

The test build given to the customer in BZ1920342 was tested by the customer and it fixed the handle leak.

Build that resolves this bug:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=35297151

reproduce with previous mingw-qemu-ga-win-101.2.0-1.el7ev
step to reproduce:
1. boot up win10-64(q35) guest with serial and qga.
2. Check the PID and start handles of qga in guest.
# win+R --> 'taskmgr' --> More details --> right click 'Name' and choose 'PID', 
# C:> handle.exe -s -p $PID
3. connect with guest and execute qga command, run following command in terminal (host).
cat <(for i in {1..10000}; do echo '{"execute":"guest-get-osinfo"}'; sleep 2; echo '{"execute":"guest-get-devices"}'; sleep 2; done)  | nc -U /tmp/qga.sock > log
4.check the end handles of QEMU-ga in cmd with following command after several minutes:
# C:> handle.exe -s -p $qga_PID

Actual result:
handles raise up without limit, actually it should be limited to less than 300, but now it cause resource leak.

Nthandle v4.22 - Handle Viewer
Copyright (C) 1997-2019 Mark Russinovich
Sysinternals - www.sysinternals.com
Handle type summary:
  ALPC Port                                   :5
  Desktop                                      :1
  Directory                                     :2
  ...
  ...
Total Handles: 437

Expected result:
handles maintain a stable value.

Verify with this build qemu-ga-win-102.0.0-1.el8.noarch.rpm on win10-64(q35)
step to verify:
as above

Actual result:
handles will raise about 5%~ and then maintain a stable value
Nthandle v4.22 - Handle Viewer
Copyright (C) 1997-2019 Mark Russinovich
Sysinternals - www.sysinternals.com
Handle type summary:
  ALPC Port                                   :5
  Desktop                                      :1
  Directory                                     :2
  ...
  ...
Total Handles: 177
Expected result:
as above.

Hi all, 
The newest qemu-ga-win came out and it has been passed from my side.  the whole test loop and new cases got passed as well.  No further errors and regression issues were found now. Thanks everyone for the time and effort.

Cheers
Dehan

(In reply to dehanmeng from comment #20)
> Hi all, 
> The newest qemu-ga-win came out and it has been passed from my side.  the
> whole test loop and new cases got passed as well.  No further errors and
> regression issues were found now. Thanks everyone for the time and effort.
> 
> Cheers
> Dehan

the qemu-ga-win version is mingw-qemu-ga-win-102.0.0-2.el8

Seems as we have two options to solve this problem , use the latest rhel 8.4.0 virtio win rpm or
update/repackage 8.3.0(z) virtio-win rpm with the latest binaries that were used for building 8.4.0 
virtio-win package. Please let us know which one looks better.

(In reply to Vadim Rozenfeld from comment #24)
> Seems as we have two options to solve this problem , use the latest rhel
> 8.4.0 virtio win rpm or
> update/repackage 8.3.0(z) virtio-win rpm with the latest binaries that were
> used for building 8.4.0 
> virtio-win package. Please let us know which one looks better.

using latest virtio-win rpm from 8.4 is ok from RHV point of view if that's the question for me.

*** Bug 1958825 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virtio-win bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:1959