Looks like we need to backport the commit into RHEL 8.4.0 and then since RHV is needs a RHEL 8.3.0 related fix as well for the guest side we'll have to make the same change and get a z-stream bz to manage that.

It's really too bad RHV cannot somehow use the RHEL-AV bits for the guest side as that would seemingly solve the issue (similarly noted in bug 1859494).

*** Bug 1879449 has been marked as a duplicate of this bug. ***

(In reply to Tomáš Golembiovský from comment #0)
> Guest agent crashes after handling guest-shutdown command. It dies with
> error:
> 
> ERROR:qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
> 
> This was introduced by commit 781f2b3d1e in version 4.0.0 and fixed by
> commit 844bd70b in version 5.1.0. The fix has to be backported to guest
> agent in base rhel.
> 
> This is especially grave because there seems to be another problem in
> systemd where the crashed service during shutdown is stuck and systemd
> cannot decide whether to start or stop the service. This prevents the
> shutdown of VM completely.

Hi Tomáš
According to the package versions you mentioned,I tried to reproduce this bug with several versions of qemu-ga-4.0.0-x and RHEL8.1.0 guest and qemu-ga-4.2.0-x with RHEL8.3.0 , didn't hit this issue,so I want to confirm two things with you:
1. whether are these qga versions I test right? 
2. `ERROR:qga/main.c:532:send_response: assertion failed: (rsp && s->channel)` this error info come from inside guest when the shutdown operation is completed?

(In reply to dehanmeng from comment #4)
> Hi Tomáš
> According to the package versions you mentioned,I tried to reproduce this
> bug with several versions of qemu-ga-4.0.0-x and RHEL8.1.0 guest and

Is that from AV? Wasn't it 0.9x up to RHEL 8.2?

> qemu-ga-4.2.0-x with RHEL8.3.0 ,

I am able to reproduce it with: qemu-guest-agent-4.2.0-29.module+el8.3.0+7212+401047e6.x86_64
from RHEL-8.3.0-Beta-1.

> didn't hit this issue,so I want to confirm
> two things with you:
> 1. whether are these qga versions I test right? 

See above.

> 2. `ERROR:qga/main.c:532:send_response: assertion failed: (rsp &&
> s->channel)` this error info come from inside guest when the shutdown
> operation is completed?

Yes it comes from the guest. Best to enable persistent journald logs so you can get back to it after next boot. It should contain something like this:

Oct 13 11:12:48 localhost.localdomain systemd[1]: Started QEMU Guest Agent.
Oct 13 11:13:32 localhost.localdomain qemu-ga[719]: info: guest-shutdown called, mode: powerdown
Oct 13 11:13:33 localhost.localdomain qemu-ga[719]: **
Oct 13 11:13:33 localhost.localdomain qemu-ga[719]: ERROR:qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
Oct 13 11:13:33 localhost.localdomain systemd[1]: Stopping QEMU Guest Agent...
Oct 13 11:13:33 localhost.localdomain systemd[1]: qemu-guest-agent.service: Main process exited, code=dumped, status=6/ABRT
Oct 13 11:13:33 localhost.localdomain systemd[1]: qemu-guest-agent.service: Failed with result 'core-dump'.
Oct 13 11:13:33 localhost.localdomain systemd[1]: Stopped QEMU Guest Agent.

(In reply to Tomáš Golembiovský from comment #5)
> (In reply to dehanmeng from comment #4)
> > Hi Tomáš
> > According to the package versions you mentioned,I tried to reproduce this
> > bug with several versions of qemu-ga-4.0.0-x and RHEL8.1.0 guest and
> 
> Is that from AV? Wasn't it 0.9x up to RHEL 8.2?
> 
> > qemu-ga-4.2.0-x with RHEL8.3.0 ,
> 
> I am able to reproduce it with:
> qemu-guest-agent-4.2.0-29.module+el8.3.0+7212+401047e6.x86_64
> from RHEL-8.3.0-Beta-1.
> 
> > didn't hit this issue,so I want to confirm
> > two things with you:
> > 1. whether are these qga versions I test right? 
> 
> See above.
> 
> > 2. `ERROR:qga/main.c:532:send_response: assertion failed: (rsp &&
> > s->channel)` this error info come from inside guest when the shutdown
> > operation is completed?
> 
> Yes it comes from the guest. Best to enable persistent journald logs so you
> can get back to it after next boot. It should contain something like this:
> 
> Oct 13 11:12:48 localhost.localdomain systemd[1]: Started QEMU Guest Agent.
> Oct 13 11:13:32 localhost.localdomain qemu-ga[719]: info: guest-shutdown
> called, mode: powerdown
> Oct 13 11:13:33 localhost.localdomain qemu-ga[719]: **
> Oct 13 11:13:33 localhost.localdomain qemu-ga[719]:
> ERROR:qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
> Oct 13 11:13:33 localhost.localdomain systemd[1]: Stopping QEMU Guest
> Agent...
> Oct 13 11:13:33 localhost.localdomain systemd[1]: qemu-guest-agent.service:
> Main process exited, code=dumped, status=6/ABRT
> Oct 13 11:13:33 localhost.localdomain systemd[1]: qemu-guest-agent.service:
> Failed with result 'core-dump'.
> Oct 13 11:13:33 localhost.localdomain systemd[1]: Stopped QEMU Guest Agent.

thanks for your explanation so that I reproduced it with qemu-guest-agent-4.2.0-29.module+el8.3.0+7212+401047e6.x86_64,and I compared with 5.1.0, there were no error info at all.

QA_ack please?

qemu-kvm is for RHEL-8.4 is blocked due to bz#1898700

Changing to ITM=5 assuming that the build that is blocked from comment 11 will be unblocked soon.  If QE would prefer a later date, that's fine too.

Reproduce with version qemu-kvm-4.2.0-29.module+el8.3.0+7212+401047e6.x86_64

Steps to Reproduce：
1.Boot rhel830 guest successfully.
2.Configure log persistence in guest:
[root@ vm-73-139~]# mkdir /var/log/journal
[root@ vm-73-139~]# chgrp systemd-journal /var/log/journal
[root@ vm-73-139~]# chmod g+s /var/log/journal
[root@ vm-73-139~]# systemctl restart systemd-journald
3.Send following command in above socket connection in the host side to shutdown the guest:
[root@dell-per440-06 ~]# nc -U /tmp/qga.sock
{"execute":"guest-ping"}
{"return": {}}
{ "execute":"guest-shutdown","arguments":{"mode":"powerdown"}}
4.Check qga.service status after guest reboot.
[root@ vm-73-139~]# journalctl -e | grep -6 -i "ERROR"

Actual result:
Nov 22 21:12:48 localhost.localdomain systemd[1]: Started QEMU Guest Agent.
Nov 22 21:13:32 localhost.localdomain qemu-ga[719]: info: guest-shutdown called, mode: powerdown
Nov 22 21:13:33 localhost.localdomain qemu-ga[719]: **
Nov 22 21:13:33 localhost.localdomain qemu-ga[719]: ERROR:qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
Nov 22 21:13:33 localhost.localdomain systemd[1]: Stopping QEMU Guest Agent...
Nov 22 21:13:33 localhost.localdomain systemd[1]: qemu-guest-agent.service: Main process exited, code=dumped, status=6/ABRT
Nov 22 21:13:33 localhost.localdomain systemd[1]: qemu-guest-agent.service: Failed with result 'core-dump'.
Nov 22 21:13:33 localhost.localdomain systemd[1]: Stopped QEMU Guest Agent.

Expected result:
There is no error that qga.service abort after shutdown.

Verified with version qemu-kvm-4.2.0-36.module+el8.4.0+8807+0c3dc3b0

Steps to Reproduce like above,but boot up rhel840 guest instead.

Actual result:
There is no error that qga.service abort after shutdown.

Expected result
There is no error that qga.service abort after shutdown.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1762