Hi José,

For this bug I have some questions:

1) What's the virtio-win version for this bug? Which virtio-win devices were attached to customer's vm? and what's the driver number for each device?

2) What's the windows VM version? and which bios mode they used? OVMF or seabios? 

3) If possible, could we get the customer vm's qemu-commands or libvirt xml file? or may be we can ask them provide a sosreport for the computer node which running the vm (cannot shutdown hit).

4) When they shutdown the VM, what's running in the VM, if have some storage writing or reading in progress?

5) I noticed you added windows update information in the bug description, but I didn't see this information in customer portal case, could you help to confirm more clear for here's reproduce steps? Maybe combine 3) questions, I just want to know what's happen before they choose shutdown?

6) And I also noticed info that "VMs do Not stop after trigger shutdown, RHV Manager shows 'RebootInprogress' Status for 20-30 mins", I'm not sure if these two actions occurred at the same time, if yes, then it seems our customer triggered "shutdown with windows update", we all know that some windows updates installation need reboot, during reboot progress, system complete the updates installation, this progress may also cost much time, as I known 20-30 mins is normally sometimes, and after the reboot finished, then go shutdown.

I'm not sure if the customer is exactly experiencing this scenario, so could you please help to confirm it?  


Thanks a lot~
Peixiu

Hi José,

Thanks for your provided information, I try do some reproduce tests today, I can reproduce the cannot shutdown scenario, but I noticed that there is "-no-shutdown" command in the qemu-kvm command line, that means the vm was not allowed to shutdown as my understand.

And I tried to delete this command "-no-shutdown" from the qemu-kvm command line, after installed windows update, the vm can be shutdown smoothly.

Could you help to confirm if our customer's vms all enabled "-no-shutdown"? If yes, that may be the key point. Thanks~ 

My test env:

Host: RHEL8.6.0
Guest OS: Windows 2016

kernel-4.18.0-372.9.1.el8.x86_64 (It's closer the customer used 4.18.0-372.16, I didn't find that version, so use this one).
qemu-kvm-6.2.0-11.module+el8.6.0+14707+5aa4b42d.x86_64
seabios-bin-1.15.0-2.module+el8.6.0+14757+c25ee005.noarch
virtio-win-1.9.18-4.el8.iso

qemu-command line:
/usr/libexec/qemu-kvm \
	-name guest=wintkbisvetl,debug-threads=on \
	-machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off \
	-accel kvm \
	-cpu Skylake-Server-noTSX-IBRS,ssbd=on,md-clear=on,mpx=off,hypervisor=on,pku=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vpindex=on,hv-runtime=on,hv-synic=on,hv-stimer=on,hv-stimer-direct=on,hv-reset=on,hv-frequencies=on,hv-reenlightenment=on,hv-tlbflush=on,hv-ipi=on \
	-m size=67108864k,slots=16,maxmem=268435456k \
	-overcommit mem-lock=off \
	-smp 8,maxcpus=120,sockets=15,dies=1,cores=8,threads=1 \
	-numa node,nodeid=0,cpus=0-119,mem=65536 \
	-uuid e942720e-e838-4039-9f66-f2a7b77e7dfc \
	-smbios type=1,manufacturer="Red Hat",product=RHEL,version=8.6-1.el8ev,serial=e942720e-e838-4039-9f66-f2a7b77e7dfc,uuid=e942720e-e838-4039-9f66-f2a7b77e7dfc,sku=8.6.0,family=RHV \
	-smbios type=2,manufacturer="Red Hat",product=RHEL-AV \
	-no-user-config -nodefaults \
	-rtc base=2023-03-21T17:15:42,driftfix=slew \
	-global kvm-pit.lost_tick_policy=delay \
	-no-hpet -no-shutdown \
	-boot strict=on,menu=on \
	-device piix3-usb-uhci,id=ua-4d8de959-f68a-45f8-9289-d1935d8d37af,bus=pci.0,addr=0x1.0x2 \
	-device virtio-serial-pci,id=ua-3cdd6881-eccd-452e-a09f-0554e4de9514,max_ports=16,bus=pci.0,addr=0x4 \
	-blockdev '{"driver":"host_device","filename":"/dev/sdc","aio":"native","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
	-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
	-device virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-2-format,id=ua-6b37a51f-3548-4304-a1c0-5b72c3852eea,bootindex=1,write-cache=on,serial=6b37a51f-3548-4304-a1c0-5b72c3852eea,werror=enospc \
	-blockdev '{"driver":"host_device","filename":"/dev/sdd","aio":"native","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
	-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
	-device virtio-blk-pci,bus=pci.0,addr=0x6,drive=libvirt-1-format,id=ua-17a8244c-f464-49bf-8bc2-f83e306e3bce,write-cache=on,serial=17a8244c-f464-49bf-8bc2-f83e306e3bce,werror=enospc \
	-netdev tap,id=hostua-10ce23dc-37c7-44ab-97b9-6ff074b9c5f1,vhost=on \
	-device virtio-net-pci,mq=on,vectors=10,host_mtu=1500,netdev=hostua-10ce23dc-37c7-44ab-97b9-6ff074b9c5f1,id=ua-10ce23dc-37c7-44ab-97b9-6ff074b9c5f1,mac=00:50:56:ad:46:c5,bus=pci.0,addr=0x3 \
	-drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/ISO/Win2016/en_windows_server_2016_updated_feb_2018_x64_dvd_11636692.iso \
        -device ide-cd,id=cd2,drive=drive_cd1,bus=ide.1,unit=1 \
        -cdrom /home/kvm_autotest_root/iso/windows/virtio-win-1.9.18-4.el8.iso \
	-vnc :2  \
	-vga std  \
	-device virtio-balloon-pci,id=ua-eb9cdc16-d058-4db9-a55e-ed0773c426a8,bus=pci.0,addr=0x7 \
	-msg timestamp=on \
	-monitor stdio

Thanks a lot~
Peixiu

Hi Peixiu,


But all the Windows Server virtual machines has enabled the no-shutdown parameter,

# grep qemu ~/03520635/0020-sosreport-xha-153-2023-05-15-jgexnot.tar.xz/sosreport-xha-153-2023-05-15-jgexnot/sos_commands/process/ps_auxwwwm | grep "guest" | wc -l
13

# grep qemu ~/03520635/0020-sosreport-xha-153-2023-05-15-jgexnot.tar.xz/sosreport-xha-153-2023-05-15-jgexnot/sos_commands/process/ps_auxwwwm | grep "guest" | grep hv | wc -l
13

# grep qemu ~/03520635/0020-sosreport-xha-153-2023-05-15-jgexnot.tar.xz/sosreport-xha-153-2023-05-15-jgexnot/sos_commands/process/ps_auxwwwm | grep "guest" | grep hv | grep no-shutdown | wc -l
13

And it makes sense, I found this mail that explains the behaviour of this parameter [1]

[1] https://listman.redhat.com/archives/libvir-list/2020-December/msg00053.html

Hi José,

Thanks for your reply and investigation~

From the virtio-win QE perspect, If we want to totally shutdown the vm, we should clear no-shutdown~
I'm not so clear why we set this param enbaled default for customer, and I'm not sure if RHV have some workaround for disable no-shutdown?

Do we possible to involve rhv qe or developer to have a look?

Thanks~
Peixiu

Hi Yan，

I can reproduce this issue with "no-shutdown", cannot reproduce this isse without "no-shutdown" param.
Could you help to check it?

Thanks a lot~
Peixiu

(In reply to Peixiu Hou from comment #8)
> Hi Yan，
> 
> I can reproduce this issue with "no-shutdown", cannot reproduce this isse
> without "no-shutdown" param.
> Could you help to check it?
> 
> Thanks a lot~
> Peixiu

Before moving the bug to virtualization management, can please test that shutting down the guest works when using qemu-ga?
1. Add no-shutdown to qemu command line
2. Run send shutdown command to qemu-ga

Tested with qemu-ga + no-shutdown in qemu command line.

VM can be shutdown in paused status, after send qemu-ga command. 
[root@dell-per750-34 ~]# nc -U /tmp/qga.sock 
{ "execute":"guest-shutdown","arguments":{"mode":"powerdown"}}

The vm get shutting down, and the vm into paused status.

(qemu) info status
VM status: paused (shutdown)
(qemu)  

The situation is same with we send shutdown -r -t 0 command in the vm directly.

I try to ping the vm before shutdown and after, vm can be pingable before send shutdown command, and vm cannot be pingable after into paused status.

virtio-win version: virtio-win-prewhql-238

Thanks~
Peixiu

Based on comment #10, comment #7 moving to libvirt

Tested about "-no-shutdown" in libvirt. The "--no-shutdown" command is add to guest by default (even when on_poweroff/on_reboot/on_crash in guest xml are set to "destroy") and I haven't found way to change/remove it.

Assigning to Michal for further investigation.