Bug 2165169

Summary:	[Azure][ARM64][RHEL8] sysrq reboot waits a long time for VMBus unloading
Product:	Red Hat Enterprise Linux 8	Reporter:	Li Tian <litian>
Component:	kernel	Assignee:	Virtualization Maintenance <virt-maint>
kernel sub component:	Hyper-V	QA Contact:	Li Tian <litian>
Status:	CLOSED MIGRATED	Docs Contact:
Severity:	unspecified
Priority:	unspecified	CC:	eterrell, litian, vkuznets, xuli, yacao, yuxisun
Version:	8.8	Keywords:	MigratedToJIRA
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2171333 (view as bug list)		Environment:
Last Closed:	2023-09-22 14:27:04 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2171333

Description Li Tian 2023-01-28 01:10:20 UTC

Description of problem:
Azure ARM RHEL 8.8 use below command to trigger crash - 
# echo 1 > /proc/sys/kernel/sysrq; echo c > /proc/sysrq-trigger

Serial console shows below logs before a restart happens - 
[   36.517830] sysrq: SysRq : Trigger a crash
[   36.520721] Kernel panic - not syncing: sysrq triggered crash
[   36.520721] 
[   36.525900] CPU: 1 PID: 1766 Comm: bash Kdump: loaded Tainted: G        W        --------- -  - 4.18.0-447.el8.aarch64 #1
[   36.533966] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 05/16/2022
[   36.542059] Call trace:
[   36.543898]  dump_backtrace+0x0/0x178
[   36.546702]  show_stack+0x28/0x38
[   36.549147]  dump_stack+0x5c/0x74
[   36.551568]  panic+0x140/0x30c
[   36.553861]  sysrq_reset_seq_param_set+0x0/0xa8
[   36.557226]  __handle_sysrq+0x9c/0x190
[   36.559981]  write_sysrq_trigger+0x7c/0x98
[   36.562968]  proc_reg_write+0x84/0xd8
[   36.565670]  __vfs_write+0x4c/0x90
[   36.568121]  vfs_write+0xb0/0x1b8
[   36.570497]  ksys_write+0x70/0xd8
[   36.572884]  __arm64_sys_write+0x28/0x38
[   36.575752]  do_el0_svc+0xb4/0x188
[   36.578249]  el0_sync_handler+0x88/0xac
[   36.581059]  el0_sync+0x140/0x180
[   36.583462] SMP: stopping secondary CPUs
[   36.586264] Kernel Offset: 0x404bc6d00000 from 0xffff800010000000
[   36.590470] PHYS_OFFSET: 0xffffa6dc00000000
[   36.593308] CPU features: 0x0000044a,84814210
[   36.596941] Memory Limit: none
[   41.601628] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   46.620415] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   51.636353] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   56.659883] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   61.676422] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   66.695090] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   71.714063] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   76.744787] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   81.762957] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   86.780668] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   91.799017] hv_vmbus: Waiting for VMBus UNLOAD to complete
[   96.823126] hv_vmbus: Waiting for VMBus UNLOAD to complete
[  101.836149] hv_vmbus: Waiting for VMBus UNLOAD to complete
[  106.860822] hv_vmbus: Waiting for VMBus UNLOAD to complete
[  111.875220] hv_vmbus: Waiting for VMBus UNLOAD to complete
[  116.891644] hv_vmbus: Waiting for VMBus UNLOAD to complete
[  121.911215] hv_vmbus: Waiting for VMBus UNLOAD to complete
[  126.924527] hv_vmbus: Waiting for VMBus UNLOAD to complete
[  131.945844] hv_vmbus: Waiting for VMBus UNLOAD to complete
[  136.960236] hv_vmbus: Waiting for VMBus UNLOAD to complete
[  136.974631] hv_vmbus: Continuing even though VMBus UNLOAD did not complete
[  136.980798] Starting crashdump kernel...
[  136.983798] Bye!
[    0.000000] Booting Linux on physical CPU 0x0000000001 [0x413fd0c1]

Version-Release number of selected component (if applicable):
4.18.0-447.el8.aarch64

How reproducible:
100% when system fully boots.

Steps to Reproduce:
1. Start an Azure ARM RHEL 8.8 VM. Wait for a couple of minutes.
2. # echo 1 > /proc/sys/kernel/sysrq; echo c > /proc/sysrq-trigger
Actual results:
VMBus unload takes a long time before system can reboot.

Expected results:
No such stall.

Additional info:
1. No such issue on RHEL 9.2.
2. No such issue on x86_64 RHEL 8.8.

Comment 5 RHEL Program Management 2023-09-22 14:24:42 UTC

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 6 RHEL Program Management 2023-09-22 14:27:04 UTC

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.