Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1457920

Summary:	Hot-unplugging PCI Express virtio devices causes driver reload on Windows Server 2016
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	Ladi Prosek <lprosek>
Component:	qemu-kvm	Assignee:	Vadim Rozenfeld <vrozenfe>
qemu-kvm sub component:	General	QA Contact:	jingzhao <jinzhao>
Status:	CLOSED CURRENTRELEASE	Docs Contact:
Severity:	high
Priority:	high	CC:	ailan, chayang, coli, jinzhao, juzhang, kanderso, knoel, lijin, marcel, rbalakri, virt-maint, vrozenfe
Version:	8.0	Flags:	knoel: mirror+
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-18 01:23:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1744438

Description Ladi Prosek 2017-06-01 14:17:25 UTC

Description of problem:
Hot-unplugging a PCIe virtio device results in Windows shutting down and unloading the driver, almost immediately followed by reloading and initializing the driver again, and then unloading it for good a few seconds later.

If the timing is right, the device's BARs may get unmapped as the driver is initializing, exercising otherwise rarely hit error code paths (see bug 1455488 for one such case).

Version-Release number of selected component (if applicable):
kernel-3.10.0-671.el7.x86_64
qemu-kvm-rhev-2.9.0-5.el7.x86_64
virtio-win-prewhql-0.1-137

Tested with virtio-serial and balloon, the observed behavior is the same. The scope is likely all virtio devices or even all PCIe devices.

How reproducible:
100%

Steps to Reproduce:
1. Boot a VM with

  -device ioh3420,bus=pcie.0,id=root1.0 \
  -device virtio-balloon,bus=root1.0,id=balloon0 \

2. device_del balloon0
3. Watch the driver shut down, reinitialize, and shut down again with DbgView for example.

Actual results:
Windows unloads, loads, and unloads the driver.

Expected results:
Windows unloads the driver.

Additional info:

Connecting the device to a downstream port doesn't seem to make a difference:

  -device ioh3420,bus=pcie.0,id=root1.0 \
  -device x3130-upstream,bus=root1.0,id=upstream1 \
  -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
  -device virtio-balloon,bus=downstream1,id=balloon0 \

Comment 2 Ladi Prosek 2017-06-01 14:25:52 UTC

CC Marcel to
1) Tell me that I'm doing it wrong.
2) Or take a guess on what might be causing this (a wild guess is fine too).

I have opened this as a qemu-kvm-rhev bug because virtio-win drivers do not directly participate in hot-(un)-plug. PnP logic is generally handled by Windows and it's unlikely that we would be able to change it.

Comment 3 Marcel Apfelbaum 2017-06-05 16:38:39 UTC

(In reply to Ladi Prosek from comment #2)
> CC Marcel to
> 1) Tell me that I'm doing it wrong.
> 2) Or take a guess on what might be causing this (a wild guess is fine too).
> 
> I have opened this as a qemu-kvm-rhev bug because virtio-win drivers do not
> directly participate in hot-(un)-plug. PnP logic is generally handled by
> Windows and it's unlikely that we would be able to change it.

Hi Ladi,
Sorry it took me some time for the first response.
From the QEMU point of view we are sending basic PCIe hot plug
events, nothing special there. (a bug is always possible, of course...)

I tried with win10 (tomorrow I'll get a win16 guest from Gal).
I opened windbbg and did a few hotplug/unplug operations and I saw nothing
special in Windbg (I especially didn't see the drivers load/unload). I selected all options for debug of course.

I have virtio 1.9 drivers.

The steps you mentioned are just fine, but I suggest to try starting
the hot-unplug sequence from windows itself. In this way you
are sure Windows does not use the device at hot-unplugged time.

Can you guide me on how to see the driver loading/unloading?

Comment 4 Ladi Prosek 2017-06-06 11:31:42 UTC

Thanks Marcel,

No worries, this is not a critical issue, it's unlikely that it would be considered a RHEL 7.4 blocker for RHEL 7.4, even if it ends up being a QEMU bug.

Please try it with virtio-net. The NetKVM driver version 100.73.104.12600 extracted from virtio-win-1.9.0-3.el7.noarch.rpm has pretty verbose debug output compiled in. Apologies for not including this info in comment 0.

My command line now has

  -netdev user,id=hostnet1 \
  -device virtio-net,netdev=hostnet1,bus=root1.0,id=virtio-net0 \

After
(qemu) device_del virtio-net0

You should see something like this in windbg or DbgView:

00000001        13.37042046     [ParaNdis6_Pause]=>
<driver shutting down>
00000070        13.63768864     [ParaNdis6_Unload]=>

then a gap of 100-150 milliseconds and:

00000072        13.74919987     [ParaNdis_DebugInitialize] Crash callback registered
00000073        13.74920845     [DriverEntry]=>
00000074        13.74921322     Aug 11 2016 04:05:29built 6.30
<driver initializing, then shutting down again>

and finally

00000175        15.62887096     [ParaNdis6_Unload]=>


Starting the hot-unplug sequence from Windows (i.e. "ejecting" the device) results in the same symptoms.

Thanks!
Ladi

Comment 10 Ademar Reis 2020-02-05 22:43:57 UTC

QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks