Bug 1376082

On a side note: the driver also prevented the installation of the qemu-ga-x86.msi file.  The installation process would hang.

Could you try with latest netkvm driver,build 126?

I will gladly test.  Could you send me a link?

I only know to look on https://fedorapeople.org/groups/virt/virtio-win/

Hi

Please let me have the link to the build 126 netkvm driver.

Thanks

should be available at fedorapeople site (https://fedorapeople.org/groups/virt/virtio-win/repo/latest/) soon.

Best regards,
Vadim.

The -126 build is there now

Hi

After installing all the build 126 drivers, and then restarting the VM, it again got to the login screen, and after entering the credentials, the Welcome screen displayed, and the little dotty wheel spun and spun and spun and spun.  At that time, the CPU was 5-6%, and the disk read/write was mostly 0, but occasionally blipped to about 200-300 KiB/s.

I reverted to the snapshot I took before loading the drivers.  Logging in now took 2 seconds for the Welcome screen to disappear and the Desktop to show.

This time, I left all the drivers at the verions that they were at (which was 118 for all, except the netkvm which was 113), and then upgraded only the netkvm to 126, and restarted.  At the login prompt, it again got stuck.  After 10 minutes observing the "Welcome" prompt and spinning wheel, on the virt-manager menu I selected Virtual Machine/Shut down/Reboot.  The VM immediately changed to the "Restarting" screen ... where it stayed until I powered the VM off 5 minutes later.  Again, the disk usage was 0 KiB/s, and this time the CPU was 0-1%.

So it appears that build 126 shares the same problem that build 118 does as pertains to the netkvm driver.

To be clear: I installed the Win8.1/amd64 driver onto a 64-bit Windows 8.1 VM.

Let me know of any other testing you need done.

(In reply to Louis van Dyk from comment #8)
> Hi
> 
> After installing all the build 126 drivers, and then restarting the VM, it
> again got to the login screen, and after entering the credentials, the
> Welcome screen displayed, and the little dotty wheel spun and spun and spun
> and spun.  At that time, the CPU was 5-6%, and the disk read/write was
> mostly 0, but occasionally blipped to about 200-300 KiB/s.
> 
> I reverted to the snapshot I took before loading the drivers.  Logging in
> now took 2 seconds for the Welcome screen to disappear and the Desktop to
> show.
> 
> This time, I left all the drivers at the verions that they were at (which
> was 118 for all, except the netkvm which was 113), and then upgraded only
> the netkvm to 126, and restarted.  At the login prompt, it again got stuck. 
> After 10 minutes observing the "Welcome" prompt and spinning wheel, on the
> virt-manager menu I selected Virtual Machine/Shut down/Reboot.  The VM
> immediately changed to the "Restarting" screen ... where it stayed until I
> powered the VM off 5 minutes later.  Again, the disk usage was 0 KiB/s, and
> this time the CPU was 0-1%.
> 
> So it appears that build 126 shares the same problem that build 118 does as
> pertains to the netkvm driver.
> 

Can you please provide the command line that is used to run the VM?

Thanks,
Yan.
> To be clear: I installed the Win8.1/amd64 driver onto a 64-bit Windows 8.1
> VM.
> 
> Let me know of any other testing you need done.

I run the process from virt-manager, so don't know of another way to give you the cmdline other than doing this:

# cat /proc/26730/cmdline 
/usr/bin/qemu-system-x86_64-machineaccel=kvm-namewin8.1-TEST,debug-threads=on-S-machinepc-i440fx-2.6,accel=kvm,usb=off,vmport=off-cpuqemu64,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff-m2048-realtimemlock=off-smp2,sockets=2,cores=1,threads=1-uuid335c567b-ee32-4225-85d8-0423ed561e62-no-user-config-nodefaults-chardevsocket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-win8.1-TEST/monitor.sock,server,nowait-monchardev=charmonitor,id=monitor,mode=control-rtcbase=localtime,driftfix=slew-globalkvm-pit.lost_tick_policy=discard-no-hpet-no-shutdown-globalPIIX4_PM.disable_s3=1-globalPIIX4_PM.disable_s4=1-bootmenu=on,strict=on-devicenec-usb-xhci,id=usb,bus=pci.0,addr=0x5-devicevirtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6-drivefile=/media/2TB-Drive/KVM_VMs/win81TEST.qcow2,format=qcow2,if=none,id=drive-virtio-disk0-devicevirtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=3-drivefile=/home/louis/Downloads/kvm/virtio-win-0.1.113.iso,format=raw,if=none,id=drive-ide0-0-1,readonly=on-deviceide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=2-netdevtap,fd=27,id=hostnet0,vhost=on,vhostfd=30-devicevirtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a0:14:15,bus=pci.0,addr=0x3-netdevtap,fd=31,id=hostnet1,vhost=on,vhostfd=32-devicevirtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:e3:b9:22,bus=pci.0,addr=0xa-chardevpty,id=charserial0-deviceisa-serial,chardev=charserial0,id=serial0-chardevspicevmc,id=charchannel0,name=vdagent-devicevirtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0-chardevsocket,id=charchannel1,path=/var/lib/libvirt/qemu/channel/target/domain-2-win8.1-TEST/org.qemu.guest_agent.0,server,nowait-devicevirtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0-deviceusb-tablet,id=input0-spiceport=5901,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on-deviceqxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2-deviceintel-hda,id=sound0,bus=pci.0,addr=0x4-devicehda-duplex,id=sound0-codec0,bus=sound0.0,cad=0-chardevspicevmc,id=charredir0,name=usbredir-deviceusb-redir,chardev=charredir0,id=redir0-chardevspicevmc,id=charredir1,name=usbredir-deviceusb-redir,chardev=charredir1,id=redir1-chardevspicevmc,id=charredir2,name=usbredir-deviceusb-redir,chardev=charredir2,id=redir2-chardevspicevmc,id=charredir3,name=usbredir-deviceusb-redir,chardev=charredir3,id=redir3-devicevirtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7-objectrng-random,id=objrng0,filename=/dev/random-devicevirtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x9-devicepvpanic,ioport=1285-msgtimestamp=on


I have also uploaded the XML definition file for my test VM.

Created attachment 1208654 [details]
/etc/libvirt/qemu/win8.1-TEST.xml

This is the XML file for the VM being used to test.

I found the "ww" parameter to ps aux which gave me a better command line output.  Here it is:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name win8.1-TEST,debug-threads=on -S -machine pc-i440fx-2.6,accel=kvm,usb=off,vmport=off -cpu qemu64,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 335c567b-ee32-4225-85d8-0423ed561e62 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-win8.1-TEST/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/media/2TB-Drive/KVM_VMs/win81TEST.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=3 -drive file=/home/louis/Downloads/kvm/virtio-win-0.1.113.iso,format=raw,if=none,id=drive-ide0-0-1,readonly=on -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=2 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a0:14:15,bus=pci.0,addr=0x3 -netdev tap,fd=31,id=hostnet1,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:e3:b9:22,bus=pci.0,addr=0xa -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channel/target/domain-2-win8.1-TEST/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -spice port=5901,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -chardev spicevmc,id=charredir2,name=usbredir -device usb-redir,chardev=charredir2,id=redir2 -chardev spicevmc,id=charredir3,name=usbredir -device usb-redir,chardev=charredir3,id=redir3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x9 -device pvpanic,ioport=1285 -msg timestamp=on

Hi, I have the similar bug, symptoms:
* Networking not working, unable to assign ip address
* Shutdown stays forever

However I'm on Windows 10 insider preview(release preview channel) on VirtualBox, with build 126 netkvm driver installed.

= Additional info =

Symptoms:
* Stuck when attempt to disable device or uninstall driver

Windows: 
* 10.0.14393 build

With further testing, I can confirm that netkvm build 117 also works.  This implies that it broke as of build 118 (with build 126 also broken).

@Vdragon:  Does it work when you change to the build 117 driver (or 113)?
You can retrieve it at:
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.117-1/
As a tip to get your Windows 10 to log in, disable / disconnect / uninstall your network device in Virtualbox.  Then you can log in and replace the driver.

@Louis van Dyk
I've tried the build 117 driver and it works properly.

(In reply to Louis van Dyk from comment #15)
We'd like to crash the guest system intentionally with creation of kernel memory dump for further analysis.
Please boot the guest system and configure it for kernel-mode memory dump according to https://support.microsoft.com/en-us/kb/969028
Then install the problematic netkvm driver (118 or higher) and reproduce the problem after restart or on shutdown (stuck system with no progress).
Issue NMI command: "sudo virsh qemu-monitor-command --hmp win8.1-TEST 'nmi'"
The guest system shall enter blue screen and start collecting data; wait until it reaches 100% and restarts, then stop the guest machine (virsh destroy, for ex.)
Restart the system with Realtek and login. Under c:\windows should be memory.dmp file of approx 200-300MB. Please ZIP it with password and share with us using DropBox or any other tool.

Thanks,
Yuri

(In reply to Ｖ字龍(Vdragon) from comment #14)
> = Additional info =
> 
> Symptoms:
> * Stuck when attempt to disable device or uninstall driver
> 
> Windows: 
> * 10.0.14393 build

It would be good to have a dump file from your system when the driver is stuck on disable.
Please boot the guest system and configure it for kernel-mode memory dump according to https://support.microsoft.com/en-us/kb/969028
If the disk file does not have too much free space, this configuration is useful:
https://blogs.msdn.microsoft.com/wer/2009/02/09/kernel-dump-storage-and-clean-up-behavior-in-windows-7/
When the system is stuck upon driver disable, please try issue reboot command in guest OS ("shutdown -r -t 0"), the system shall produce blue screen in 10 minutes or less. Alternative is to issue NMI event to VM ("VBoxManage.exe debugvm "Windows VM" injectnmi") should do the job. Under c:\windows should be memory.dmp file of approx 200-300MB. Please ZIP it with password and share with us using DropBox or any other tool.

Thanks,
Yuri

(In reply to ybendito from comment #18)
> It would be good to have a dump file from your system when the driver is
> stuck on disable.
> Please boot the guest system and configure it for kernel-mode memory dump
> according to https://support.microsoft.com/en-us/kb/969028
> If the disk file does not have too much free space, this configuration is
> useful:
> https://blogs.msdn.microsoft.com/wer/2009/02/09/kernel-dump-storage-and-
> clean-up-behavior-in-windows-7/
> When the system is stuck upon driver disable, please try issue reboot
> command in guest OS ("shutdown -r -t 0"), the system shall produce blue
> screen in 10 minutes or less. Alternative is to issue NMI event to VM
> ("VBoxManage.exe debugvm "Windows VM" injectnmi") should do the job. Under
> c:\windows should be memory.dmp file of approx 200-300MB. Please ZIP it with
> password and share with us using DropBox or any other tool.
> 
> Thanks,
> Yuri

I've sent the memory dump link via the mail address.  The memory dump from the first mail is produced under shutting down the guest **without** disabling the driver first using the NMI method, while the second one is fully following the direction and generated automatically by "Bug Check 0x9F: DRIVER_POWER_STATE_FAILURE"

> I've sent the memory dump link via the mail address.  The memory dump from
> the first mail is produced under shutting down the guest **without**
> disabling the driver first using the NMI method, while the second one is
> fully following the direction and generated automatically by "Bug Check
> 0x9F: DRIVER_POWER_STATE_FAILURE"

Working on dumps. 

Thanks,
Yuri

BTW note that the Windows copy in the guest I'm using is actually installed natively on host, which then exposed and booted in Oracle Virtualbox using the "raw hard disk access" feature.  I also overwrote a few guest's DMI information and ACPI tables using host ones so the memory dump may some what be suspicious.

(In reply to Louis van Dyk from comment #15)
> With further testing, I can confirm that netkvm build 117 also works.  This
> implies that it broke as of build 118 (with build 126 also broken).
> 
I would be very helpful to have a dump file from the system running with qemu/virsh. It is possible that the problem with VBox is different than one with latest builds on system running on qemu.

Thanks,
Yuri

(In reply to Ｖ字龍(Vdragon) from comment #14)
> = Additional info =
> 
> Symptoms:
> * Stuck when attempt to disable device or uninstall driver
> 
> Windows: 
> * 10.0.14393 build

I see the network device driver by VBox works with line interrupt and not with MSIX. This may cause the device to not work properly, expecially on latest builds.
I do not find whether this is possible to configure the device to work with MSIX. If not, I would suggest to use network driver for XP/2003 - it is in folder XP or 2k3 in drivers package. The issue is under investigation.

Hi

I must apologise profusely.  I have been away for work, then ill, and then been away for work agin, and have been unable to attend to providing the diagnostics.

I will get onto that this week.

Thanks

Hi

I have completed the kernel dump as requested.  I installed NETKVM Driver 126 and rebooted.  Upon logging in the screen remained at logging in stage with the spinning circle of balls.  There I issued the NMI command, and let it reboot, then killed the VM, changed Network Cards to Realtek and logged in.

The below link is to the MEMORY.DMP file, zipped as requested.  The password is
driver126
https://www.dropbox.com/s/q1imphgcm3bmwg1/MEMORY.DMP.zip?dl=0

Again apologies for my late response to this.

Let me know what else you need me to to - I won't take so long again.

Regards
Louis

(In reply to Louis van Dyk from comment #25)
> Hi
> 
> I have completed the kernel dump as requested.  I installed NETKVM Driver
> 126 and rebooted.  Upon logging in the screen remained at logging in stage
> with the spinning circle of balls.  There I issued the NMI command, and let
> it reboot, then killed the VM, changed Network Cards to Realtek and logged
> in.
> 
> The below link is to the MEMORY.DMP file, zipped as requested.  The password
> is
> driver126
> https://www.dropbox.com/s/q1imphgcm3bmwg1/MEMORY.DMP.zip?dl=0
> 
> Again apologies for my late response to this.
> 
> Let me know what else you need me to to - I won't take so long again.
> 
> Regards
> Louis

Unfortunately, you run preview build of Win8.1 (16610) for which OS symbols are not available for public and we're not able to explore OS internals to get closer to root cause. I can see that network driver internals are OK and the driver does not seem to cause the problem directly. Is there any chance that the problem can be reproduced on official release of Win8.1 or any other official release?

Good news at last!!

Whatever fault was introduced from v118 has been resolved.  I can confirm that virtio-win-0.1.130-1.noarch no longer exhibits this problem.

(I was having a hard time locating another VM to test against.  I had eventually found a Corporate 8.1 but that didn't have the issue that I was experiencing with the 118 etc drivers!)

Thank you for your patience and assistance in this ticket.

You may close this call.

(In reply to Louis van Dyk from comment #27)
> Good news at last!!
> 
> Whatever fault was introduced from v118 has been resolved.  I can confirm
> that virtio-win-0.1.130-1.noarch no longer exhibits this problem.
> 
> (I was having a hard time locating another VM to test against.  I had
> eventually found a Corporate 8.1 but that didn't have the issue that I was
> experiencing with the 118 etc drivers!)
> 
> Thank you for your patience and assistance in this ticket.
> 
> You may close this call.

Thank you for testing the driver. I am glad that the problem is solved.

I tried with MSFT officailly released win8.1(en_windows_8_1_enterprise_x64_dvd_2971902.iso),can not reproduce with virtio-win-prewhql-118;

According to comment#26 and comment#27,seems this issue can only reproduced on win8.1 preview build 16610.

Louis,could you please provide win8.1-64 build16610 iso somewhere I can access so that QE can reproduce and verify this bug?

I re-tested this at first with an Enterprise DVD also did not reproduce the bug.  The version I was running on my problem case was Win 8.1 Pro.  I have just obtained a Wi8.1 Pro DVD, and also do not have the problem with v118 or v126.

It seems that I was testing using an old (preview) DVD image.

Do you still need to go through the testing since it works on the released images, and v130 has fixed the issue on the preview image?  If you do, I'll create a new Google account and upload the image there for you.

(In reply to Louis van Dyk from comment #31)
> I re-tested this at first with an Enterprise DVD also did not reproduce the
> bug.  The version I was running on my problem case was Win 8.1 Pro.  I have
> just obtained a Wi8.1 Pro DVD, and also do not have the problem with v118 or
> v126.
> 
> It seems that I was testing using an old (preview) DVD image.
> 
> Do you still need to go through the testing since it works on the released
> images, and v130 has fixed the issue on the preview image?  If you do, I'll
> create a new Google account and upload the image there for you.

No. I think we are OK with current results.

Thank you very much for testing.

change status to verifed according to comment#27 and comment#32