Bug 619010

Summary: Win2003 virtio net driver doesn't work through host upgrade due to PCI addr change
Product: Red Hat Enterprise Linux 6 Reporter: Keqin Hong <khong>
Component: qemu-kvmAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: low    
Version: 6.0CC: michen, mkenneth, tburke, virt-maint, ykaul
Target Milestone: rcKeywords: TestBlocker
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-07-29 10:03:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Win2003 virtio net driver doesn't work through host upgrade
none
device manager full screen shot
none
pci slot change with -M rhel5.4.0 on rhel6 host none

Description Keqin Hong 2010-07-28 10:50:12 UTC
Created attachment 434978 [details]
Win2003 virtio net driver doesn't work through host upgrade

Description of problem:
Install a Window2003-32/64 guest on rhel5.5 with virtio-drivers-1.0.0-45801.vfd. Then, upgrade virtio drivers (storage, nic) with virtio-win-1.1.8-0.vfd.
After that, boot the guest on RHEL6 host and the netkvm driver for virtual NIC doesn't work no matter booting with -M rhel5.4.0 or -M rhel6.0.0.


Version-Release number of selected component (if applicable):
virtio-win-1.1.8-0
qemu-kvm-0.12.1.2-2.99.el6.x86_64
kernel-2.6.32-52.el6.x86_64

How reproducible:
2/2

Steps to Reproduce:
1. Install a Win2003 guest on rhel5.5 with virtio-drivers-1.0.0-45801.vfd
# /usr/libexec/qemu-kvm -S -M rhel5.4.0 -m 1024 -smp 1 -name win2003-64-100 -uuid be004cc0-f89e-5c70-f9b3-3efea105bee2 -no-kvm-pit-reinjection -monitor stdio -localtime -boot c -drive file=/var/lib/libvirt/images/176nfs/100/win2003-64.img,if=virtio,index=0,cache=none,boot=on -drive file=/var/lib/libvirt/images/176nfs/winISO/en_win_srv_2003_r2_enterprise_x64_with_sp2_cd1_X13-06188.iso,if=ide,media=cdrom,index=2 -drive file=/var/lib/libvirt/images/176nfs/virtio-win/virtio-drivers-1.0.0-45801.vfd,if=floppy,index=0 -net nic,model=virtio,macaddr=54:52:00:1d:c0:af,vlan=0 -net tap,script=/etc/qemu-ifup,vlan=0 -parallel none -usb -usbdevice tablet -vnc 10.66.91.176:2 -k en-us

2. Upgrade virtio-win driver to virtio-win-1.1.8-0

3. Start the guest on RHEL6 host 
# /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name win2003-64-100 -uuid be004cc0-f89e-5c70-f9b3-3efea105bee2 -nodefconfig -nodefaults -monitor stdio -rtc base=localtime -boot c -drive file=/var/lib/libvirt/images/176nfs/100/win2003-64-sn1.img,if=none,id=drive-virtio-disk0,boot=on,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:1d:c0:af,bus=pci.0,addr=0x4 -usb -device usb-tablet,id=input0 -vnc 10.66.91.175:2 -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
  
Actual results:
virtio net driver doesn't work through host upgrade

Expected results:
virtio net driver work as it is on rhel5. Guest network is ok.


Additional info:

Comment 3 Yvugenfi@redhat.com 2010-07-28 16:28:23 UTC
Please provide more information on the "doesn't work" symptom, for example:

1. What do you see in device manage? Is the driver installed or there are unrecognized devices? Or maybe there are not corresponding devices at all.

2. If network driver is installed, what do you see when running "ipconfig -a" and "getmac"? what happens to ping?


From the first glance - if the drivers where not signed , they might not be installed after reboot (as you use specific bus location for RHEL6.0 - device might switch its place on the bus). This is MS Windows behavior, please use only signed drivers for this test.

Comment 4 Keqin Hong 2010-07-29 03:02:51 UTC
(In reply to comment #3)
> Please provide more information on the "doesn't work" symptom, for example:
> 
> 1. What do you see in device manage? Is the driver installed or there are
> unrecognized devices? Or maybe there are not corresponding devices at all.
> 
Forgot to mention there's an attachment in Comment 0.
Pls look at https://bugzilla.redhat.com/attachment.cgi?id=434978

> 2. If network driver is installed, what do you see when running "ipconfig -a"
> and "getmac"? what happens to ping?
The device is not configured correctly. No IP or MAC available.

> 
> 
> From the first glance - if the drivers where not signed , they might not be
> installed after reboot (as you use specific bus location for RHEL6.0 - device
> might switch its place on the bus). This is MS Windows behavior, please use
> only signed drivers for this test.    
The guest installed on RHEL5 with virtio-drivers-1.0.0-45801 (which is a signed version) couldn't be booted on RHEL6 (It got BSOD which was a known issue). I had to update the driver to 1.1.8 to boot the guest on RHEL6. Also the problem doesn't exist for win7/xp/2008.

Comment 5 Keqin Hong 2010-07-29 03:30:52 UTC
The problem doesn't exist if I keep the PCI addr unchanged for the virtual NIC. (On RHEL5 under qemu monitor, I used "info pci" to get the pci addr of the NIC, which I later passed to CLI of RHEL6)

However, as mentioned in Comment 4, there's no need to do this for winXP/7/2008.

BTW, to boot on RHEL6 host Win-guests (2003/win7/2008) which were originally installed on RHEL5 host, we need to update virtio-win driver to 1.1.8. Or else, the guests will get BSOD (XP-32 seems to be an exception). Using Vadim's workaround (cache=writethrough) doesn't seem to work for win2003 SFAIK, see https://bugzilla.redhat.com/show_bug.cgi?id=609831#c2.

Comment 6 Yvugenfi@redhat.com 2010-07-29 06:07:50 UTC
* Please provide screen shot of full device manger window.
* Please provide screen shot of "general" tab for NIC.

Comment 7 Keqin Hong 2010-07-29 06:16:56 UTC
Created attachment 435192 [details]
device manager full screen shot

Comment 8 Yvugenfi@redhat.com 2010-07-29 06:30:05 UTC
As it can be seen from the screen shot - driver is not installed.

Driver is not installed because the device changed its location and the driver is not signed. This is "by design" behavior for Windows OS.

Please either move the bug to QEMU component (why the device moved?) or close the bug.

Comment 9 Keqin Hong 2010-07-29 07:48:47 UTC
OK, I will move this to qemu-kvm and change the title a bit.

Comment 10 Keqin Hong 2010-07-29 08:04:59 UTC
On rhel5.5, the typical default PCI addresses assigned are:
 QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info pci
 ...
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0xffffffff [0x001e].
  Bus  0, device   4, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0xffffffff [0x003e].
  Bus  0, device   5, function 0:
    RAM controller: PCI device 1af4:1002
      IRQ 0.
      BAR0: I/O at 0xffffffff [0x001e].

Whereas currently management tool like libvirt assigns pci addr=0x3 to virtio-balloon device by default. This will cause virtio net driver not to work when moving Win2003 guest from rhel5 to rhel6 host.
Therefore, a mechanism to keep PCI addr unchanged is in high demand.

Comment 11 Dor Laor 2010-07-29 10:03:36 UTC
I'll close this bug since libvirt will make sure to keep the pci slots the same when using -M rhel5.X so it should be fine.

Can you also make sure that the rhel5 virtio drivers will work w/o upgrade over new rhel6 host (from an image that run over rhel5 host).
Please keep the pci slots the same.

Comment 12 Keqin Hong 2010-07-30 10:09:36 UTC
(In reply to comment #11)
> I'll close this bug since libvirt will make sure to keep the pci slots the same
> when using -M rhel5.X so it should be fine.
OK, still Old guests created by libvirt do not have their pci slots kept in xml configuration file.

> 
> Can you also make sure that the rhel5 virtio drivers will work w/o upgrade over
> new rhel6 host (from an image that run over rhel5 host).
> Please keep the pci slots the same.    

Tested with winxp(32)/7/2003/2008 and pci slots unchanged. All except XP got BSOD without virtio-win-driver being upgraded, and all seemed to work well with virtio-win-driver being upgraded to 1.1.9.

Comment 13 Dor Laor 2010-08-01 13:21:43 UTC
(In reply to comment #12)

> > Can you also make sure that the rhel5 virtio drivers will work w/o upgrade over
> > new rhel6 host (from an image that run over rhel5 host).
> > Please keep the pci slots the same.    
> 
> Tested with winxp(32)/7/2003/2008 and pci slots unchanged. All except XP got
> BSOD without virtio-win-driver being upgraded, and all seemed to work well with
> virtio-win-driver being upgraded to 1.1.9.    

We need to use the -M rhel5.5 flag when you run those.
Please run only one of them to see if it changes something.
Also please provide the BSOD details in case it fails

Comment 14 Keqin Hong 2010-08-02 03:36:50 UTC
(In reply to comment #13)

> We need to use the -M rhel5.5 flag when you run those.

-M rhel5.5, not -M rhel5.4.0? SFAIK we used -M rhel5.4.0 by default for all guests hosted by rhel5.

> Please run only one of them to see if it changes something.
> Also please provide the BSOD details in case it fails    

No BSOD was found when using virtio-win-1.1.9 with either -M rhel5.4.0 or -M rhel6.0.0.

Still, one thing I should mention that there's a slot change when running the same guest with -M rhel5.4.0 and without assigning slot explicitly on rhel6 host. But libvirt may deal with it.
On rhel5 host, (qemu) info pci
"  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 11.
      BAR0: I/O at 0xc040 [0xc05f].
  Bus  0, device   4, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 10.
      BAR0: I/O at 0xc080 [0xc0bf]."
while on rhel6 host with -M rhel5.4.0 and without assigning slot explicitly, (qemu) info pci
  Bus  0, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0xffffffffffffffff [0x003e].
      id "virtio-disk0"
  Bus  0, device   4, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0xffffffffffffffff [0x001e].
      id "net0"

Details can be found through attachment of next Comment.

Comment 15 Keqin Hong 2010-08-02 03:37:49 UTC
Created attachment 435948 [details]
pci slot change with -M rhel5.4.0 on rhel6 host

Comment 16 Dor Laor 2010-08-02 09:24:10 UTC
(In reply to comment #14)
> (In reply to comment #13)
> 
> > We need to use the -M rhel5.5 flag when you run those.
> 
> -M rhel5.5, not -M rhel5.4.0? SFAIK we used -M rhel5.4.0 by default for all
> guests hosted by rhel5.

Please switch to rhel5.5 (I think even rhel5.5.4 which is latest Z stream update). It contains some kvmclock and live migration fixes.

> 
> > Please run only one of them to see if it changes something.
> > Also please provide the BSOD details in case it fails    
> 
> No BSOD was found when using virtio-win-1.1.9 with either -M rhel5.4.0 or -M
> rhel6.0.0.
> 
> Still, one thing I should mention that there's a slot change when running the
> same guest with -M rhel5.4.0 and without assigning slot explicitly on rhel6
> host. But libvirt may deal with it.

That's expected, this is why libvirt is needed

> On rhel5 host, (qemu) info pci
> "  Bus  0, device   3, function 0:
>     Ethernet controller: PCI device 1af4:1000
>       IRQ 11.
>       BAR0: I/O at 0xc040 [0xc05f].
>   Bus  0, device   4, function 0:
>     SCSI controller: PCI device 1af4:1001
>       IRQ 10.
>       BAR0: I/O at 0xc080 [0xc0bf]."
> while on rhel6 host with -M rhel5.4.0 and without assigning slot explicitly,
> (qemu) info pci
>   Bus  0, device   3, function 0:
>     SCSI controller: PCI device 1af4:1001
>       IRQ 0.
>       BAR0: I/O at 0xffffffffffffffff [0x003e].
>       id "virtio-disk0"
>   Bus  0, device   4, function 0:
>     Ethernet controller: PCI device 1af4:1000
>       IRQ 0.
>       BAR0: I/O at 0xffffffffffffffff [0x001e].
>       id "net0"
> 
> Details can be found through attachment of next Comment.

Comment 17 Keqin Hong 2010-08-03 10:21:03 UTC
Hi Dor,
According to my testing, using -M rhel5.5.0 also works for winxp/7/2003/2008/2008R2 with virtio-win-1.1.9 or 1.1.10.
If there's any problem later on, I will open new bug tracking it. (Currently I suspect switching to different machine types back and forth may cause problem over which I will test more).