Bug 630830

Summary: nic:Network become unavailable in windows 2008 x64 guest when doing netperf test
Product: Red Hat Enterprise Linux 6 Reporter: Shirley Zhou <szhou>
Component: virtio-winAssignee: Yvugenfi <yvugenfi>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.0CC: bcao, byount, dawu, dfleytma, khong, mshao, plyons, qzhang, rhod, roberto.berto, tburke, vrozenfe, wqyuwss, ybendito
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---Flags: vrozenfe: needinfo-
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 706791 1038402 (view as bug list) Environment:
Last Closed: 2011-05-19 14:31:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 693757, 706791    
Attachments:
Description Flags
full netperf result
none
virtio-win-prewhql-0.1-7-nic.tar.gz none

Description Shirley Zhou 2010-09-07 06:32:29 UTC
Description of problem:
do netperf test in windows 2008 x64 guest with virtio network, after test last about 20mins,guest network become unavailable, can not ping host and netserver(external host).

C:\Users\Administrator>ping 10.66.85.229 (HOST)



Pinging 10.66.85.229 with 32 bytes of data:

Reply from 10.66.85.160: Destination host unreachable.

Reply from 10.66.85.160: Destination host unreachable.

Reply from 10.66.85.160: Destination host unreachable.



Ping statistics for 10.66.85.229:

    Packets: Sent = 3, Received = 3, Lost = 0 (0% loss)

Control-C

^C

C:\Users\Administrator>ping 10.66.85.230 (netserver)



Pinging 10.66.85.230 with 32 bytes of data:

Reply from 10.66.85.160: Destination host unreachable.

Reply from 10.66.85.160: Destination host unreachable.

Reply from 10.66.85.160: Destination host unreachable.



Ping statistics for 10.66.85.230:

    Packets: Sent = 3, Received = 3, Lost = 0 (0% loss)

Version-Release number of selected component (if applicable):
kernel-2.6.32-71.el6.x86_64
qemu-kvm-0.12.1.2-2.113.el6.x86_64
virtio-win-1.1.13-0(08/30/2010,6.0.209.605)

How reproducible:
Tried 3 times, all reproduce.

Steps to Reproduce:
1.run win08x64 guest using virt-manager,cli as:
/usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 24576 -smp 8,sockets=8,cores=1,threads=1 -name win2008x64 -uuid 9ef7359c-20b8-d247-812d-aba609c18594 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/win2008x64.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=localtime -boot c -drive file=/home/win2008x64.img,if=none,id=drive-virtio-disk0,boot=on,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/home/virtio-win-1.1.13-0.vfd,if=none,id=drive-fdc0-0-0,format=raw,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -drive file=/home/Windows2008-x64-customized.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/home/data.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fa:ce:1e,bus=pci.0,addr=0x3 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga std -device AC97,id=sound0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
2.run netserver on external host
3.run netperf in windows guest as script:

SERVER_IP=10.66.85.230

for proto in TCP_STREAM UDP_STREAM
do
    for sz in 32 64 128 256 512 1024 2048 4096 8192 16384 32768
    do
        netperf -H $SERVER_IP -c -C -t $proto -f m -l 180s -- -m $sz
    done
done

  
Actual results:
after about 20mins,netperf running window prompts:

TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.85.230 (10.66.85.

230) port 0 AF_INET

netperf: data send error: Connection reset by peer

len was -1

establish control: are you sure there is a netserver listening on 10.66.85.230 a

t port 12865?

establish_control could not establish the control connection from 0.0.0.0 port 0

 address family AF_UNSPEC to 10.66.85.230 port 12865 address family AF_UNSPEC

Then ping host and netserver, can not ping

Expected results:
netperf test can finish

Additional info:
This issue does not happen when running netperf on rtl8139 network.

Comment 1 Shirley Zhou 2010-09-07 06:34:46 UTC
Created attachment 443409 [details]
full netperf result

Comment 3 Keqin Hong 2010-09-08 08:35:11 UTC
The problem also exists on win7-32 and win7-64 guest

Version-Release number of selected component (if applicable):
kernel-2.6.32-71.el6.x86_64
qemu-kvm-0.12.1.2-2.113.el6.x86_64
virtio-win-1.1.13-0(08/30/2010,6.0.209.605)

Comment 4 Yvugenfi@redhat.com 2010-09-14 09:58:22 UTC
Can you test if this is reproducible without vhost?

Thanks.

Comment 5 Shirley Zhou 2010-09-14 10:08:54 UTC
(In reply to comment #4)
> Can you test if this is reproducible without vhost?
> 
> Thanks.

Hi,Yan

We did not use vhost in comment0 and comment3.

Thanks,

Comment 6 Shirley Zhou 2010-09-26 08:58:30 UTC
This bug reproduce when transfer big file(500M) from external box to win7 x64 guest. Raise severity to High.

Comment 7 Mike Cao 2011-01-28 06:57:47 UTC
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.132.el6.x86_64
# rpm -q virtio-win
virtio-win-1.1.16-0.el6.noarch

Reproduced in windows 7 64 bit guest when doing NTttcp network testing.

Comment 8 Yvugenfi@redhat.com 2011-01-28 19:26:40 UTC
Did you try to run QEMU with"sndbuf=0" parameter?

Comment 9 Mike Cao 2011-01-30 08:14:07 UTC
(In reply to comment #8)
> Did you try to run QEMU with"sndbuf=0" parameter?

start VM with -netdev tap,sndbuf=0..-device virtio-net-pci..

Also Reproduced this issue.

Comment 10 Dor Laor 2011-02-16 15:46:40 UTC
Can QE supply a DUMP image of the guest kernel once the network stops?
Dump file can be created by issuing an NMI command from the monitor.

Comment 17 Vadim Rozenfeld 2011-03-13 19:18:40 UTC
Please try the latest drivers from:
http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/6/win/virtio-win-prewhql-0.1.zip

Comment 18 Qunfang Zhang 2011-03-15 05:53:25 UTC
Reproduced this issue with win2k8-64 guests 2/2 with virtio-win-1.1.16-0.
Verified pass with win2k8-64 guests 2/2 with virtio-win-prewhql-0.1-6(above build).
Used the same steps as bug description.
Packages:
kernel-2.6.32-118.el6.x86_64
qemu-kvm-0.12.1.2-2.149.el6.x86_64
seabios-0.6.1.2-3.el6.x86_64
vgabios-0.6b-3.5.el6.noarch

I will try more guests and then update results here.

Comment 19 Qunfang Zhang 2011-03-16 04:23:38 UTC
Reproduced on win7-32 guest with virtio-win-1.1.16 and verified pass with virtio-win-prewhql-0.1-6.
So, I will change the status to VERIFIED.

Comment 21 Roberto Bertó 2011-03-18 19:12:36 UTC
Same issue here on win-2008-r2

With about 40Mbps and 20 tcp connections downloading data from SSH server windows shows network still connected but TCP doesn't work anymore.

Comment 22 Qunfang Zhang 2011-03-22 07:44:12 UTC
(In reply to comment #21)
> Same issue here on win-2008-r2
> 
> With about 40Mbps and 20 tcp connections downloading data from SSH server
> windows shows network still connected but TCP doesn't work anymore.

Could you provide the virtio-win version?

Comment 23 Roberto Bertó 2011-03-22 11:14:17 UTC
virtio-win-1.1.16-0 too

Comment 24 Yvugenfi@redhat.com 2011-03-22 11:38:17 UTC
Could you test with latest version from:
http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/6/win/virtio-win-prewhql-0.1.zip

Comment 25 Roberto Bertó 2011-03-22 11:55:07 UTC
Download link is broken download.devel.redhat.com has no IP

dig download.devel.redhat.com @8.8.8.8
;download.devel.redhat.com.	IN	A

Comment 26 Qunfang Zhang 2011-03-23 02:45:39 UTC
Created attachment 486947 [details]
virtio-win-prewhql-0.1-7-nic.tar.gz

Comment 27 Qunfang Zhang 2011-03-23 02:48:39 UTC
(In reply to comment #25)
> Download link is broken download.devel.redhat.com has no IP
> 
> dig download.devel.redhat.com @8.8.8.8
> ;download.devel.redhat.com. IN A

Hi, 
I upload a attachment. Please download and decompress it. You will get 2 directories "Vista" and "XP". 
For XP directory, it is for:
Windows XP and Windows 2003 (XP\x86 - 32bit, XP\amd64 - 64 bit)
For Vista direcory:
Windows Vista, Windows 7, Windows 2008, Windows 2008 R2 (Vista\x86 - 32bit, Vista\amd64 - 64 bit)

Comment 28 Roberto Bertó 2011-03-23 14:41:32 UTC
I ran same tests and issue did not happened again.

It appear to be SOLVED here

Comment 29 Qunfang Zhang 2011-03-24 01:54:22 UTC
(In reply to comment #28)
> I ran same tests and issue did not happened again.
> 
> It appear to be SOLVED here

OK, thanks for the verification.

Comment 31 Roberto Bertó 2011-04-05 15:43:19 UTC
Hi, it happened too on a Windows 2003 x64 Enterprise. 

After upgrading driver it fixed problem

Comment 35 errata-xmlrpc 2011-05-19 14:31:24 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0782.html

Comment 37 Leslie 2013-12-05 03:28:26 UTC
Hi,

I'm trying to run windows VM in openstack, then seems like the issue can be easily reproduced this problem in my environment with latest virt_win-0.1-74.iso. Below is related version information. 

Guest machine: windows 2008 R2 Service Pack 1
Guest Redhat VirtIO Ethernet Adapter version: 
   - Driver date: 6/19/2013
   - Driver version: 61.65.104.6500
Note that my initial virt_win.iso is version 0.1-65, then I update driver to 0.1-74 although system prompt it is latest one already. 

Host machine:  Ubuntu 12.04.3 LTS
Kernel: 3.8.0-29-generic
QEMU-KVM version: QEMU emulator version 1.0 (qemu-kvm-1.0)
Openstack: Havana.

Note that this problem is easily reproduced if you copy files cross 2 VMs on 2 seperate physical bare-metal server. If within same bare-metal server, you have to increase traffic load, then it can be reproduced.

It is very important for us. Please kindly let me know whether there are anything wrong.

Best Regards
Leslie

Comment 38 Yvugenfi@redhat.com 2013-12-05 08:27:45 UTC
(In reply to Leslie from comment #37)
> Hi,
> 
> I'm trying to run windows VM in openstack, then seems like the issue can be
> easily reproduced this problem in my environment with latest
> virt_win-0.1-74.iso. Below is related version information. 
> 
> Guest machine: windows 2008 R2 Service Pack 1
> Guest Redhat VirtIO Ethernet Adapter version: 
>    - Driver date: 6/19/2013
>    - Driver version: 61.65.104.6500
> Note that my initial virt_win.iso is version 0.1-65, then I update driver to
> 0.1-74 although system prompt it is latest one already. 
> 

The question - did you managed to update the driver after all? In case if you have WHQL signed driver and try to install newer version of the driver, Windows will behave as if the driver you have now (the old one) is a better match. 
Please verify the driver was updated.

> Host machine:  Ubuntu 12.04.3 LTS
> Kernel: 3.8.0-29-generic
> QEMU-KVM version: QEMU emulator version 1.0 (qemu-kvm-1.0)
> Openstack: Havana.
> 
> Note that this problem is easily reproduced if you copy files cross 2 VMs on
> 2 seperate physical bare-metal server. If within same bare-metal server, you
> have to increase traffic load, then it can be reproduced.
> 
> It is very important for us. Please kindly let me know whether there are
> anything wrong.
> 
> Best Regards
> Leslie

The question

Comment 39 Ronen Hod 2013-12-05 09:18:34 UTC
Leslie,

Thank you for taking the time to enter a bug report with us. We appreciate the feedback and look to use reports such as this to guide our efforts at improving our products. That being said, this bug tracking system is not a mechanism for requesting support, and we are not able to  guarantee the timeliness or suitability of a resolution.
 
If this issue is critical or in any way time sensitive, please raise a ticket through your regular Red Hat support channels to make certain  it receives the proper attention and prioritization to assure a timely resolution. 
 
For information on how to contact the Red Hat production support team, please visit:
https://www.redhat.com/support/process/production/#howto