Bug 740465

Summary: Host got crash when guest running netperf client with UDP_STREAM protocol with IPV6
Product: Red Hat Enterprise Linux 6 Reporter: Qunfang Zhang <qzhang>
Component: kernelAssignee: jason wang <jasowang>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.2CC: acathrow, dhoward, flang, herbert.xu, herrold, jpallich, jpirko, juzhang, michen, mkenneth, pmatouse, sforsber, tburke, virt-maint, xwei
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-2.6.32-211.el6 Doc Type: Bug Fix
Doc Text:
A scenario for this bug involves two hosts, configured to use IPv4 network, and two guests, configured to use IPv6 network. When a guest on host A attempted to send a large UDP datagram to host B, host A terminated unexpectedly. With this update, the ipv6_select_ident() function has been fixed to accept the in6_addr parameter and to use the destination address in IPv6 header when no route is attached, and the crashes no longer occur in the described scenario.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 14:14:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 723433, 748554, 748808    
Attachments:
Description Flags
Log for the host "vmcore" file
none
bt information for host vmcore file none

Description Qunfang Zhang 2011-09-22 06:55:06 UTC 
Description of problem:
Boot 2 guests on 2 hosts separately. Hosts are IPV4 network while the two guests are configured with IPV6 network. The two guests can ping each other. Then running netserver and netperf on the 2 guests separately. As a result, the host that run netperf client guest got crash. Running TCP_STERAM protocol is ok while meet the problem when running UPD_STREAM protocol.  

Version-Release number of selected component (if applicable):
kernel-2.6.32-198.el6.x86_64
qemu-kvm-0.12.1.2-2.190.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Boot 2 guests on 2 hosts. Guest command line:

/usr/libexec/qemu-kvm -M rhel6.2.0 -cpu cpu64-rhel6,+x2apic  -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -name RHEL6 -uuid 5820cc38-984e-49ed-b451-ea627b09ddda -monitor stdio -rtc base=localtime -boot c -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive file=/media/RHEL6.2-64-virtio-0823.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop -device ide-drive,bus=ide.0,unit=0,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:10:20:58,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/tmp/foo,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -usb -spice port=5930,disable-ticketing 

2. Configure IPV6 network for each guests.
#ifconfig
#ip addr flush eth0
#ip addr add 2002:5::10/64 dev eth0 (guest A)
#ip addr add 2002:5::8/64 dev eth0  (guest B)

3.Ping between the 2 guests.
#ping6 -I eth0 2002:5::10(8)

4.Run netserver on host A
#netserver -6

5.Run netperf client on host B (TCP_STREAM)
#netperf -6 -H 2002:5::10 -t TCP_STREAM

6.Run netperf client on host B (UDP_STREAM)
#netperf -6 -H 2002:5::10 -t UDP_STREAM
  
Actual results:
Step 3: ping available.
Step 5: can finish netperf TCP_STREAM test successfully.
Step 6: host got crash.

Expected results:
After step 6, no crash happens for both host and guest and netperf can finish successfully.

Additional info:
The log and bt file for host "vmcore" will be attached later.
Comment 1 Qunfang Zhang 2011-09-22 06:56:08 UTC 
Created attachment 524326 [details]
Log for the host "vmcore" file
Comment 2 Qunfang Zhang 2011-09-22 06:56:44 UTC 
Created attachment 524327 [details]
bt information for host vmcore file
Comment 3 Qunfang Zhang 2011-09-22 06:58:17 UTC 
Test with the same steps with IPV4 network for 5 times, no crash happens.
Comment 8 RHEL Program Management 2011-09-27 10:01:44 UTC 
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.
Comment 12 Qunfang Zhang 2011-09-29 08:52:05 UTC 
I re-test on latest rhel5.8 host, do not hit the issue.

Host A (rhel5.8):
kernel-2.6.18-286.el5
kvm-83-239.el5

1. Boot a rhel6.2 guest A on host A (rhel5.8), configure the ipv6 ip according to the bug description steps.  (netperf client)
2. Boot another rhel6.2 guest B on another host B, configure this guest with ipv6 addr. (netserver)
3. Run netperf on guest A,  the same steps as bug description too.

Result: No crash happens, both host and guest work well.
Comment 13 Jiri Pirko 2011-09-29 16:31:18 UTC 
Qunfang, would you please test upstream stable kernel (with the patch for CVE-2011-2699 applied) if it has the same problem? Thanks.
Comment 14 Qunfang Zhang 2011-09-30 02:38:23 UTC 
(In reply to comment #13)
> Qunfang, would you please test upstream stable kernel (with the patch for
> CVE-2011-2699 applied) if it has the same problem? Thanks.

OK, will do it and update result here after finish.
Comment 15 Qunfang Zhang 2011-09-30 02:45:30 UTC 
(In reply to comment #13)
> Qunfang, would you please test upstream stable kernel (with the patch for
> CVE-2011-2699 applied) if it has the same problem? Thanks.

Hi, Jiri
Sorry could you tell me where can I download a stable kernel?  Thanks.
Comment 16 Jiri Pirko 2011-09-30 05:35:30 UTC 
(In reply to comment #15)
> (In reply to comment #13)
> > Qunfang, would you please test upstream stable kernel (with the patch for
> > CVE-2011-2699 applied) if it has the same problem? Thanks.
> 
> Hi, Jiri
> Sorry could you tell me where can I download a stable kernel?  Thanks.

Usually it's on git.kernel.org (when it's up). Now I can't find it anywhere. Maybe please email greghk and ask him. Thanks.
Comment 17 Qunfang Zhang 2011-10-09 05:22:49 UTC 
(In reply to comment #13)
> Qunfang, would you please test upstream stable kernel (with the patch for
> CVE-2011-2699 applied) if it has the same problem? Thanks.

Hi, Jiri
The issue can be reproduced with the latest stable kernel.
Comment 18 Jiri Pirko 2011-10-10 11:08:08 UTC 
(In reply to comment #17)
> (In reply to comment #13)
> > Qunfang, would you please test upstream stable kernel (with the patch for
> > CVE-2011-2699 applied) if it has the same problem? Thanks.
> 
> Hi, Jiri
> The issue can be reproduced with the latest stable kernel.

Okay. So there would be best to resolve this via upstream. Please report this issue via mail to netdev.org ccing author or the patch Eric Dumazet <eric.dumazet>

Thanks.
Comment 19 Qunfang Zhang 2011-10-11 02:22:13 UTC 
(In reply to comment #18)
> (In reply to comment #17)
> > (In reply to comment #13)
> > > Qunfang, would you please test upstream stable kernel (with the patch for
> > > CVE-2011-2699 applied) if it has the same problem? Thanks.
> > 
> > Hi, Jiri
> > The issue can be reproduced with the latest stable kernel.
> 
> Okay. So there would be best to resolve this via upstream. Please report this
> issue via mail to netdev.org ccing author or the patch Eric Dumazet
> <eric.dumazet>
> 
> Thanks.

Ok, I have sent the mail to them.  Thanks.
Comment 20 jason wang 2011-10-11 02:33:40 UTC 
(In reply to comment #18)
> (In reply to comment #17)
> > (In reply to comment #13)
> > > Qunfang, would you please test upstream stable kernel (with the patch for
> > > CVE-2011-2699 applied) if it has the same problem? Thanks.
> > 
> > Hi, Jiri
> > The issue can be reproduced with the latest stable kernel.
> 
> Okay. So there would be best to resolve this via upstream. Please report this
> issue via mail to netdev.org ccing author or the patch Eric Dumazet
> <eric.dumazet>
> 
> Thanks.

Hello Jiri:

I've reported this upstream and post a patch for -stable.

Thanks
Comment 23 Aristeu Rozanski 2011-10-19 15:29:28 UTC 
Patch(es) available on kernel-2.6.32-211.el6
Comment 28 Qunfang Zhang 2011-10-26 08:11:52 UTC 
Verified the bug with the following packages with the same steps as bug description, and the issue does not exist any more after about 10 times attempt.

kernel-2.6.32-211.el6.x86_64
qemu-kvm-0.12.1.2-2.200.el6.x86_64

So, this issue is fixed.
Comment 29 Tomas Capek 2011-11-23 15:34:43 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
A scenario for this bug involves two hosts, configured to use IPv4 network, and two guests, configured to use IPv6 network. When a guest on host A attempted to send a large UDP datagram to host B, host A terminated unexpectedly. With this update, the ipv6_select_ident() function has been fixed to accept the in6_addr parameter and to use the destination address in IPv6 header when no route is attached, and the crashes no longer occur in the described scenario.
Comment 30 errata-xmlrpc 2011-12-06 14:14:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2011-1530.html