Bug 591494

Summary: Virtio: Transfer file caused guest in same vlan abnormally quit
Product: Red Hat Enterprise Linux 6 Reporter: Amos Kong <akong>
Component: qemu-kvmAssignee: Michael S. Tsirkin <mst>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.0CC: ailan, amit.shah, ddumas, gyue, mjenner, stanislav.polasek, syeghiay, tburke, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-0.12.1.2-2.101.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 606238 (view as bug list) Environment:
Last Closed: 2010-11-10 21:24:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 606238    

Description Amos Kong 2010-05-12 12:17:23 UTC
Description of problem:
Boot up two guests, and join them in same valn, when transfer file between two guests, the receive guest unexpected quit. QEMU process of guest2 ouputs "virtio-net truncating packet".
This problem only exists when using Virtio nic.


Version-Release number of selected component (if applicable):
guest kernel: 2.6.9-89.25.ELsmp
host kernel: 2.6.32-24.el6.x86_64
# rpm -qa |grep qemu
qemu-img-0.12.1.2-2.51.el6.x86_64
gpxe-roms-qemu-0.9.7-6.3.el6.noarch
qemu-kvm-debuginfo-0.12.1.2-2.51.el6.x86_64
qemu-kvm-tools-0.12.1.2-2.51.el6.x86_64
qemu-kvm-0.12.1.2-2.51.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Boot up two rhel4 guests
2. Join guest into this vlan
guest1)# vconfig add eth0 10
guest1)# ifconfig eth0.10 192.168.1.1
guest2)# vconfig add eth0 10
guest2)# ifconfig eth0.10 192.168.1.2
3. Transfer file from guest1 to guest2
guest2)# nc -l -p 2012
guest1)# dd if=/dev/urandom of=a.out bs=1M count=1
guest1)# nc 192.168.1.2 2012 < a.out
  
Actual results:
receive guest(guest2) unexpected quit

Expected results:
transfer file successfully

Additional info:

1. command line:
guest1)# qemu -name vm1 -monitor tcp:0:6001,server,nowait -drive file=/usr/local/staf/test/RHEV/kvm-new/kvm-test/tests/kvm/images/RHEL-4.8-64-virtio.qcow2,if=virtio,cache=none,boot=on -net nic,vlan=0,model=virtio,macaddr=00:03:FC:1C:52:55 -net tap,vlan=0,ifname=e1000_0_6001,script=/usr/local/staf/test/RHEV/kvm-new/kvm-test/tests/kvm/scripts/qemu-ifup-switch,downscript=no -m 4096 -smp 2 -soundhw ac97 -snapshot -redir tcp:5000::22 -vnc :0 -serial unix:/tmp/serial-20100512-161240-4BCA,server,nowait

guest2)# qemu -name vm2 -monitor tcp:0:6002,server,nowait -drive file=/usr/local/staf/test/RHEV/kvm-new/kvm-test/tests/kvm/images/RHEL-4.8-64-virtio.qcow2,if=virtio,cache=none,boot=on -net nic,vlan=0,model=virtio,macaddr=00:03:FC:1C:8e:26 -net tap,vlan=0,ifname=e1000_0_6002,script=/usr/local/staf/test/RHEV/kvm-new/kvm-test/tests/kvm/scripts/qemu-ifup-switch,downscript=no -m 4096 -smp 2 -soundhw ac97 -snapshot -redir tcp:5000::22 -vnc :1 -serial unix:/tmp/serial-20100512-161242-i0KL,server,nowait
(qemu) virtio-net truncating packet

Comment 2 RHEL Program Management 2010-05-12 14:36:11 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Comment 3 Amos Kong 2010-05-13 03:22:54 UTC
This bug was only reproduced for RHEL4.8 guest with virtio NIC.

Comment 15 Amos Kong 2010-06-23 11:32:14 UTC
host kernel: 2.6.32-36.el6.x86_64
guest kernel: 2.6.9-89.28.ELsmp
# rpm -qa |grep qemu
qemu-kvm-0.12.1.2-2.77.el6.x86_64
gpxe-roms-qemu-0.9.7-6.3.el6.noarch
qemu-kvm-tools-0.12.1.2-2.77.el6.x86_64
qemu-img-0.12.1.2-2.77.el6.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.77.el6.x86_64

[ The test refult of different qemu option ]

1. Command line with '-net tap' option:
Bug can be reproduced.
Output: virtio-net truncating packet: offset 1514, size 1518, hdr_len 10

#qemu-kvm -name 'vm1' -monitor tcp:0:6002,server,nowait -drive file=/root/autotest/client/tests/kvm/images/RHEL-4.8-64.qcow2,if=virtio,cache=none,boot=on,aio=native -net nic,vlan=0,model=virtio,macaddr=02:B7:5E:9A:6f:36 -net tap,vlan=0,ifname=virtio_0_6002,downscript=no,vhost=on -m 512 -smp 1 -soundhw ac97 -vnc :1 -spice port=8001,disable-ticketing  -serial unix:/tmp/serial-20100623-190659-BTxs,server,nowait


2. Command line with '-netdev tap' option:
Guest could not crash, but file transfer always fail, I'll report another bug later with detail info.
Output: virtio-net truncating packet: offset 1514, size 1518, hdr_len 10
#qemu -name 'vm1' -monitor unix:'/tmp/monitor-humanmonitor1-20100623-184427-46jt',server,nowait -drive file='/root/autotest/client/tests/kvm/images/RHEL-4.8-64.qcow2',if=virtio,cache=none,boot=on,aio=native -net nic,vlan=0,netdev=pO8A,model=virtio,macaddr='02:B7:5E:9A:93:15' -netdev tap,id=pO8A,ifname=virtio_0_8001,script=/root/autotest/client/tests/kvm/scripts/qemu-ifup-switch,downscript=no,vhost=on -m 512 -smp 1 -vnc :1 -spice port=8001,disable-ticketing  -serial unix:/tmp/serial-20100623-184427-46jt,server,nowait


Do we only need test with netdev option ? do you try to reproduce with '-net tap' option ?

I'm testing with you provided packages, will reply the result later.

Comment 16 Amos Kong 2010-06-23 11:43:52 UTC
The state of this package is failed.
https://brewweb.devel.redhat.com/taskinfo?taskID=2541095

--

Bug also existed with this packages (using '-net tap' option)
http://brewweb.devel.redhat.com/brew/taskinfo?taskID=2541095

Comment 18 Michael S. Tsirkin 2010-06-23 11:47:26 UTC
yes, we only care about -netdev

Comment 20 Michael S. Tsirkin 2010-06-23 16:26:31 UTC
OK, I went over the code. There are several issues:
1. when offloads are disabled, rhel4.8 posts packets which
   are too short:
#define MAX_PACKET_LEN (ETH_HLEN+ETH_DATA_LEN)

this is fixed upstream e918085aaff34086e265f825dd469926b1aec4a4

2. qemu crashes when not enough buffers. should drop
   packet instead.

3. why are offloads disabled? 
   I will send a debug breaw build to debug this.

4. vhost seems to truncate packet. should drop instead.
     I will send a debug breaw build to debug this.

Comment 21 Amos Kong 2010-06-24 09:34:41 UTC
Tested with https://brewweb.devel.redhat.com/taskinfo?taskID=2545310

qemu of guest2 outputted: (using '-netdev' option,  vhost=off)
virtio-net truncating packet: i 1 mergeable 0 offset 1524, size 1528, guest hdr 10, host hdr 10 num 2 base 0x7f122f9bc270 len 10 size 1524 size0 10

# gdb -p $guest2_pid
...
(gdb) c
Continuing.
[Thread 0x7f1234c0b710 (LWP 10160) exited]
[New Thread 0x7f1234c0b710 (LWP 10166)]
[Thread 0x7f1234c0b710 (LWP 10166) exited]
[New Thread 0x7f1234c0b710 (LWP 10167)]
[Thread 0x7f1234c0b710 (LWP 10167) exited]
[New Thread 0x7f1234c0b710 (LWP 10169)]
[Thread 0x7f1234c0b710 (LWP 10169) exited]
[New Thread 0x7f1234c0b710 (LWP 10171)]
[Thread 0x7f1234c0b710 (LWP 10171) exited]
[Thread 0x7f120f065710 (LWP 10141) exited]
[Thread 0x7f1236810710 (LWP 10129) exited]

Program exited with code 01.
(gdb) bt
No stack.

Comment 22 Michael S. Tsirkin 2010-06-24 10:12:11 UTC
I think that with the original command line (-net with vlan=, no -netdev
or id= in either guest 1 or guest 2), it will no longer crash with qemu-kvm-0.12.1.2-2.72.el6 
Correct?

Comment 23 Amos Kong 2010-06-24 11:26:00 UTC
(In reply to comment #22)
> I think that with the original command line (-net with vlan=, no -netdev
> or id= in either guest 1 or guest 2), it will no longer crash with
> qemu-kvm-0.12.1.2-2.72.el6 
> Correct?    

Incorrect, bug also existed.

Comment 24 Michael S. Tsirkin 2010-06-24 16:03:31 UTC
Please test with
http://brewweb.devel.redhat.com/brew/taskinfo?taskID=2550177
this should fix the exit, but file transfer will still fail.

Comment 25 Amos Kong 2010-06-29 08:23:42 UTC
(In reply to comment #24)
> Please test with
> http://brewweb.devel.redhat.com/brew/taskinfo?taskID=2550177
> this should fix the exit, but file transfer will still fail.    

yes.   guest doesn't quit, and transfer file failed.

host dmesg exists noise when vhost is enable.
"Discarded truncated rx packet:  len 1528 > 1524"

When disable vhost, no noise.

Comment 29 Michael S. Tsirkin 2010-07-12 16:48:25 UTC
fixes sent.
note we can not make vlans work without guest changes but we can make qemu not exit.

Comment 34 Amos Kong 2010-08-04 08:29:10 UTC
Could not reproduce bug with qemu-kvm-0.12.1.2-2.104.el6
Move to VERIFIED.

Comment 35 releng-rhel@redhat.com 2010-11-10 21:24:53 UTC
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.