Bug 518022

Summary: 2.6.31 virtio_net oops in skb_copy_from_linear_data_offset()
Product: [Fedora] Fedora Reporter: James Laska <jlaska>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: rawhideCC: herbert.xu, itamar, jturner, kernel-maint, markmc, mtosatti, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-2.6.31-0.200.rc8.git2.fc12 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-03 14:42:30 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 498968    

Description James Laska 2009-08-18 14:00:59 UTC 
Description of problem:

Completing an nfs KVM install of rawhide using virtio disks (on an F-11 host) panics while rebooting

Version-Release number of selected component (if applicable):

 * Host (x86_64):
   * kernel-2.6.29.6-217.2.7.fc11.x86_64
   * libvirt-0.6.2-14.fc11.x86_64
   * qemu-system-x86-0.10.5-3.fc11.x86_64

 * Guest (i386):
   * kernel-2.6.31-0.125.4.2.rc5.git2.fc12.i686

How reproducible:

 * Only saw this 1 time so far

Steps to Reproduce:
1. Perform an automated kickstart install over nfs on a KVM virtio guest
  
Actual results:

disabling swap...
	/dev/mapper/vg_test1197-lv_swap
unmounting filesystems...
	/mnt/runtime done
	disabling /dev/loop0 LOOP_CLR_FD failed: 16
	/proc done
	/dev/pts done
	/sys done
	/mnt/stage2 done
	/selinux done
	/mnt/sysimage/boot done
	/mnt/sysimage/dev/pts done
	/mnt/sysimage/dev/shm done
	/mnt/sysimage/dev done
	/mnt/sysimage/proc done
	/mnt/sysimage/sys done
	/mnt/sysimage/selinux done
	/mnt/sysimage done
BUG: unable to handle kernel paging request at fffb680c
IP: [<c07740ca>] skb_copy_from_linear_data_offset+0x2c/0x4e
*pde = 00009067 *pte = 37842163 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/devices/virtio-pci/virtio1/block/vda/removable
Modules linked in: aes_generic fcoe libfcoe libfc scsi_transport_fc scsi_tgt ipv6 xts lrw gf128mul sha256_generic cbc dm_crypt dm_round_robin dm_multipath linear raid10 raid456 raid6_pq async_xor async_memcpy async_tx xor raid1 raid0 nfs lockd fscache nfs_acl auth_rpcgss sunrpc virtio_net virtio_blk virtio_pci iscsi_ibft iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi squashfs pcspkr edd floppy cramfs

Pid: 192, comm: wpa_supplicant Not tainted (2.6.31-0.125.4.2.rc5.git2.fc12.i686 #1) 
EIP: 0060:[<c07740ca>] EFLAGS: 00210282 CPU: 0
EIP is at skb_copy_from_linear_data_offset+0x2c/0x4e
EAX: f5ea8f00 EBX: 000005a8 ECX: 0000016a EDX: c213f304
ESI: c213f304 EDI: fffb680c EBP: c2161ab8 ESP: c2161aa8
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Process wpa_supplicant (pid: 192, ti=c2160000 task=f63cab80 task.ti=c2160000)
Stack:
 481d84b0 000005a8 000005a8 000005a8 c2161aec c0774a72 000005a8 00000001
<0> c2161ad8 f5ea8f00 fffb680c 00000000 000005a8 481d84b0 c2161b68 000005a8
<0> 00000000 c2161b04 f7d7af3c 000005a8 481d84b0 0000080c f62aa748 c2161b34
Call Trace:
 [<c0774a72>] ? skb_copy_bits+0x5e/0x1a0
 [<f7d7af3c>] ? xdr_skb_read_bits+0x34/0x60 [sunrpc]
 [<f7d7ad89>] ? xdr_partial_copy_from_skb+0x121/0x185 [sunrpc]
 [<f7d7af08>] ? xdr_skb_read_bits+0x0/0x60 [sunrpc]
 [<f7d7c700>] ? xs_tcp_data_recv+0x371/0x53b [sunrpc]
 [<f7d7af08>] ? xdr_skb_read_bits+0x0/0x60 [sunrpc]
 [<c07b6dbc>] ? tcp_read_sock+0x7d/0x193
 [<f7d7c38f>] ? xs_tcp_data_recv+0x0/0x53b [sunrpc]
 [<f7d7c36b>] ? xs_tcp_data_ready+0x6a/0x8e [sunrpc]
 [<c07be7db>] ? tcp_rcv_established+0x4fe/0x63a
 [<c07c5603>] ? tcp_v4_do_rcv+0x171/0x2c7
 [<c07c695c>] ? tcp_v4_rcv+0x3ea/0x5e2
 [<c07ab5b4>] ? ip_local_deliver_finish+0x13f/0x1ee
 [<c07ab6d7>] ? ip_local_deliver+0x74/0x8d
 [<c07ab05e>] ? ip_rcv_finish+0x31f/0x35a
 [<c07ab2bb>] ? ip_rcv+0x222/0x266
 [<c077df66>] ? netif_receive_skb+0x38e/0x3bf
 [<f7d39714>] ? virtnet_poll+0x4ca/0x633 [virtio_net]
 [<c077b9e0>] ? __rcu_read_lock+0x0/0x45
 [<c077e724>] ? net_rx_action+0xa7/0x1d3
 [<c044adbb>] ? __do_softirq+0x60/0x192
 [<c044ae23>] ? __do_softirq+0xc8/0x192
 [<c044af36>] ? do_softirq+0x49/0x7f
 [<c044b08a>] ? irq_exit+0x48/0x8c
 [<c0405705>] ? do_IRQ+0x92/0xb7
 [<c0404095>] ? common_interrupt+0x35/0x3c
 [<c04264c5>] ? paravirt_leave_lazy_mmu+0x0/0x22
 [<c042593d>] ? kvm_leave_lazy_mmu+0x5c/0x7e
 [<c04d74a5>] ? unmap_vmas+0x489/0x5c0
 [<c0446200>] ? task_unlock+0xe/0x34
 [<c04dc13f>] ? exit_mmap+0xb2/0x113
 [<c04421b3>] ? mmput+0x57/0xc0
 [<c0446492>] ? exit_mm+0xeb/0x104
 [<c0447fd0>] ? do_exit+0x19e/0x648
 [<c046f278>] ? trace_hardirqs_on_caller+0x122/0x155
 [<c04484ec>] ? do_group_exit+0x72/0x99
 [<c044853a>] ? sys_exit_group+0x27/0x3c
 [<c0403a50>] ? syscall_call+0x7/0xb
Code: 89 e5 57 56 53 83 ec 04 0f 1f 44 00 00 8b 5d 08 89 cf 65 8b 0d 14 00 00 00 89 4d f0 31 c9 89 d9 c1 e9 02 03 90 b4 00 00 00 89 d6 <f3> a5 89 d9 83 e1 03 74 02 f3 a4 8b 45 f0 65 33 05 14 00 00 00 
EIP: [<c07740ca>] skb_copy_from_linear_data_offset+0x2c/0x4e SS:ESP 0068:c2161aa8
CR2: 00000000fffb680c
---[ end trace f6d9dc8fdbaaf440 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 192, comm: wpa_supplicant Tainted: G      D    2.6.31-0.125.4.2.rc5.git2.fc12.i686 #1
Call Trace:
 [<c081ab29>] ? printk+0x22/0x39
 [<c081aa60>] panic+0x4b/0xf2
 [<c081e964>] oops_end+0xa0/0xc0
 [<c0427687>] no_context+0x11a/0x135
 [<c04277e4>] __bad_area_nosemaphore+0x142/0x15b
 [<c04268f4>] ? pvclock_clocksource_read+0x59/0xf4
 [<c04268f4>] ? pvclock_clocksource_read+0x59/0xf4
 [<c042781f>] bad_area_nosemaphore+0x22/0x39
 [<c081fc5c>] do_page_fault+0x168/0x2fa
 [<c081faf4>] ? do_page_fault+0x0/0x2fa
 [<c081dec8>] error_code+0x78/0x80
 [<c07740ca>] ? skb_copy_from_linear_data_offset+0x2c/0x4e
 [<c0774a72>] skb_copy_bits+0x5e/0x1a0
 [<f7d7af3c>] xdr_skb_read_bits+0x34/0x60 [sunrpc]
 [<f7d7ad89>] xdr_partial_copy_from_skb+0x121/0x185 [sunrpc]
 [<f7d7af08>] ? xdr_skb_read_bits+0x0/0x60 [sunrpc]
 [<f7d7c700>] xs_tcp_data_recv+0x371/0x53b [sunrpc]
 [<f7d7af08>] ? xdr_skb_read_bits+0x0/0x60 [sunrpc]
 [<c07b6dbc>] tcp_read_sock+0x7d/0x193
 [<f7d7c38f>] ? xs_tcp_data_recv+0x0/0x53b [sunrpc]
 [<f7d7c36b>] xs_tcp_data_ready+0x6a/0x8e [sunrpc]
 [<c07be7db>] tcp_rcv_established+0x4fe/0x63a
 [<c07c5603>] tcp_v4_do_rcv+0x171/0x2c7
 [<c07c695c>] tcp_v4_rcv+0x3ea/0x5e2
 [<c07ab5b4>] ip_local_deliver_finish+0x13f/0x1ee
 [<c07ab6d7>] ip_local_deliver+0x74/0x8d
 [<c07ab05e>] ip_rcv_finish+0x31f/0x35a
 [<c07ab2bb>] ip_rcv+0x222/0x266
 [<c077df66>] netif_receive_skb+0x38e/0x3bf
 [<f7d39714>] virtnet_poll+0x4ca/0x633 [virtio_net]
 [<c077b9e0>] ? __rcu_read_lock+0x0/0x45
 [<c077e724>] net_rx_action+0xa7/0x1d3
 [<c044adbb>] ? __do_softirq+0x60/0x192
 [<c044ae23>] __do_softirq+0xc8/0x192
 [<c044af36>] do_softirq+0x49/0x7f
 [<c044b08a>] irq_exit+0x48/0x8c
 [<c0405705>] do_IRQ+0x92/0xb7
 [<c0404095>] common_interrupt+0x35/0x3c
 [<c04264c5>] ? paravirt_leave_lazy_mmu+0x0/0x22
 [<c042593d>] ? kvm_leave_lazy_mmu+0x5c/0x7e
 [<c04d74a5>] unmap_vmas+0x489/0x5c0
 [<c0446200>] ? task_unlock+0xe/0x34
 [<c04dc13f>] exit_mmap+0xb2/0x113
 [<c04421b3>] mmput+0x57/0xc0
 [<c0446492>] exit_mm+0xeb/0x104
 [<c0447fd0>] do_exit+0x19e/0x648
 [<c046f278>] ? trace_hardirqs_on_caller+0x122/0x155
 [<c04484ec>] do_group_exit+0x72/0x99
 [<c044853a>] sys_exit_group+0x27/0x3c
 [<c0403a50>] syscall_call+0x7/0xb

Expected results:


Additional info:
Comment 1 Mark McLoughlin 2009-08-21 09:10:33 UTC 
Oh dear, this is new. It's nothing to do with the other NFS issues we're having in libvirt

Looks identical to http://www.kerneloops.org/raw.php?rawid=642150

Pity it's not reproducible

Could be a pvmmu issue perhaps?
Comment 2 Herbert Xu 2009-08-21 09:41:02 UTC 
The bogus pointer (fffb680c) comes from EDI, which is supplied by the sunrpc code.  So it doesn't appear to be a virtio-net or even general networking problem.  As the only report appears with KVM, I agree with you that perhaps it's a KVM MM issue.
Comment 3 Mark McLoughlin 2009-09-03 14:42:30 UTC 
Justin has just committed this patch from Marcelo

  http://patchwork.kernel.org/patch/43695/

* Thu Sep 03 2009 Justin M.Forbes <jforbes>
- Fix xen guest with stack protector. (#508120)
- Small kvm fixes.