Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 522458

Summary: Kernel BUG at drivers/net/tun.c:476 Process qemu-kvm (igb NIC)
Product: Red Hat Enterprise Linux 5 Reporter: Gurhan Ozen <gozen>
Component: kernelAssignee: Glauber Costa <gcosta>
Status: CLOSED DUPLICATE QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: high Docs Contact:
Priority: high    
Version: 5.4CC: aron.griffis, ehabkost, jburke, sgruszka, virt-maint, ykaul
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-03-28 12:27:33 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 483646    
Bug Blocks: 533192    

Description Gurhan Ozen 2009-09-10 11:22:46 UTC
Description of problem:

kvm: 7680: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 7680: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xffd33d30
kvm: 7680: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x530079
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at drivers/net/tun.c:476
invalid opcode: 0000 [1] SMP 
last sysfs file: /class/net/lo/ifindex
CPU 15 
Modules linked in: tun nls_utf8 loop nfs fscache nfs_acl ipt_MASQUERADE iptable_nat ip_nat xt_state ip_conntrack nfnetlink ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table dm_multipath scsi_dh video hwmon backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport ksm(U) kvm_intel(U) kvm(U) igb shpchp i2c_i801 8021q sr_mod i2c_core serio_raw cdrom sg pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 7680, comm: qemu-kvm Tainted: G      2.6.18-164.el5 #1
RIP: 0010:[<ffffffff8872d7d9>]  [<ffffffff8872d7d9>] :tun:tun_chr_readv+0x2b1/0x3a6
RSP: 0018:ffff81051fa8de48  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff81051fa8de98 RCX: 0000000045505450
RDX: ffff810728beb700 RSI: ffff81051fa8de9e RDI: ffff81051fa8de92
RBP: 0000000000010ff6 R08: 0000000000000000 R09: 0000000000000001
R10: ffff81051fa8de94 R11: 0000000000000048 R12: ffff81093ae98380
R13: ffff8108dd043d00 R14: 0000000000000000 R15: ffff81051fa8def8
FS:  00002b66e4fbf080(0000) GS:ffff81093fcbf0c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000006b2578 CR3: 000000049a929000 CR4: 00000000000026e0
Process qemu-kvm (pid: 7680, threadinfo ffff81051fa8c000, task ffff8108d4ad4080)
Stack:  ffff81093c383380 ffff81093f935a80 0000000000000000 ffff8108d4ad4080
 ffffffff8008be71 ffff8108dd043d28 ffff8108dd043d28 ffff8104bd3329c0
 000005a805ea0000 0000000000000000 000078f8503e1600 0000000000000000
Call Trace:
 [<ffffffff8008be71>] default_wake_function+0x0/0xe
 [<ffffffff8872d8e8>] :tun:tun_chr_read+0x1a/0x1f
 [<ffffffff8000b695>] vfs_read+0xcb/0x171
 [<ffffffff80011b72>] sys_read+0x45/0x6e
 [<ffffffff8005d28d>] tracesys+0xd5/0xe0


Code: 0f 0b 68 c8 e4 72 88 c2 dc 01 f6 42 0a 08 74 0c 80 4c 24 41 
RIP  [<ffffffff8872d7d9>] :tun:tun_chr_readv+0x2b1/0x3a6
 RSP <ffff81051fa8de48>
 <0>Kernel panic - not syncing: Fatal exception

See: http://rhts.redhat.com/cgi-bin/rhts/test_log.cgi?id=9934078
I wasn't doing anything special, just scheduled the job trying to test virtinstall scripts for some changes. I haven't run into this issue before, so it might very well be hardware specific. The host is:
http://lab.rhts.bos.redhat.com/cgi-bin/rhts/system.cgi?id=1105

Version-Release number of selected component (if applicable):
kvm-83-105.el5.x86_64
kernel-2.6.18-164.el5.x86_64

How reproducible:
Unsure. 

Steps to Reproduce:
1. I just submitted a test run which had a bunch of guests scheduled to be installed. It looks like this happened during installation of rhel4_x86_64_hvm_guest  
Actual results:


Expected results:


Additional info:

Comment 1 Yaniv Kaul 2009-09-10 11:36:50 UTC
Is that a different bug from https://bugzilla.redhat.com/show_bug.cgi?id=510789

Comment 3 Eduardo Habkost 2009-09-10 14:09:06 UTC
It is a kernel bug; the KVM module doesn't deal with networking, and qemu-kvm shouldn't be able to break it from userspace.

It may not be the same as bug #510789, but it may be caused by a similar bug on another NIC module (bug #510789 was for mlx4_en).

Comment 4 Eduardo Habkost 2009-09-10 14:20:44 UTC
Oops, I set the needinfo flag by mistake. I was going to ask for system info, but it is available on RHTS.

The host uses the 'igb' NIC module.

Comment 5 Eduardo Habkost 2009-09-22 16:43:05 UTC
I was this changed back to 'kvm' component? This is a kernel bug.

Comment 6 Stanislaw Gruszka 2009-10-20 13:12:30 UTC
Crash should be fixed now in 2.6.18-169 kernel, see bug 503851.

Comment 7 Eduardo Habkost 2009-10-20 14:02:51 UTC
(In reply to comment #6)
> Crash should be fixed now in 2.6.18-169 kernel, see bug 503851.  

I see no mention of a fix in 2.6.18-169 on that bug. I just see a patch for be2net and the URL for a test package.

Comment 8 Stanislaw Gruszka 2009-10-20 14:16:34 UTC
(In reply to comment #7)
> > Crash should be fixed now in 2.6.18-169 kernel, see bug 503851.  
> 
> I see no mention of a fix in 2.6.18-169 on that bug. I just see a patch for
> be2net and the URL for a test package. 

Ah - this is not mentioned, these patches are now in 169:
http://people.redhat.com/agospoda/rhel5/0049-lro-add-check-to-warn-if-forwarding-on-devices-that.patch
http://people.redhat.com/agospoda/rhel5/0131-tun-fix-LRO-crash.patch  

I think, you have to give new kernel a try.

Comment 9 Eduardo Habkost 2009-10-20 15:01:08 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > > Crash should be fixed now in 2.6.18-169 kernel, see bug 503851.  
> > 
> > I see no mention of a fix in 2.6.18-169 on that bug. I just see a patch for
> > be2net and the URL for a test package. 
> 
> Ah - this is not mentioned, these patches are now in 169:
> http://people.redhat.com/agospoda/rhel5/0049-lro-add-check-to-warn-if-forwarding-on-devices-that.patch
> http://people.redhat.com/agospoda/rhel5/0131-tun-fix-LRO-crash.patch  

It looks like those patches will make the packets be dropped instead of generating a "Kernel BUG" message, but won't solve the issue.

> 
> I think, you have to give new kernel a try.  

Testing won't hurt, anyway. Gurhan Ozen: could you test this using 2.6.18-169.el5?

Comment 10 Aron Griffis 2010-01-12 23:24:24 UTC
I think I'm seeing this bug, running RHEL 5.4 on HP bl460c. This is completely reproducible for me. What information would you like other than the trace?

Kernel BUG at drivers/net/tun.c:476
invalid opcode: 0000 [1] SMP
last sysfs file: /class/net/lo/ifindex
CPU 10                             
Modules linked in: tun ipt_MASQUERADE iptable_nat ip_nat xt_state ip_conntrack n
le_filter ip_tables x_tables autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc br
_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio c
scsi2 scsi_transport_iscsi2 scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq f
o hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug a
intel(U) kvm(U) bnx2x e1000e hpilo shpchp sg serio_raw pcspkr dm_raid45 dm_messa
napshot dm_zero dm_mirror dm_log dm_mod qla2xxx scsi_transport_fc cciss sd_mod s
ehci_hcd
Pid: 15820, comm: qemu-kvm Tainted: G      2.6.18-164.el5 #1
RIP: 0010:[<ffffffff888347d9>]  [<ffffffff888347d9>] :tun:tun_chr_readv+0x2b1/0x
RSP: 0018:ffff8108c02cfe48  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8108c02cfe98 RCX: 0000000051144414
RDX: ffff81049e105700 RSI: ffff8108c02cfe9e RDI: ffff8108c02cfe92
RBP: 0000000000010ff6 R08: 0000000000000000 R09: 0000000000000001
R10: ffff8108c02cfe94 R11: 0000000000000202 R12: ffff81049bdae1c0
R13: ffff8108fd6b2d00 R14: 0000000000000000 R15: ffff8108c02cfef8
FS:  00002ab60f5bcf90(0000) GS:ffff8104a00759c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002b7fa8489000 CR3: 00000008c01da000 CR4: 00000000000026e0
Process qemu-kvm (pid: 15820, threadinfo ffff8108c02ce000, task ffff8108c1bed820
Stack:  ffff8108c079fce0 ffff81091ef503c0 0000000000000000 ffff8108c1bed820                                                                                                                                                                                                                              
 ffffffff8008be71 ffff8108fd6b2d28 ffff8108fd6b2d28 00000001130f8080
 000005ea05ea0000 00000006c0080000 ffff028042005254 ffff81043ab38000
Call Trace:
 [<ffffffff8008be71>] default_wake_function+0x0/0xe
 [<ffffffff888348e8>] :tun:tun_chr_read+0x1a/0x1f
 [<ffffffff8000b695>] vfs_read+0xcb/0x171
 [<ffffffff80011b72>] sys_read+0x45/0x6e
 [<ffffffff8005d28d>] tracesys+0xd5/0xe0


Code: 0f 0b 68 c8 54 83 88 c2 dc 01 f6 42 0a 08 74 0c 80 4c 24 41
RIP  [<ffffffff888347d9>] :tun:tun_chr_readv+0x2b1/0x3a6
 RSP <ffff8108c02cfe48>
 <0>Kernel panic - not syncing: Fatal exception

Comment 11 Stanislaw Gruszka 2010-01-13 06:45:27 UTC
(In reply to comment #10)
> I think I'm seeing this bug, running RHEL 5.4 on HP bl460c. This is completely
> reproducible for me. What information would you like other than the trace?

Could you try kernel from http://people.redhat.com/jwilson/el5/ and see if this helps?

Comment 12 Aron Griffis 2010-01-14 22:57:52 UTC
(In reply to comment #11)
> Could you try kernel from http://people.redhat.com/jwilson/el5/ and see if this
> helps?    

Yes, 2.6.18-183.el5 fixes the problem

Comment 13 Stanislaw Gruszka 2010-01-15 07:52:28 UTC
(In reply to comment #12)
> Yes, 2.6.18-183.el5 fixes the problem    

Ok. If I remember correctly there can be performance problems (or maybe now that is fixed as well, don't know). If performance will be bad LRO should be disabled in network driver, in case of bnx2x driver use disable_tpa=1 option.

Comment 14 Stanislaw Gruszka 2010-01-21 12:59:55 UTC
I think we can close this bug, no?

Comment 15 Glauber Costa 2010-01-21 13:04:41 UTC
history shows it was never moved to POST, ON_QA, or whatever. Although there is a fix, how sure we are about it getting into the product?

I'll verify it right now

Comment 16 Gurhan Ozen 2010-01-21 15:54:46 UTC
I don't know why this fell out of the radar. I'll reserve 
intel-s3ea2-04.rhts.bos.redhat.com   and retest.

Comment 18 Prarit Bhargava 2010-03-28 12:27:33 UTC
According to comment #8, this is a dup of 483646.

*** This bug has been marked as a duplicate of bug 483646 ***