Bug 716864

Summary: ipheth: WARNING: at net/sched/sch_generic.c:256 dev_watchdog+0x108/0x17c():
Product: [Fedora] Fedora Reporter: Dmitry Petrich <dmitry>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: ASSIGNED --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: bowie.owens, dane_ruyle, fullung, gansalmon, itamar, jforbes, jonathan, kernel-maint, madhu.chinakonda, michele, mlbianchi, woodard
Target Milestone: ---Keywords: Tracking
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: abrt_hash:869ffb09debb1aa0e831caffab68dc618e7bc2d0
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-07-11 16:36:45 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 702723    

Description Dmitry Petrich 2011-06-27 05:56:46 EDT
abrt version: 2.0.3
architecture:   x86_64
cmdline:        ro root=UUID=161e99ae-1b8c-4e76-945a-3d4f8ff93b86 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us-acentos rhgb quiet
comment:        1. Unknown
component:      kernel
kernel:         2.6.38.8-32.fc15.x86_64
kernel_tainted: 512
kernel_tainted_long: Taint on warning.
os_release:     Fedora release 15 (Lovelock)
package:        kernel
reason:         WARNING: at net/sched/sch_generic.c:256 dev_watchdog+0x108/0x17c()
time:           Mon Jun 27 12:50:34 2011

backtrace:
:WARNING: at net/sched/sch_generic.c:256 dev_watchdog+0x108/0x17c()
:Hardware name: MMMM49G
:NETDEV WATCHDOG: eth0 (ipheth): transmit queue 0 timed out
:Modules linked in: hidp sunrpc rfcomm sco bnep cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 l2cap ip6table_filter ip6_tables fuse snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep snd_seq arc4 snd_seq_device iwlagn iwlcore r852 sm_common nand snd_pcm nand_ids mac80211 nand_ecc btusb snd_timer r8169 ipheth mii uvcvideo asus_laptop videodev mtd sparse_keymap v4l2_compat_ioctl32 joydev serio_raw iTCO_wdt snd cfg80211 iTCO_vendor_support soundcore bluetooth snd_page_alloc rfkill microcode ipv6 mmc_block sdhci_pci sdhci mmc_core firewire_ohci firewire_core crc_itu_t nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
:Pid: 0, comm: kworker/0:0 Not tainted 2.6.38.8-32.fc15.x86_64 #1
:Call Trace:
: <IRQ>  [<ffffffff8105511a>] warn_slowpath_common+0x83/0x9b
: [<ffffffff810551d5>] warn_slowpath_fmt+0x46/0x48
: [<ffffffff813db469>] ? netif_tx_lock+0x4a/0x7c
: [<ffffffff813db5f7>] dev_watchdog+0x108/0x17c
: [<ffffffff81061378>] run_timer_softirq+0x1a4/0x266
: [<ffffffff8107aaf8>] ? tick_do_broadcast.constprop.3+0x56/0x8e
: [<ffffffff813db4ef>] ? dev_watchdog+0x0/0x17c
: [<ffffffff8105ae4c>] __do_softirq+0xd2/0x19d
: [<ffffffff810acaa1>] ? handle_IRQ_event+0x58/0x11f
: [<ffffffff8100aadc>] call_softirq+0x1c/0x30
: [<ffffffff8100c101>] do_softirq+0x46/0x81
: [<ffffffff8105afd0>] irq_exit+0x49/0x8b
: [<ffffffff8147c006>] do_IRQ+0x8e/0xa5
: [<ffffffff81475f13>] ret_from_intr+0x0/0x15
: <EOI>  [<ffffffff8129cd1a>] ? arch_local_irq_enable+0xb/0xd
: [<ffffffff81074211>] ? sched_clock_idle_wakeup_event+0x17/0x1a
: [<ffffffff8129d90b>] acpi_idle_enter_bm+0x21d/0x255
: [<ffffffff81398d54>] cpuidle_idle_call+0xe7/0x166
: [<ffffffff81008321>] cpu_idle+0xa5/0xdf
: [<ffffffff81464dba>] start_secondary+0x20c/0x20e
Comment 1 Dave Jones 2011-07-11 16:36:45 EDT

*** This bug has been marked as a duplicate of bug 702723 ***
Comment 2 Dmitry Petrich 2011-07-26 07:45:24 EDT
Package: kernel
Architecture: x86_64
OS Release: Fedora release 15 (Lovelock)

Comment
-----
no idea, how it happens
Comment 3 Josh Boyer 2011-09-26 15:07:03 EDT
Does this still happen with the 2.6.40.x kernel updates in F15?
Comment 4 Dane 2011-10-02 18:36:52 EDT
For me, this is happening on Hyper-V.  How to get the Kernel updates ?
How about an ISO build?  :)
Comment 5 Josh Boyer 2011-10-03 07:25:55 EDT
(In reply to comment #4)
> For me, this is happening on Hyper-V.  How to get the Kernel updates ?

Boot into the VM, then run a yum update?

> How about an ISO build?  :)

There is no F15 iso that has a 2.6.40 kernel.  You could try the F16 Beta isos that should be released tomorrow.
Comment 6 Dane 2011-10-03 11:25:48 EDT
Can't run YUM.  Well I can, but it won't find anything because Kernel crash when it hits the network layer.  That's the whole problem.  NICs can get their IP's from DHCP but when the OS goes to use the NIC, the Kernel crashes.  

I spent several hours on this last night.  Got frustrated and moved to CentOS where I find the same issue occurs when I have more then one CPU added to the VM.
1 CPU NIC works fine.   2 CPUs and NIC no longer works, kernel crash.  Went to 1 CPU and 2 NICs and craziness again.  I think I can repeat/remedy the problem by simply changing the number of CPUs in the VM. I think I can repeat/remedy the problem by simply changing the number of CPUs in the VM.  

Unfortunately, this is not the case with Fedora 15.

Sure hope this is the same issue. ?
Comment 7 Dane 2011-10-03 14:50:26 EDT
Also tried installing Msft's Linux integration components v3.0 (I think that is the latest).  And it does not install correctly.   Think it is something with the RPM arguments.  It complains the current RPM (itself) is needed by kernel and missing dependency.  I can force it with the --nodeps  but when verifying installation after reboot, it modules are not found.. Example, /sbin/modinfo hv_vmbus
Comment 8 Dane 2011-10-03 15:14:51 EDT
Sorry for the spam. 

Just tried Alpha16.   Downloaded the 255MB install, it seems to be working, downloading and installing what it needs.   Also downloading the 3.3GB DVD.
I'm thinking this looks good.  Would still like to verify with the Integration Components right?
Comment 9 Josh Boyer 2011-10-03 15:26:44 EDT
(In reply to comment #8)
> Sorry for the spam. 
> 
> Just tried Alpha16.   Downloaded the 255MB install, it seems to be working,
> downloading and installing what it needs.   Also downloading the 3.3GB DVD.
> I'm thinking this looks good. 

That's good news.

> Would still like to verify with the Integration Components right?

I'm not sure what those are, but if they are some out-of-tree Microsoft provided kernel modules, then we don't really need to know anything about those.  We don't support them.
Comment 10 Dane 2011-10-04 13:25:42 EDT
Kernel crash after install completed and system rebooted.

Think it was because I selected to send the system profile at the end of installation.

When the system came up, it tried to send the profile and kernel crashes, no more network.

Why would the installer be OK but not the installed operating system?
Comment 11 Josh Boyer 2011-11-20 10:14:58 EST
*** Bug 755158 has been marked as a duplicate of this bug. ***
Comment 12 Josh Boyer 2011-11-28 09:14:40 EST
*** Bug 757230 has been marked as a duplicate of this bug. ***
Comment 13 Albert Strasheim 2012-03-29 14:13:37 EDT
I've seen this bug or something like it with 3.3.0-4.fc16.x86_64.debug.

[ 2384.630835] ------------[ cut here ]------------
[ 2384.635621] WARNING: at net/sched/sch_generic.c:256 dev_watchdog+0x267/0x270()
[ 2384.643042] Hardware name: X8DTH-i/6/iF/6F
[ 2384.647291] NETDEV WATCHDOG: eth0 (igb): transmit queue 0 timed out
[ 2384.653699] Modules linked in: netconsole binfmt_misc ses enclosure mlx4_ib mlx4_en ib_ipoib ib_cm ib_addr ib_sa ib_uverbs ib_umad ib_mad ib_core ipmi_poweroff microcode serio_raw joydev ipmi_watchdog i2c_i801 iTCO_wdt ipmi_devintf iTCO_vendor_support mpt2sas i2c_core scsi_transport_sas ioatdma i7core_edac raid_class mlx4_core igb edac_core dca ipmi_si ipmi_msghandler [last unloaded: netconsole]
[ 2384.692878] Pid: 0, comm: swapper/5 Not tainted 3.3.0-4.fc16.x86_64.debug #1
[ 2384.700062] Call Trace:
[ 2384.702653]  <IRQ>  [<ffffffff81061f6f>] warn_slowpath_common+0x7f/0xc0
[ 2384.709524]  [<ffffffff81062066>] warn_slowpath_fmt+0x46/0x50
[ 2384.715415]  [<ffffffff81585987>] dev_watchdog+0x267/0x270
[ 2384.721052]  [<ffffffff81074645>] run_timer_softirq+0x195/0x6d0
[ 2384.727134]  [<ffffffff810745b0>] ? run_timer_softirq+0x100/0x6d0
[ 2384.733381]  [<ffffffff816a2987>] ? _raw_spin_unlock_irqrestore+0x77/0x80
[ 2384.740311]  [<ffffffff8133360b>] ? debug_object_activate+0x12b/0x1a0
[ 2384.746894]  [<ffffffff81585720>] ? pfifo_fast_init+0x90/0x90
[ 2384.752791]  [<ffffffff8106aef8>] __do_softirq+0xc8/0x3c0
[ 2384.758341]  [<ffffffff81039e7d>] ? lapic_next_event+0x1d/0x30
[ 2384.764323]  [<ffffffff816ad06c>] call_softirq+0x1c/0x30
[ 2384.769785]  [<ffffffff8101c565>] do_softirq+0xa5/0xe0
[ 2384.775065]  [<ffffffff8106b54e>] irq_exit+0xbe/0xf0
[ 2384.780180]  [<ffffffff816ad9ee>] smp_apic_timer_interrupt+0x6e/0x99
[ 2384.786686]  [<ffffffff816ac673>] apic_timer_interrupt+0x73/0x80
[ 2384.792831]  <EOI>  [<ffffffff8137f877>] ? intel_idle+0xe7/0x150
[ 2384.799109]  [<ffffffff8137f87b>] ? intel_idle+0xeb/0x150
[ 2384.804661]  [<ffffffff8137f877>] ? intel_idle+0xe7/0x150
[ 2384.810210]  [<ffffffff8151c377>] cpuidle_idle_call+0xb7/0x5d0
[ 2384.816184]  [<ffffffff816a730d>] ? __atomic_notifier_call_chain+0x4d/0x130
[ 2384.823286]  [<ffffffff8101923f>] cpu_idle+0xdf/0x130
[ 2384.828485]  [<ffffffff8168f64d>] start_secondary+0x2a0/0x2a2
[ 2384.834374] ---[ end trace eb910926dcb78354 ]---
[ 2384.839210] igb 0000:01:00.0: eth0: Reset adapter
[ 2387.622841] igb 0000:01:00.0: eth0: Reset adapter
[ 2390.058543] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 2395.625142] igb 0000:01:00.0: eth0: Reset adapter
[ 2398.609178] igb 0000:01:00.0: eth0: Reset adapter
[ 2401.079872] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 2406.587602] igb 0000:01:00.0: eth0: Reset adapter
[ 2409.700656] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 2419.560643] igb 0000:01:00.0: eth0: Reset adapter
[ 2422.863213] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Comment 14 Josh Boyer 2012-05-01 08:29:39 EDT
*** Bug 817749 has been marked as a duplicate of this bug. ***
Comment 15 Josh Boyer 2012-05-14 10:40:59 EDT
*** Bug 821152 has been marked as a duplicate of this bug. ***
Comment 16 Josh Boyer 2012-06-06 11:27:59 EDT
Moving to rawhide so this doesn't get closed due to F15 EOL
Comment 17 Josh Boyer 2012-07-05 09:12:47 EDT
*** Bug 837711 has been marked as a duplicate of this bug. ***
Comment 18 Josh Boyer 2012-07-12 14:58:17 EDT
*** Bug 839756 has been marked as a duplicate of this bug. ***
Comment 19 Fedora End Of Life 2013-04-03 15:28:16 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
Comment 20 Justin M. Forbes 2013-04-05 12:17:14 EDT
Is this still a problem with 3.9 based F19 kernels?
Comment 21 Michele Baldessari 2013-11-24 06:15:47 EST
Note this BZ is for the ipheth driver, please do not add trace for other 
drivers as these dev_watchdog warnings are completely driver specific.

As to ipheth there is a truckload of reports about this issue:
http://oops.kernel.org/oops/?oopsclass=default&oopstype=default&oopsdistro=default&module=&driver=ipheth&function=&file=&bugline=&oopskernel=&search=submit

No code changes/solution in sight yet though