Bug 1277022

Summary: tg3:WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()
Product: Red Hat Enterprise Linux 7 Reporter: xmu
Component: kernelAssignee: Ivan Vecera <ivecera>
kernel sub component: NIC Drivers QA Contact: Jianlin Shi <jishi>
Status: CLOSED DUPLICATE Docs Contact:
Severity: unspecified    
Priority: unspecified CC: jtoppins, linville, mchan, network-qe, prashant, sanjeevb, siva.kallam
Version: 7.3   
Target Milestone: rc   
Target Release: 7.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-29 16:28:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
This patch will linearize the skb if available descriptors are not sufficient for gso segment none

Description xmu 2015-11-02 05:27:14 UTC 
Description of problem:
WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()

Version-Release number of selected component (if applicable):
RHEL-7.2-20151027.3
3.10.0-326.el7.x86_64

How reproducible:
no clear reproducer
but run the test case:tcp_mss_denial_bz845555 two times, both failed.
both NIC are tg3.

Steps to Reproduce:
TBD

Actual results:
watchdog warning
Checking dmesg for specific failures!
[  997.255541] ------------[ cut here ]------------
[  997.260709] WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()
[  997.268773] NETDEV WATCHDOG: em1 (tg3): transmit queue 0 timed out
[  997.275672] Modules linked in: sctp xprtrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr mlx4_en mlx4_core vxlan ip6_udp_tunnel udp_tunnel rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr cryptd sg iTCO_wdt ipmi_ssif ipmi_devintf iTCO_vendor_support sb_edac dcdbas edac_core mei_me ipmi_si lpc_ich wmi mei ipmi_msghandler shpchp mfd_core acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea
[  997.355122]  sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ahci libahci tg3 crct10dif_pclmul drm crct10dif_common ptp crc32c_intel libata i2c_core pps_core megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[  997.375215] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-326.el7.x86_64 #1
[  997.383279] Hardware name: Dell Inc. PowerEdge R730/0H21J3, BIOS 1.3.6 06/03/2015
[  997.391633]  ffff88046f403d88 9765e51a52c19f40 ffff88046f403d40 ffffffff816351e1
[  997.399929]  ffff88046f403d78 ffffffff8107b200 0000000000000000 ffff880463298000
[  997.408226]  ffff880465503580 0000000000000005 0000000000000000 ffff88046f403de0
[  997.416522] Call Trace:
[  997.419249]  <IRQ>  [<ffffffff816351e1>] dump_stack+0x19/0x1b
[  997.425686]  [<ffffffff8107b200>] warn_slowpath_common+0x70/0xb0
[  997.432393]  [<ffffffff8107b29c>] warn_slowpath_fmt+0x5c/0x80
[  997.438811]  [<ffffffff8154ca80>] dev_watchdog+0x270/0x280
[  997.444935]  [<ffffffff8154c810>] ? dev_graft_qdisc+0x80/0x80
[  997.451351]  [<ffffffff8108b0a6>] call_timer_fn+0x36/0x110
[  997.457475]  [<ffffffff8154c810>] ? dev_graft_qdisc+0x80/0x80
[  997.463890]  [<ffffffff8108dd97>] run_timer_softirq+0x237/0x340
[  997.470500]  [<ffffffff81084b0f>] __do_softirq+0xef/0x280
[  997.476528]  [<ffffffff8164721c>] call_softirq+0x1c/0x30
[  997.482461]  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[  997.488196]  [<ffffffff81084ea5>] irq_exit+0x115/0x120
[  997.493932]  [<ffffffff81647e95>] smp_apic_timer_interrupt+0x45/0x60
[  997.501028]  [<ffffffff8164655d>] apic_timer_interrupt+0x6d/0x80
[  997.507734]  <EOI>  [<ffffffff814d4542>] ? cpuidle_enter_state+0x52/0xc0
[  997.515233]  [<ffffffff814d4538>] ? cpuidle_enter_state+0x48/0xc0
[  997.522036]  [<ffffffff814d4689>] cpuidle_idle_call+0xd9/0x210
[  997.528549]  [<ffffffff8101e4be>] arch_cpu_idle+0xe/0x30
[  997.534481]  [<ffffffff810d6305>] cpu_startup_entry+0x245/0x290
[  997.541092]  [<ffffffff81624df7>] rest_init+0x77/0x80
[  997.546735]  [<ffffffff81a8d057>] start_kernel+0x429/0x44a
[  997.552859]  [<ffffffff81a8ca37>] ? repair_env_string+0x5c/0x5c
[  997.559469]  [<ffffffff81a8c120>] ? early_idt_handlers+0x120/0x120
[  997.566369]  [<ffffffff81a8c5ee>] x86_64_start_reservations+0x2a/0x2c
[  997.573560]  [<ffffffff81a8c742>] x86_64_start_kernel+0x152/0x175
[  997.580361] ---[ end trace 6c8c7a3a4f8b32a6 ]---

End of log.


Expected results:
no watchdog warning

Additional info:
Comment 3 Ivan Vecera 2015-11-04 11:07:44 UTC 
Is it reproducibe with TSO disabled on NIC via ethtool?
Comment 5 Ivan Vecera 2015-11-04 17:58:27 UTC 
Hi Prashant,
it's easily reproducible on RHEL6, RHEL7 as well as on the latest upstream.

Server side:

# ip link set eth0 mtu 128 (eth0 is any kind of NIC)
# echo 88 > /proc/sys/net/ipv4/route/min_adv_mss
# iperf -s

Client side: (tg3 NIC - I have tested 5718 and 5719)

# iperf -i 1 -c <server_ip>
Connecting to host 10.16.47.38, port 5201
[  4] local 10.16.47.94 port 41199 connected to 10.16.47.38 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  37.0 MBytes   310 Mbits/sec    0    195 KBytes       
[  4]   1.00-2.00   sec  32.5 MBytes   273 Mbits/sec    0    195 KBytes       
[  4]   2.00-3.00   sec  33.8 MBytes   283 Mbits/sec    0    195 KBytes       
[  4]   3.00-4.00   sec  27.5 MBytes   231 Mbits/sec    0    195 KBytes       
[  4]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   76.0 Bytes       
[  4]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   76.0 Bytes       
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0   76.0 Bytes       
[  4]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   76.0 Bytes       
[  4]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   76.0 Bytes       
...
[  320.533007] tg3 0000:06:00.0 eno1: transmit timed out, resetting 
[  321.796011] tg3 0000:06:00.0 eno1: 0x00000000: 0x165714e4, 0x00100446, 0x02000001, 0x00800000 
[  321.805518] tg3 0000:06:00.0 eno1: 0x00000010: 0x82a5000c, 0x00000000, 0x82a4000c, 0x00000000 
[  321.815024] tg3 0000:06:00.0 eno1: 0x00000020: 0x82a3000c, 0x00000000, 0x00000000, 0x04911014 
[  321.824529] tg3 0000:06:00.0 eno1: 0x00000030: 0x00000000, 0x00000048, 0x00000000, 0x0000010a 
[  321.834033] tg3 0000:06:00.0 eno1: 0x00000040: 0x00000000, 0x6b000000, 0xc8035001, 0x64002008 
[  321.843540] tg3 0000:06:00.0 eno1: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000 
[  321.853045] tg3 0000:06:00.0 eno1: 0x00000060: 0x00000000, 0x00000000, 0xf0010298, 0x00380081 
[  321.862551] tg3 0000:06:00.0 eno1: 0x00000070: 0x00071090, 0x00000000, 0x00000000, 0x00000000 
[  321.872055] tg3 0000:06:00.0 eno1: 0x00000080: 0x165714e4, 0x40000046, 0x00000000, 0x00000111 
[  321.881561] tg3 0000:06:00.0 eno1: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x000004a7 
[  321.891065] tg3 0000:06:00.0 eno1: 0x000000a0: 0x8010ac11, 0x00000004, 0x00001004, 0x00020010 
[  321.900570] tg3 0000:06:00.0 eno1: 0x000000b0: 0x10008d81, 0x00102406, 0x0004cc42, 0x10220140 
[  321.910074] tg3 0000:06:00.0 eno1: 0x000000d0: 0x0000001f, 0x00000009, 0x00000000, 0x00000001 
[  321.919580] tg3 0000:06:00.0 eno1: 0x000000f0: 0x00000000, 0x05719001, 0x00000000, 0x37dcfdfc 
[  321.929084] tg3 0000:06:00.0 eno1: 0x00000100: 0x13c10001, 0x00000000, 0x00100000, 0x000e7030 
[  321.938589] tg3 0000:06:00.0 eno1: 0x00000110: 0x00002000, 0x000031c0, 0x000000a0, 0x00000000 
[  321.948094] tg3 0000:06:00.0 eno1: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003 
[  321.957599] tg3 0000:06:00.0 eno1: 0x00000140: 0x8b5d6758, 0x00006cae, 0x00000000, 0x00000000 
[  321.967104] tg3 0000:06:00.0 eno1: 0x00000150: 0x16010004, 0x00000000, 0x0007812c, 0x00000001 
[  321.976608] tg3 0000:06:00.0 eno1: 0x00000160: 0x00010002, 0x00000000, 0x00000000, 0x00000000 
[  321.986113] tg3 0000:06:00.0 eno1: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000 
[  321.995619] tg3 0000:06:00.0 eno1: 0x00000200: 0x00000000, 0x6b000000, 0x00000000, 0xc1000000 
[  322.005124] tg3 0000:06:00.0 eno1: 0x00000210: 0x00000000, 0x00000000, 0x00000000, 0x1b000000 
[  322.014628] tg3 0000:06:00.0 eno1: 0x00000220: 0x00000000, 0x60000000, 0x00000000, 0x00000000 
[  322.024133] tg3 0000:06:00.0 eno1: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x000004a7 
[  322.033639] tg3 0000:06:00.0 eno1: 0x00000280: 0x00000000, 0x00000111, 0x00000000, 0x00000299 
[  322.043143] tg3 0000:06:00.0 eno1: 0x00000290: 0x00000000, 0x000002dd, 0x00000000, 0x00000558 
[  322.052647] tg3 0000:06:00.0 eno1: 0x00000300: 0x00000000, 0x00000190, 0x00000000, 0x00000000 
[  322.062162] tg3 0000:06:00.0 eno1: 0x00000400: 0x18e04808, 0x00400000, 0x00001000, 0x00000880 
[  322.071666] tg3 0000:06:00.0 eno1: 0x00000410: 0x00006cae, 0x8b5d6758, 0x00006cae, 0x8b5d6758 
[  322.081181] tg3 0000:06:00.0 eno1: 0x00000420: 0x00006cae, 0x8b5d6758, 0x00006cae, 0x8b5d6758 
[  322.090687] tg3 0000:06:00.0 eno1: 0x00000430: 0x00000400, 0x00000000, 0x0000024b, 0x000005f2 
[  322.100193] tg3 0000:06:00.0 eno1: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04384400 
[  322.109697] tg3 0000:06:00.0 eno1: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102 
[  322.119203] tg3 0000:06:00.0 eno1: 0x00000460: 0x00000008, 0x00002620, 0x01ff0002, 0x00000000 
[  322.128708] tg3 0000:06:00.0 eno1: 0x00000470: 0x80000000, 0x20000000, 0x00000000, 0x40000000 
[  322.138212] tg3 0000:06:00.0 eno1: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff 
[  322.147718] tg3 0000:06:00.0 eno1: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000 
[  322.157223] tg3 0000:06:00.0 eno1: 0x00000540: 0x00006cae, 0x8b5d675c, 0x00000000, 0x00000000 
[  322.166727] tg3 0000:06:00.0 eno1: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000 
[  322.176232] tg3 0000:06:00.0 eno1: 0x000005b0: 0x00000000, 0x00000008, 0x00000000, 0x00000000 
[  322.185737] tg3 0000:06:00.0 eno1: 0x000005c0: 0x7adbc438, 0x7734fa3d, 0x00000000, 0x00000000 
[  322.195242] tg3 0000:06:00.0 eno1: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x00001f04 
[  322.204747] tg3 0000:06:00.0 eno1: 0x00000610: 0xffffffff, 0x00000000, 0x07c00004, 0x00000000 
[  322.214251] tg3 0000:06:00.0 eno1: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000 
[  322.223756] tg3 0000:06:00.0 eno1: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123 
[  322.233261] tg3 0000:06:00.0 eno1: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123 
[  322.242767] tg3 0000:06:00.0 eno1: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123 
[  322.252271] tg3 0000:06:00.0 eno1: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123 
[  322.261776] tg3 0000:06:00.0 eno1: 0x00000670: 0x5f865437, 0xe4ac62cc, 0x50103a45, 0x36621985 
[  322.271281] tg3 0000:06:00.0 eno1: 0x00000680: 0xbf14c0e8, 0x1bc27a1e, 0x84f4b556, 0x094ea6fe 
[  322.280787] tg3 0000:06:00.0 eno1: 0x00000690: 0x7dda01e7, 0xc04d7481, 0x00000000, 0x00000000 
[  322.290291] tg3 0000:06:00.0 eno1: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000 
[  322.299795] tg3 0000:06:00.0 eno1: 0x00000800: 0x00001392, 0xffffffff, 0x00000000, 0x00000000 
[  322.309300] tg3 0000:06:00.0 eno1: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000 
[  322.318804] tg3 0000:06:00.0 eno1: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000 
[  322.328309] tg3 0000:06:00.0 eno1: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.337814] tg3 0000:06:00.0 eno1: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.347319] tg3 0000:06:00.0 eno1: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.356823] tg3 0000:06:00.0 eno1: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x0000000f 
[  322.366327] tg3 0000:06:00.0 eno1: 0x00000880: 0x00000772, 0x000094ef, 0x00000000, 0x0000000f 
[  322.375833] tg3 0000:06:00.0 eno1: 0x00000890: 0x00000002, 0x00000002, 0x00000000, 0x00000000 
[  322.385338] tg3 0000:06:00.0 eno1: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000 
[  322.394843] tg3 0000:06:00.0 eno1: 0x00000900: 0x3455a452, 0xffffffff, 0x00000000, 0x00000000 
[  322.404348] tg3 0000:06:00.0 eno1: 0x00000910: 0x00000000, 0xffffffff, 0x00000000, 0x00000000 
[  322.413853] tg3 0000:06:00.0 eno1: 0x00000920: 0x00000000, 0x00000000, 0xffffffff, 0x00000000 
[  322.423358] tg3 0000:06:00.0 eno1: 0x00000930: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.432863] tg3 0000:06:00.0 eno1: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.442368] tg3 0000:06:00.0 eno1: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.451872] tg3 0000:06:00.0 eno1: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x005bc1ad 
[  322.461377] tg3 0000:06:00.0 eno1: 0x00000970: 0x00000009, 0x00000004, 0x00000000, 0x00000000 
[  322.470881] tg3 0000:06:00.0 eno1: 0x00000980: 0x00000c96, 0x000094ef, 0x00000000, 0x00127aac 
[  322.480386] tg3 0000:06:00.0 eno1: 0x00000990: 0x000002a4, 0x000001bc, 0x00000000, 0x00000000 
[  322.489891] tg3 0000:06:00.0 eno1: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001 
[  322.499396] tg3 0000:06:00.0 eno1: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x0d000000 
[  322.508900] tg3 0000:06:00.0 eno1: 0x00000c80: 0x0000296d, 0x00000000, 0x00000000, 0x00000000 
[  322.518405] tg3 0000:06:00.0 eno1: 0x00000ce0: 0x25baacb4, 0x00000008, 0x0072004c, 0x000c062c 
[  322.527910] tg3 0000:06:00.0 eno1: 0x00000cf0: 0x00000000, 0x40000010, 0x404c0072, 0x00000000 
[  322.537416] tg3 0000:06:00.0 eno1: 0x00001000: 0x00000002, 0x00000000, 0xa0008a8b, 0x00000000 
[  322.546920] tg3 0000:06:00.0 eno1: 0x00001010: 0x01901901, 0x00018a8b, 0x00000000, 0x00000000 
[  322.556426] tg3 0000:06:00.0 eno1: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000000 
[  322.565942] tg3 0000:06:00.0 eno1: 0x00001440: 0x00000190, 0x00000190, 0x00000190, 0x00000190 
[  322.575447] tg3 0000:06:00.0 eno1: 0x00001450: 0x00000190, 0x00000190, 0x00000190, 0x00000190 
[  322.584952] tg3 0000:06:00.0 eno1: 0x00001460: 0x00000190, 0x00000190, 0x00000190, 0x00000190 
[  322.594457] tg3 0000:06:00.0 eno1: 0x00001470: 0x00000190, 0x00000190, 0x00000190, 0x00000190 
[  322.603961] tg3 0000:06:00.0 eno1: 0x00001480: 0x00001111, 0x00000000, 0x00000000, 0x00000000 
[  322.613466] tg3 0000:06:00.0 eno1: 0x00001800: 0x00000016, 0x00000000, 0x00000190, 0x00000000 
[  322.622981] tg3 0000:06:00.0 eno1: 0x00001830: 0x00000000, 0x00000000, 0x00000000, 0x4ec80000 
[  322.632485] tg3 0000:06:00.0 eno1: 0x00001840: 0x4ec80000, 0x00000008, 0x00000201, 0xc0000000 
[  322.641989] tg3 0000:06:00.0 eno1: 0x00001850: 0x0000001f, 0x00000000, 0x00004010, 0x01900190 
[  322.651493] tg3 0000:06:00.0 eno1: 0x00001860: 0x0f000000, 0x00000000, 0x4ec81810, 0x00000008 
[  322.660997] tg3 0000:06:00.0 eno1: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000 
[  322.670503] tg3 0000:06:00.0 eno1: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000 
[  322.680009] tg3 0000:06:00.0 eno1: 0x00002010: 0x00000181, 0x00000001, 0x00780003, 0x00000000 
[  322.689514] tg3 0000:06:00.0 eno1: 0x00002100: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.699018] tg3 0000:06:00.0 eno1: 0x00002110: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.708523] tg3 0000:06:00.0 eno1: 0x00002120: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.718028] tg3 0000:06:00.0 eno1: 0x00002130: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.727533] tg3 0000:06:00.0 eno1: 0x00002140: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.737037] tg3 0000:06:00.0 eno1: 0x00002150: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.746542] tg3 0000:06:00.0 eno1: 0x00002160: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.756047] tg3 0000:06:00.0 eno1: 0x00002170: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.765552] tg3 0000:06:00.0 eno1: 0x00002180: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.775058] tg3 0000:06:00.0 eno1: 0x00002190: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.784563] tg3 0000:06:00.0 eno1: 0x000021a0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.794069] tg3 0000:06:00.0 eno1: 0x000021b0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.803575] tg3 0000:06:00.0 eno1: 0x000021c0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.813091] tg3 0000:06:00.0 eno1: 0x000021d0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.822595] tg3 0000:06:00.0 eno1: 0x000021e0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.832100] tg3 0000:06:00.0 eno1: 0x000021f0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.841605] tg3 0000:06:00.0 eno1: 0x00002200: 0x001280ca, 0x00000000, 0x00000000, 0x00000000 
[  322.851109] tg3 0000:06:00.0 eno1: 0x00002400: 0x00010012, 0x00000000, 0x00400001, 0x00000000 
[  322.860614] tg3 0000:06:00.0 eno1: 0x00002410: 0x0000000f, 0x00005d00, 0x00000000, 0x00000000 
[  322.870119] tg3 0000:06:00.0 eno1: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00044400 
[  322.879625] tg3 0000:06:00.0 eno1: 0x00002450: 0x00000008, 0x51560000, 0x08001800, 0x00040000 
[  322.889130] tg3 0000:06:00.0 eno1: 0x00002470: 0x00000000, 0x000003df, 0x00000000, 0x00000000 
[  322.898634] tg3 0000:06:00.0 eno1: 0x00002500: 0x00000000, 0x00000000, 0x00000002, 0x00044800 
[  322.908138] tg3 0000:06:00.0 eno1: 0x00002510: 0x00000000, 0x00000000, 0x00000002, 0x00040400 
[  322.917644] tg3 0000:06:00.0 eno1: 0x00002520: 0x00000000, 0x00000000, 0x00000002, 0x00044c00 
[  322.927148] tg3 0000:06:00.0 eno1: 0x00002530: 0x00000000, 0x00000000, 0x00000002, 0x00040800 
[  322.936654] tg3 0000:06:00.0 eno1: 0x00002540: 0x00000000, 0x00000000, 0x00000002, 0x00045000 
[  322.946159] tg3 0000:06:00.0 eno1: 0x00002550: 0x00000000, 0x00000000, 0x00000002, 0x00040c00 
[  322.955665] tg3 0000:06:00.0 eno1: 0x00002560: 0x00000000, 0x00000000, 0x00000002, 0x00045400 
[  322.965170] tg3 0000:06:00.0 eno1: 0x00002570: 0x00000000, 0x00000000, 0x00000002, 0x00041000 
[  322.974675] tg3 0000:06:00.0 eno1: 0x00002580: 0x00000000, 0x00000000, 0x00000002, 0x00045800 
[  322.984180] tg3 0000:06:00.0 eno1: 0x00002590: 0x00000000, 0x00000000, 0x00000002, 0x00041400 
[  322.993685] tg3 0000:06:00.0 eno1: 0x000025a0: 0x00000000, 0x00000000, 0x00000002, 0x00045c00 
[  323.003190] tg3 0000:06:00.0 eno1: 0x000025b0: 0x00000000, 0x00000000, 0x00000002, 0x00041800 
[  323.012694] tg3 0000:06:00.0 eno1: 0x000025c0: 0x00000000, 0x00000000, 0x00000002, 0x00046000 
[  323.022200] tg3 0000:06:00.0 eno1: 0x000025d0: 0x00000000, 0x00000000, 0x00000002, 0x00041c00 
[  323.031705] tg3 0000:06:00.0 eno1: 0x000025e0: 0x00000000, 0x00000000, 0x00000002, 0x00046400 
[  323.041209] tg3 0000:06:00.0 eno1: 0x000025f0: 0x00000000, 0x00000000, 0x00000002, 0x00042000 
[  323.050714] tg3 0000:06:00.0 eno1: 0x00002600: 0x00000000, 0x00000000, 0x00000002, 0x00046800 
[  323.060219] tg3 0000:06:00.0 eno1: 0x00002610: 0x00000000, 0x00000000, 0x00000002, 0x00042400 
[  323.069724] tg3 0000:06:00.0 eno1: 0x00002620: 0x00000000, 0x00000000, 0x00000002, 0x00046c00 
[  323.079229] tg3 0000:06:00.0 eno1: 0x00002630: 0x00000000, 0x00000000, 0x00000002, 0x00042800 
[  323.088733] tg3 0000:06:00.0 eno1: 0x00002640: 0x00000000, 0x00000000, 0x00000002, 0x00047000 
[  323.098239] tg3 0000:06:00.0 eno1: 0x00002650: 0x00000000, 0x00000000, 0x00000002, 0x00042c00 
[  323.107744] tg3 0000:06:00.0 eno1: 0x00002660: 0x00000000, 0x00000000, 0x00000002, 0x00047400 
[  323.117250] tg3 0000:06:00.0 eno1: 0x00002670: 0x00000000, 0x00000000, 0x00000002, 0x00043000 
[  323.126754] tg3 0000:06:00.0 eno1: 0x00002680: 0x00000000, 0x00000000, 0x00000002, 0x00047800 
[  323.136267] tg3 0000:06:00.0 eno1: 0x00002690: 0x00000000, 0x00000000, 0x00000002, 0x00043400 
[  323.145773] tg3 0000:06:00.0 eno1: 0x000026a0: 0x00000000, 0x00000000, 0x00000002, 0x00047c00 
[  323.155278] tg3 0000:06:00.0 eno1: 0x000026b0: 0x00000000, 0x00000000, 0x00000002, 0x00043800 
[  323.164783] tg3 0000:06:00.0 eno1: 0x000026c0: 0x00000000, 0x00000000, 0x00000002, 0x00048000 
[  323.174289] tg3 0000:06:00.0 eno1: 0x000026d0: 0x00000000, 0x00000000, 0x00000002, 0x00043c00 
[  323.183794] tg3 0000:06:00.0 eno1: 0x000026e0: 0x00000000, 0x00000000, 0x00000002, 0x00048400 
[  323.193299] tg3 0000:06:00.0 eno1: 0x000026f0: 0x00000000, 0x00000000, 0x00000002, 0x00044000 
[  323.202803] tg3 0000:06:00.0 eno1: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000 
[  323.212308] tg3 0000:06:00.0 eno1: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x0000045f 
[  323.221813] tg3 0000:06:00.0 eno1: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c 
[  323.231319] tg3 0000:06:00.0 eno1: 0x00002c20: 0x00000001, 0x00000000, 0x00000000, 0x00000000 
[  323.240824] tg3 0000:06:00.0 eno1: 0x00002d00: 0x00000080, 0x00000040, 0x00000000, 0x00000000 
[  323.250330] tg3 0000:06:00.0 eno1: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x0000045f 
[  323.259835] tg3 0000:06:00.0 eno1: 0x00003400: 0x00000006, 0x00000000, 0x00000000, 0x00000000 
[  323.269340] tg3 0000:06:00.0 eno1: 0x00003600: 0x00000000, 0x00170000, 0x00110000, 0x00000000 
[  323.278846] tg3 0000:06:00.0 eno1: 0x00003610: 0x00130000, 0x00000000, 0x00130000, 0x00000011 
[  323.288351] tg3 0000:06:00.0 eno1: 0x00003620: 0x00110011, 0x00000000, 0x00000000, 0x00032080 
[  323.297855] tg3 0000:06:00.0 eno1: 0x00003630: 0x00000000, 0x87748774, 0x02c01000, 0x00000000 
[  323.307360] tg3 0000:06:00.0 eno1: 0x00003640: 0x00000000, 0x00000000, 0x00000020, 0x00000019 
[  323.316865] tg3 0000:06:00.0 eno1: 0x00003650: 0x00000171, 0x000f03ff, 0x05719001, 0x00000000 
[  323.326370] tg3 0000:06:00.0 eno1: 0x00003660: 0x00000000, 0x00000000, 0x02000000, 0x00000202 
[  323.335875] tg3 0000:06:00.0 eno1: 0x00003670: 0x00000000, 0xfefffff7, 0x00000000, 0x00000000 
[  323.345380] tg3 0000:06:00.0 eno1: 0x00003680: 0x30018010, 0x00000000, 0x00000000, 0x00000000 
[  323.354886] tg3 0000:06:00.0 eno1: 0x000036a0: 0x000001a0, 0x00000000, 0x00000000, 0x00000000 
[  323.364392] tg3 0000:06:00.0 eno1: 0x000036b0: 0x0010034c, 0x07ff07ff, 0x07ff07ff, 0x01000004 
[  323.373897] tg3 0000:06:00.0 eno1: 0x000036c0: 0xffffffff, 0x00000000, 0x00000000, 0x0011357e 
[  323.383403] tg3 0000:06:00.0 eno1: 0x000036d0: 0x0000019d, 0x00000000, 0x00000000, 0x0000493f 
[  323.392907] tg3 0000:06:00.0 eno1: 0x000036f0: 0x00000000, 0x00000000, 0x00000000, 0x00012b01 
[  323.402412] tg3 0000:06:00.0 eno1: 0x00003800: 0x00000001, 0x00000000, 0x0000000e, 0x0516028b 
[  323.411918] tg3 0000:06:00.0 eno1: 0x00003810: 0x00000024, 0x000000ed, 0x00000000, 0x00000000 
[  323.421422] tg3 0000:06:00.0 eno1: 0x00003c00: 0x00000302, 0x00000000, 0x00000000, 0x00000048 
[  323.430927] tg3 0000:06:00.0 eno1: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000 
[  323.440432] tg3 0000:06:00.0 eno1: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000 
[  323.449937] tg3 0000:06:00.0 eno1: 0x00003c30: 0x00000000, 0x00000000, 0x00000008, 0x4ec9b000 
[  323.459441] tg3 0000:06:00.0 eno1: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000 
[  323.468946] tg3 0000:06:00.0 eno1: 0x00003c50: 0x00000000, 0x000003df, 0x00000000, 0x00000000 
[  323.478450] tg3 0000:06:00.0 eno1: 0x00003c80: 0x00000111, 0x00000299, 0x000002dd, 0x00000558 
[  323.487955] tg3 0000:06:00.0 eno1: 0x00003cc0: 0x00000190, 0x00000000, 0x00000000, 0x00000000 
[  323.497460] tg3 0000:06:00.0 eno1: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000 
[  323.506966] tg3 0000:06:00.0 eno1: 0x00003d00: 0x00000008, 0x4ec88000, 0x00000008, 0x4ec89000 
[  323.516471] tg3 0000:06:00.0 eno1: 0x00003d10: 0x00000008, 0x4ec8a000, 0x00000008, 0x4ec8b000 
[  323.525976] tg3 0000:06:00.0 eno1: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000 
[  323.535482] tg3 0000:06:00.0 eno1: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000 
[  323.544987] tg3 0000:06:00.0 eno1: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000 
[  323.554493] tg3 0000:06:00.0 eno1: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000 
[  323.563997] tg3 0000:06:00.0 eno1: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000 
[  323.573501] tg3 0000:06:00.0 eno1: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000 
[  323.583007] tg3 0000:06:00.0 eno1: 0x00003fc0: 0x0003dfa7, 0x00000000, 0x00000000, 0x00000000 
[  323.592514] tg3 0000:06:00.0 eno1: 0x00004000: 0x00000002, 0x00000000, 0x000eb75f, 0x000d88e1 
[  323.602019] tg3 0000:06:00.0 eno1: 0x00004010: 0x00000000, 0x002f2012, 0x00000480, 0x00857c42 
[  323.611524] tg3 0000:06:00.0 eno1: 0x00004020: 0x00000000, 0x00000000, 0x00000010, 0x00000000 
[  323.621028] tg3 0000:06:00.0 eno1: 0x00004030: 0x00000010, 0x00000050, 0x00000000, 0x00000000 
[  323.630534] tg3 0000:06:00.0 eno1: 0x00004040: 0x00000000, 0x00000000, 0x01081e20, 0x00000000 
[  323.640039] tg3 0000:06:00.0 eno1: 0x00004050: 0x00000000, 0x00000000, 0x002dd010, 0x0048b002 
[  323.649542] tg3 0000:06:00.0 eno1: 0x00004060: 0x0048c000, 0x00000000, 0x00000000, 0x00000000 
[  323.659048] tg3 0000:06:00.0 eno1: 0x00004400: 0x80000006, 0x00000000, 0x00010000, 0x0000a000 
[  323.668552] tg3 0000:06:00.0 eno1: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000 
[  323.678057] tg3 0000:06:00.0 eno1: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000 
[  323.687563] tg3 0000:06:00.0 eno1: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x02004c8d 
[  323.697068] tg3 0000:06:00.0 eno1: 0x00004450: 0x0002053f, 0x00f300f4, 0x00000000, 0x00000000 
[  323.706573] tg3 0000:06:00.0 eno1: 0x00004800: 0x180303fe, 0x00000000, 0x00000000, 0x00000020 
[  323.716079] tg3 0000:06:00.0 eno1: 0x00004810: 0x00000000, 0x0000000b, 0x004c9c78, 0x00002a28 
[  323.725594] tg3 0000:06:00.0 eno1: 0x00004820: 0x0000008c, 0x00000000, 0xf0330000, 0x25baad00 
[  323.735100] tg3 0000:06:00.0 eno1: 0x00004830: 0x00000000, 0x0000003d, 0x0000003d, 0x00000000 
[  323.744604] tg3 0000:06:00.0 eno1: 0x00004840: 0x005bb791, 0x005bb791, 0x000e2200, 0x8c42ec00 
[  323.754108] tg3 0000:06:00.0 eno1: 0x00004850: 0x2996dd3b, 0x7779008e, 0x8b8a8b8a, 0x00000000 
[  323.763614] tg3 0000:06:00.0 eno1: 0x00004860: 0x0000008c, 0x112fda93, 0x00000800, 0x7a000000 
[  323.773119] tg3 0000:06:00.0 eno1: 0x00004870: 0x00000080, 0x00000000, 0x00000000, 0x00000000 
[  323.782624] tg3 0000:06:00.0 eno1: 0x00004900: 0x00090404, 0x00305407, 0x00000000, 0x00000000 
[  323.792127] tg3 0000:06:00.0 eno1: 0x00004910: 0x000f001c, 0x00000000, 0x00000000, 0x00000000 
[  323.801632] tg3 0000:06:00.0 eno1: 0x00004a00: 0x180303fe, 0x00200000, 0x00200020, 0x7bc70000 
[  323.811138] tg3 0000:06:00.0 eno1: 0x00004a10: 0x51568b80, 0x008c5914, 0x00200011, 0x00000000 
[  323.820643] tg3 0000:06:00.0 eno1: 0x00004a20: 0x00000000, 0x00000000, 0xf02c0000, 0x51568be0 
[  323.830149] tg3 0000:06:00.0 eno1: 0x00004a30: 0x00000000, 0x0000009c, 0x0000009c, 0x00000000 
[  323.839654] tg3 0000:06:00.0 eno1: 0x00004a40: 0x51568b80, 0x51568b20, 0x51568b40, 0x51568b60 
[  323.849159] tg3 0000:06:00.0 eno1: 0x00004a50: 0x00200020, 0x00200020, 0x00200020, 0x00200020 
[  323.858664] tg3 0000:06:00.0 eno1: 0x00004a70: 0x00090404, 0x00305407, 0x000f001c, 0x00000000 
[  323.868169] tg3 0000:06:00.0 eno1: 0x00004b00: 0x180303fe, 0x04890003, 0x30000000, 0x00560020 
[  323.877676] tg3 0000:06:00.0 eno1: 0x00004b10: 0x00560040, 0x002a0003, 0x0489dc70, 0x00000000 
[  323.887182] tg3 0000:06:00.0 eno1: 0x00004b20: 0x00000091, 0x02480000, 0x51b664c1, 0x0f890088 
[  323.896687] tg3 0000:06:00.0 eno1: 0x00004b30: 0x2f7facb9, 0x0f560058, 0x2f7fe0b9, 0x0f560058 
[  323.906192] tg3 0000:06:00.0 eno1: 0x00004b40: 0x503e3801, 0x0f2a0030, 0xc0c07e7e, 0xa2a2ecec 
[  323.915696] tg3 0000:06:00.0 eno1: 0x00004b50: 0xf0330000, 0x51b66548, 0xaf490000, 0x91900090 
[  323.925201] tg3 0000:06:00.0 eno1: 0x00004b60: 0xf0330000, 0x2f7fad10, 0xaf060000, 0x40000195 
[  323.934705] tg3 0000:06:00.0 eno1: 0x00004b70: 0xf0330000, 0x2f7fe110, 0xaf060000, 0x000000ff 
[  323.944210] tg3 0000:06:00.0 eno1: 0x00004b80: 0x00000091, 0x112fda93, 0x00000800, 0x7a000000 
[  323.953715] tg3 0000:06:00.0 eno1: 0x00004b90: 0x00000080, 0x00090404, 0x00305407, 0x000f001c 
[  323.963221] tg3 0000:06:00.0 eno1: 0x00004ba0: 0x00f00024, 0x00000000, 0x00000000, 0x00000000 
[  323.972725] tg3 0000:06:00.0 eno1: 0x00004bb0: 0x51b662c1, 0x2f7fccb1, 0x2f7fccb9, 0x521e4cc1 
[  323.982231] tg3 0000:06:00.0 eno1: 0x00004bc0: 0x51b660ba, 0x2f7facba, 0x2f7fe0ba, 0x503e3802 
[  323.991736] tg3 0000:06:00.0 eno1: 0x00004bd0: 0x521d60ce, 0x2f7fccb2, 0x2f7fccba, 0x521e48ba 
[  324.001240] tg3 0000:06:00.0 eno1: 0x00004be0: 0x00560489, 0x002a0056, 0x005e0042, 0x05ea0056 
[  324.010745] tg3 0000:06:00.0 eno1: 0x00004bf0: 0xf0330000, 0x503e3830, 0xaf030000, 0x0000ffff 
[  324.020251] tg3 0000:06:00.0 eno1: 0x00004c00: 0x200003fe, 0x00000000, 0x00000000, 0x00000000 
[  324.029755] tg3 0000:06:00.0 eno1: 0x00004c10: 0x00000015, 0x00000000, 0x00000006, 0x00000000 
[  324.039261] tg3 0000:06:00.0 eno1: 0x00004c20: 0x00000000, 0x00000000, 0x00000000, 0x00000006 
[  324.048765] tg3 0000:06:00.0 eno1: 0x00004c30: 0x00000000, 0x00a64000, 0x000000fb, 0x000000fb 
[  324.058269] tg3 0000:06:00.0 eno1: 0x00004c40: 0x00000000, 0x51a22200, 0x001d0020, 0x00140020 
[  324.067775] tg3 0000:06:00.0 eno1: 0x00004c50: 0xdc298110, 0x00110557, 0x582dd299, 0x005f6b6b 
[  324.077281] tg3 0000:06:00.0 eno1: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000 
[  324.086785] tg3 0000:06:00.0 eno1: 0x00005000: 0x00009800, 0x80004000, 0x00000000, 0x00000000 
[  324.096289] tg3 0000:06:00.0 eno1: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08002178 
[  324.105794] tg3 0000:06:00.0 eno1: 0x00005020: 0xafa40014, 0x00000000, 0x00000000, 0x40000020 
[  324.115298] tg3 0000:06:00.0 eno1: 0x00005030: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.124803] tg3 0000:06:00.0 eno1: 0x00005040: 0x00000000, 0x00000000, 0x080021be, 0x00000000 
[  324.134307] tg3 0000:06:00.0 eno1: 0x00005080: 0x00009800, 0x80004000, 0x00000000, 0x00000000 
[  324.143813] tg3 0000:06:00.0 eno1: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x08002110 
[  324.153318] tg3 0000:06:00.0 eno1: 0x000050a0: 0xaea02e48, 0x00000000, 0x00000000, 0x40000020 
[  324.162823] tg3 0000:06:00.0 eno1: 0x000050b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.172329] tg3 0000:06:00.0 eno1: 0x000050c0: 0x00000000, 0x00000000, 0x08002194, 0x00000000 
[  324.181834] tg3 0000:06:00.0 eno1: 0x00005100: 0x00009800, 0x80004000, 0x00000000, 0x00000000 
[  324.191338] tg3 0000:06:00.0 eno1: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08001bbc 
[  324.200843] tg3 0000:06:00.0 eno1: 0x00005120: 0xaea02e48, 0x00000000, 0x00000000, 0x40000020 
[  324.210347] tg3 0000:06:00.0 eno1: 0x00005130: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.219852] tg3 0000:06:00.0 eno1: 0x00005140: 0x00000000, 0x00000000, 0x08001bde, 0x00000000 
[  324.229357] tg3 0000:06:00.0 eno1: 0x00005180: 0x00009800, 0x80008000, 0x00000000, 0x00000000 
[  324.238862] tg3 0000:06:00.0 eno1: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x08002dc8 
[  324.248365] tg3 0000:06:00.0 eno1: 0x000051a0: 0x14600011, 0x00000000, 0x00000000, 0x40000020 
[  324.257871] tg3 0000:06:00.0 eno1: 0x000051b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.267375] tg3 0000:06:00.0 eno1: 0x000051c0: 0x00000000, 0x00000000, 0x08001bde, 0x00000000 
[  324.276880] tg3 0000:06:00.0 eno1: 0x00005200: 0x00000000, 0x00000000, 0x02c9f5c5, 0x146001a5 
[  324.286386] tg3 0000:06:00.0 eno1: 0x00005210: 0x00000000, 0x00000000, 0x08000000, 0x08002e00 
[  324.295890] tg3 0000:06:00.0 eno1: 0x00005220: 0x00000000, 0xb49a89ab, 0x00000000, 0x08006f98 
[  324.305395] tg3 0000:06:00.0 eno1: 0x00005230: 0x06000000, 0x01020000, 0x00000015, 0x00000000 
[  324.314900] tg3 0000:06:00.0 eno1: 0x00005240: 0xffffffff, 0x146001a5, 0x00000000, 0x00000000 
[  324.324406] tg3 0000:06:00.0 eno1: 0x00005250: 0xc0010000, 0x08006f98, 0x08006f80, 0xb49a89ab 
[  324.333910] tg3 0000:06:00.0 eno1: 0x00005260: 0x08000000, 0x02c9f5c5, 0x146001a5, 0x00000000 
[  324.343414] tg3 0000:06:00.0 eno1: 0x00005270: 0x00000000, 0x08000000, 0x08006f90, 0x00000000 
[  324.352919] tg3 0000:06:00.0 eno1: 0x00005280: 0x00009800, 0x80004000, 0x00000000, 0x00000000 
[  324.362425] tg3 0000:06:00.0 eno1: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x08001974 
[  324.371930] tg3 0000:06:00.0 eno1: 0x000052a0: 0x30632000, 0x00000000, 0x00000000, 0x40000020 
[  324.381434] tg3 0000:06:00.0 eno1: 0x000052b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.390939] tg3 0000:06:00.0 eno1: 0x000052c0: 0x00000000, 0x00000000, 0x08002868, 0x00000000 
[  324.400444] tg3 0000:06:00.0 eno1: 0x00005300: 0x00009800, 0x80000000, 0x00000000, 0x00000000 
[  324.409949] tg3 0000:06:00.0 eno1: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x08002110 
[  324.419455] tg3 0000:06:00.0 eno1: 0x00005320: 0x14400005, 0x00000000, 0x00000000, 0x40000020 
[  324.428960] tg3 0000:06:00.0 eno1: 0x00005330: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.438465] tg3 0000:06:00.0 eno1: 0x00005340: 0x00000000, 0x00000000, 0x08001986, 0x00000000 
[  324.447970] tg3 0000:06:00.0 eno1: 0x00005380: 0x00009800, 0x80004000, 0x00000000, 0x00000000 
[  324.457475] tg3 0000:06:00.0 eno1: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x08001ac0 
[  324.466980] tg3 0000:06:00.0 eno1: 0x000053a0: 0x00031f82, 0x00000000, 0x00000000, 0x40000020 
[  324.476485] tg3 0000:06:00.0 eno1: 0x000053b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.485989] tg3 0000:06:00.0 eno1: 0x000053c0: 0x00000000, 0x00000000, 0x08001bde, 0x00000000 
[  324.495496] tg3 0000:06:00.0 eno1: 0x00005800: 0x6b000000, 0x6b000000, 0xc1000000, 0x00000000 
[  324.505001] tg3 0000:06:00.0 eno1: 0x00005810: 0x00000000, 0x00000000, 0x1b000000, 0x00000000 
[  324.514507] tg3 0000:06:00.0 eno1: 0x00005820: 0x60000000, 0x00000000, 0x00000000, 0x00000000 
[  324.524011] tg3 0000:06:00.0 eno1: 0x00005860: 0x00000000, 0x00000000, 0x000004a7, 0x000004a7 
[  324.533515] tg3 0000:06:00.0 eno1: 0x00005880: 0x00000111, 0x00000111, 0x00000299, 0x00000299 
[  324.543020] tg3 0000:06:00.0 eno1: 0x00005890: 0x000002dd, 0x000002dd, 0x00000558, 0x00000558 
[  324.552525] tg3 0000:06:00.0 eno1: 0x00005900: 0x00000190, 0x00000190, 0x00000000, 0x00000000 
[  324.562029] tg3 0000:06:00.0 eno1: 0x00005980: 0x00000190, 0x00000000, 0x00000000, 0x00000000 
[  324.571534] tg3 0000:06:00.0 eno1: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000 
[  324.581039] tg3 0000:06:00.0 eno1: 0x00006000: 0x00010082, 0x00000000, 0x00000000, 0x00000000 
[  324.590545] tg3 0000:06:00.0 eno1: 0x00006400: 0x00000000, 0x00000000, 0x00010091, 0xc0000000 
[  324.600051] tg3 0000:06:00.0 eno1: 0x00006410: 0x0a000064, 0x0a000064, 0x00000000, 0x00000000 
[  324.609555] tg3 0000:06:00.0 eno1: 0x00006430: 0x00000000, 0x14e41657, 0x04911014, 0x01020000 
[  324.619061] tg3 0000:06:00.0 eno1: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000000 
[  324.628566] tg3 0000:06:00.0 eno1: 0x000064c0: 0x00000010, 0x00000004, 0x00001004, 0x00000000 
[  324.638070] tg3 0000:06:00.0 eno1: 0x000064d0: 0x00000000, 0x10008d81, 0x00000000, 0x00315e42 
[  324.647586] tg3 0000:06:00.0 eno1: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000 
[  324.657090] tg3 0000:06:00.0 eno1: 0x000064f0: 0x00000002, 0x00000031, 0x00000000, 0x00000000 
[  324.666594] tg3 0000:06:00.0 eno1: 0x00006500: 0x01e10003, 0x8b5d6758, 0x00006cae, 0x00000003 
[  324.676100] tg3 0000:06:00.0 eno1: 0x00006510: 0x0007812c, 0x00058116, 0x0004610a, 0x00000000 
[  324.685604] tg3 0000:06:00.0 eno1: 0x00006550: 0x00000001, 0x02800000, 0x00000000, 0x00000000 
[  324.695109] tg3 0000:06:00.0 eno1: 0x000065f0: 0x00000000, 0x00000109, 0x00000000, 0x00000000 
[  324.704615] tg3 0000:06:00.0 eno1: 0x00006800: 0x041b0034, 0x20089082, 0x01c09208, 0x12200c45 
[  324.714119] tg3 0000:06:00.0 eno1: 0x00006810: 0x01020000, 0xffffffff, 0x00000000, 0x00000000 
[  324.723624] tg3 0000:06:00.0 eno1: 0x00006830: 0xffffffff, 0xffffffff, 0x00000000, 0x00000000 
[  324.733129] tg3 0000:06:00.0 eno1: 0x00006840: 0x00000000, 0x00000001, 0x00000000, 0x00000000 
[  324.742635] tg3 0000:06:00.0 eno1: 0x00006890: 0x00000000, 0x88003800, 0x00000000, 0x04102040 
[  324.752139] tg3 0000:06:00.0 eno1: 0x000068a0: 0x00000020, 0x00000001, 0x03ff03ff, 0x00000000 
[  324.761645] tg3 0000:06:00.0 eno1: 0x000068b0: 0xe0011514, 0x00000000, 0x00000000, 0x00000000 
[  324.771149] tg3 0000:06:00.0 eno1: 0x000068f0: 0x00ff000e, 0x00ff0000, 0x00000000, 0x04444444 
[  324.780654] tg3 0000:06:00.0 eno1: 0x00006900: 0x895e9870, 0x14138fbf, 0x00000000, 0x00000000 
[  324.790160] tg3 0000:06:00.0 eno1: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000 
[  324.799666] tg3 0000:06:00.0 eno1: 0x00007000: 0x00000008, 0x00000000, 0x00000000, 0x00004a8c 
[  324.809180] tg3 0000:06:00.0 eno1: 0x00007010: 0x9ed97dea, 0x02408003, 0x000500db, 0x03000a00 
[  324.818687] tg3 0000:06:00.0 eno1: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000 
[  324.828192] tg3 0000:06:00.0 eno1: 0x00007030: 0x00020000, 0x00004a90, 0x000d0030, 0x00000000 
[  324.837699] tg3 0000:06:00.0 eno1: 0: Host status block [00000001:0000006b:(0000:03cf:0000):(0000:0190)] 
[  324.848282] tg3 0000:06:00.0 eno1: 0: NAPI info [0000006b:0000006b:(0190:0190:01ff):0000:(04ad:0000:0000:0000)] 
[  324.859531] tg3 0000:06:00.0 eno1: 1: Host status block [00000001:000000c7:(0000:0000:0000):(0117:0000)] 
[  324.870103] tg3 0000:06:00.0 eno1: 1: NAPI info [000000c7:000000c7:(0000:0000:01ff):0117:(0117:0117:0000:0000)] 
[  324.881352] tg3 0000:06:00.0 eno1: 2: Host status block [00000001:00000000:(0299:0000:0000):(0000:0000)] 
[  324.891922] tg3 0000:06:00.0 eno1: 2: NAPI info [00000000:00000000:(0000:0000:01ff):0299:(0299:0299:0000:0000)] 
[  324.903172] tg3 0000:06:00.0 eno1: 3: Host status block [00000001:0000001b:(0000:0000:0000):(0000:0000)] 
[  324.913742] tg3 0000:06:00.0 eno1: 3: NAPI info [0000001b:0000001b:(0000:0000:01ff):02dd:(02dd:02dd:0000:0000)] 
[  324.924991] tg3 0000:06:00.0 eno1: 4: Host status block [00000001:00000060:(0000:0000:0558):(0000:0000)] 
[  324.935561] tg3 0000:06:00.0 eno1: 4: NAPI info [00000060:00000060:(0000:0000:01ff):0558:(0558:0558:0000:0000)]
Comment 6 Ivan Vecera 2015-11-04 18:01:17 UTC 
Btw. it is reproducible only when TSO is enabled.
Comment 7 xmu 2015-11-05 02:11:47 UTC 
Hi Ivan,

BTW, there are some similar bug has been opened before, they all have the watchdog warning:WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()

e1000e (bz1276596),
igb(1276603),
FCoE ixgbe(1263044), 
i40e(bz1250574)
Comment 8 Ivan Vecera 2015-11-05 07:18:57 UTC 
(In reply to xmu from comment #7)
> Hi Ivan,
> 
> BTW, there are some similar bug has been opened before, they all have the
> watchdog warning:WARNING: at net/sched/sch_generic.c:297
> dev_watchdog+0x270/0x280()
> 
> e1000e (bz1276596),
> igb(1276603),
> FCoE ixgbe(1263044), 
> i40e(bz1250574)

This tx watchdog timeouts are drivers specific bugs. The bugs you mentioned have a different root cause and each should be solved separately.
Comment 9 xmu 2015-11-05 08:14:07 UTC 
(In reply to Ivan Vecera from comment #8)
OK, got it.
Thanks
Comment 10 Prashant 2015-11-23 16:00:54 UTC 
Adding Siva to look at tg3 specific TX timeout.
Comment 11 Ivan Vecera 2015-12-11 17:45:33 UTC 
Siva, do you see any useful in the log?
Comment 12 Ivan Vecera 2015-12-11 17:48:09 UTC 
AFAIK, it is reproducible only when TSO is enabled.
Comment 13 Siva Reddy Kallam 2015-12-12 04:43:57 UTC 
Ivan,

We are able to reproduce this and fix is under review. 
Soon, we will share the patch.
Comment 14 Ivan Vecera 2015-12-15 16:06:21 UTC 
Siva,

thanks a lot. Could you please attach here the proposed fix. I would like to look on it.

Ivan
Comment 15 Siva Reddy Kallam 2015-12-17 10:55:53 UTC 
Created attachment 1106691 [details]
This patch will linearize the skb if available descriptors are not sufficient for gso segment

Here the patch for your review. we will be up-streaming in next batch.
Comment 16 Ivan Vecera 2016-01-15 14:18:56 UTC 
(In reply to Siva Reddy Kallam from comment #15)
> Created attachment 1106691 [details]
> This patch will linearize the skb if available descriptors are not
> sufficient for gso segment
> 
> Here the patch for your review. we will be up-streaming in next batch.

Siva, the patch looks good for me. When are you planning to submit it upstream?
Comment 17 Siva Reddy Kallam 2016-01-18 09:19:50 UTC 
(In reply to Ivan Vecera from comment #16)
> (In reply to Siva Reddy Kallam from comment #15)
> > Created attachment 1106691 [details]
> > This patch will linearize the skb if available descriptors are not
> > sufficient for gso segment
> > 
> > Here the patch for your review. we will be up-streaming in next batch.
> 
> Siva, the patch looks good for me. When are you planning to submit it
> upstream?

We will send the patch to netdev in a week time.
Comment 18 Ivan Vecera 2016-03-29 16:28:42 UTC 

*** This bug has been marked as a duplicate of bug 1258728 ***