RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1277022 - tg3:WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()
Summary: tg3:WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()
Keywords:
Status: CLOSED DUPLICATE of bug 1258728
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 7.3
Assignee: Ivan Vecera
QA Contact: Jianlin Shi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-11-02 05:27 UTC by xmu
Modified: 2016-11-29 18:57 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-03-29 16:28:42 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
This patch will linearize the skb if available descriptors are not sufficient for gso segment (2.79 KB, patch)
2015-12-17 10:55 UTC, Siva Reddy Kallam 		no flags Details | Diff
View All

Description xmu 2015-11-02 05:27:14 UTC 
Description of problem:
WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()

Version-Release number of selected component (if applicable):
RHEL-7.2-20151027.3
3.10.0-326.el7.x86_64

How reproducible:
no clear reproducer
but run the test case:tcp_mss_denial_bz845555 two times, both failed.
both NIC are tg3.

Steps to Reproduce:
TBD

Actual results:
watchdog warning
Checking dmesg for specific failures!
[  997.255541] ------------[ cut here ]------------
[  997.260709] WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()
[  997.268773] NETDEV WATCHDOG: em1 (tg3): transmit queue 0 timed out
[  997.275672] Modules linked in: sctp xprtrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr mlx4_en mlx4_core vxlan ip6_udp_tunnel udp_tunnel rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr cryptd sg iTCO_wdt ipmi_ssif ipmi_devintf iTCO_vendor_support sb_edac dcdbas edac_core mei_me ipmi_si lpc_ich wmi mei ipmi_msghandler shpchp mfd_core acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea
[  997.355122]  sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ahci libahci tg3 crct10dif_pclmul drm crct10dif_common ptp crc32c_intel libata i2c_core pps_core megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[  997.375215] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-326.el7.x86_64 #1
[  997.383279] Hardware name: Dell Inc. PowerEdge R730/0H21J3, BIOS 1.3.6 06/03/2015
[  997.391633]  ffff88046f403d88 9765e51a52c19f40 ffff88046f403d40 ffffffff816351e1
[  997.399929]  ffff88046f403d78 ffffffff8107b200 0000000000000000 ffff880463298000
[  997.408226]  ffff880465503580 0000000000000005 0000000000000000 ffff88046f403de0
[  997.416522] Call Trace:
[  997.419249]  <IRQ>  [<ffffffff816351e1>] dump_stack+0x19/0x1b
[  997.425686]  [<ffffffff8107b200>] warn_slowpath_common+0x70/0xb0
[  997.432393]  [<ffffffff8107b29c>] warn_slowpath_fmt+0x5c/0x80
[  997.438811]  [<ffffffff8154ca80>] dev_watchdog+0x270/0x280
[  997.444935]  [<ffffffff8154c810>] ? dev_graft_qdisc+0x80/0x80
[  997.451351]  [<ffffffff8108b0a6>] call_timer_fn+0x36/0x110
[  997.457475]  [<ffffffff8154c810>] ? dev_graft_qdisc+0x80/0x80
[  997.463890]  [<ffffffff8108dd97>] run_timer_softirq+0x237/0x340
[  997.470500]  [<ffffffff81084b0f>] __do_softirq+0xef/0x280
[  997.476528]  [<ffffffff8164721c>] call_softirq+0x1c/0x30
[  997.482461]  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[  997.488196]  [<ffffffff81084ea5>] irq_exit+0x115/0x120
[  997.493932]  [<ffffffff81647e95>] smp_apic_timer_interrupt+0x45/0x60
[  997.501028]  [<ffffffff8164655d>] apic_timer_interrupt+0x6d/0x80
[  997.507734]  <EOI>  [<ffffffff814d4542>] ? cpuidle_enter_state+0x52/0xc0
[  997.515233]  [<ffffffff814d4538>] ? cpuidle_enter_state+0x48/0xc0
[  997.522036]  [<ffffffff814d4689>] cpuidle_idle_call+0xd9/0x210
[  997.528549]  [<ffffffff8101e4be>] arch_cpu_idle+0xe/0x30
[  997.534481]  [<ffffffff810d6305>] cpu_startup_entry+0x245/0x290
[  997.541092]  [<ffffffff81624df7>] rest_init+0x77/0x80
[  997.546735]  [<ffffffff81a8d057>] start_kernel+0x429/0x44a
[  997.552859]  [<ffffffff81a8ca37>] ? repair_env_string+0x5c/0x5c
[  997.559469]  [<ffffffff81a8c120>] ? early_idt_handlers+0x120/0x120
[  997.566369]  [<ffffffff81a8c5ee>] x86_64_start_reservations+0x2a/0x2c
[  997.573560]  [<ffffffff81a8c742>] x86_64_start_kernel+0x152/0x175
[  997.580361] ---[ end trace 6c8c7a3a4f8b32a6 ]---

End of log.


Expected results:
no watchdog warning

Additional info:
Comment 3 Ivan Vecera 2015-11-04 11:07:44 UTC 
Is it reproducibe with TSO disabled on NIC via ethtool?
Comment 5 Ivan Vecera 2015-11-04 17:58:27 UTC 
Hi Prashant,
it's easily reproducible on RHEL6, RHEL7 as well as on the latest upstream.

Server side:

# ip link set eth0 mtu 128 (eth0 is any kind of NIC)
# echo 88 > /proc/sys/net/ipv4/route/min_adv_mss
# iperf -s

Client side: (tg3 NIC - I have tested 5718 and 5719)

# iperf -i 1 -c <server_ip>
Connecting to host 10.16.47.38, port 5201
[  4] local 10.16.47.94 port 41199 connected to 10.16.47.38 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  37.0 MBytes   310 Mbits/sec    0    195 KBytes       
[  4]   1.00-2.00   sec  32.5 MBytes   273 Mbits/sec    0    195 KBytes       
[  4]   2.00-3.00   sec  33.8 MBytes   283 Mbits/sec    0    195 KBytes       
[  4]   3.00-4.00   sec  27.5 MBytes   231 Mbits/sec    0    195 KBytes       
[  4]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   76.0 Bytes       
[  4]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   76.0 Bytes       
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0   76.0 Bytes       
[  4]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   76.0 Bytes       
[  4]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   76.0 Bytes       
...
[  320.533007] tg3 0000:06:00.0 eno1: transmit timed out, resetting 
[  321.796011] tg3 0000:06:00.0 eno1: 0x00000000: 0x165714e4, 0x00100446, 0x02000001, 0x00800000 
[  321.805518] tg3 0000:06:00.0 eno1: 0x00000010: 0x82a5000c, 0x00000000, 0x82a4000c, 0x00000000 
[  321.815024] tg3 0000:06:00.0 eno1: 0x00000020: 0x82a3000c, 0x00000000, 0x00000000, 0x04911014 
[  321.824529] tg3 0000:06:00.0 eno1: 0x00000030: 0x00000000, 0x00000048, 0x00000000, 0x0000010a 
[  321.834033] tg3 0000:06:00.0 eno1: 0x00000040: 0x00000000, 0x6b000000, 0xc8035001, 0x64002008 
[  321.843540] tg3 0000:06:00.0 eno1: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000 
[  321.853045] tg3 0000:06:00.0 eno1: 0x00000060: 0x00000000, 0x00000000, 0xf0010298, 0x00380081 
[  321.862551] tg3 0000:06:00.0 eno1: 0x00000070: 0x00071090, 0x00000000, 0x00000000, 0x00000000 
[  321.872055] tg3 0000:06:00.0 eno1: 0x00000080: 0x165714e4, 0x40000046, 0x00000000, 0x00000111 
[  321.881561] tg3 0000:06:00.0 eno1: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x000004a7 
[  321.891065] tg3 0000:06:00.0 eno1: 0x000000a0: 0x8010ac11, 0x00000004, 0x00001004, 0x00020010 
[  321.900570] tg3 0000:06:00.0 eno1: 0x000000b0: 0x10008d81, 0x00102406, 0x0004cc42, 0x10220140 
[  321.910074] tg3 0000:06:00.0 eno1: 0x000000d0: 0x0000001f, 0x00000009, 0x00000000, 0x00000001 
[  321.919580] tg3 0000:06:00.0 eno1: 0x000000f0: 0x00000000, 0x05719001, 0x00000000, 0x37dcfdfc 
[  321.929084] tg3 0000:06:00.0 eno1: 0x00000100: 0x13c10001, 0x00000000, 0x00100000, 0x000e7030 
[  321.938589] tg3 0000:06:00.0 eno1: 0x00000110: 0x00002000, 0x000031c0, 0x000000a0, 0x00000000 
[  321.948094] tg3 0000:06:00.0 eno1: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003 
[  321.957599] tg3 0000:06:00.0 eno1: 0x00000140: 0x8b5d6758, 0x00006cae, 0x00000000, 0x00000000 
[  321.967104] tg3 0000:06:00.0 eno1: 0x00000150: 0x16010004, 0x00000000, 0x0007812c, 0x00000001 
[  321.976608] tg3 0000:06:00.0 eno1: 0x00000160: 0x00010002, 0x00000000, 0x00000000, 0x00000000 
[  321.986113] tg3 0000:06:00.0 eno1: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000 
[  321.995619] tg3 0000:06:00.0 eno1: 0x00000200: 0x00000000, 0x6b000000, 0x00000000, 0xc1000000 
[  322.005124] tg3 0000:06:00.0 eno1: 0x00000210: 0x00000000, 0x00000000, 0x00000000, 0x1b000000 
[  322.014628] tg3 0000:06:00.0 eno1: 0x00000220: 0x00000000, 0x60000000, 0x00000000, 0x00000000 
[  322.024133] tg3 0000:06:00.0 eno1: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x000004a7 
[  322.033639] tg3 0000:06:00.0 eno1: 0x00000280: 0x00000000, 0x00000111, 0x00000000, 0x00000299 
[  322.043143] tg3 0000:06:00.0 eno1: 0x00000290: 0x00000000, 0x000002dd, 0x00000000, 0x00000558 
[  322.052647] tg3 0000:06:00.0 eno1: 0x00000300: 0x00000000, 0x00000190, 0x00000000, 0x00000000 
[  322.062162] tg3 0000:06:00.0 eno1: 0x00000400: 0x18e04808, 0x00400000, 0x00001000, 0x00000880 
[  322.071666] tg3 0000:06:00.0 eno1: 0x00000410: 0x00006cae, 0x8b5d6758, 0x00006cae, 0x8b5d6758 
[  322.081181] tg3 0000:06:00.0 eno1: 0x00000420: 0x00006cae, 0x8b5d6758, 0x00006cae, 0x8b5d6758 
[  322.090687] tg3 0000:06:00.0 eno1: 0x00000430: 0x00000400, 0x00000000, 0x0000024b, 0x000005f2 
[  322.100193] tg3 0000:06:00.0 eno1: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04384400 
[  322.109697] tg3 0000:06:00.0 eno1: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102 
[  322.119203] tg3 0000:06:00.0 eno1: 0x00000460: 0x00000008, 0x00002620, 0x01ff0002, 0x00000000 
[  322.128708] tg3 0000:06:00.0 eno1: 0x00000470: 0x80000000, 0x20000000, 0x00000000, 0x40000000 
[  322.138212] tg3 0000:06:00.0 eno1: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff 
[  322.147718] tg3 0000:06:00.0 eno1: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000 
[  322.157223] tg3 0000:06:00.0 eno1: 0x00000540: 0x00006cae, 0x8b5d675c, 0x00000000, 0x00000000 
[  322.166727] tg3 0000:06:00.0 eno1: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000 
[  322.176232] tg3 0000:06:00.0 eno1: 0x000005b0: 0x00000000, 0x00000008, 0x00000000, 0x00000000 
[  322.185737] tg3 0000:06:00.0 eno1: 0x000005c0: 0x7adbc438, 0x7734fa3d, 0x00000000, 0x00000000 
[  322.195242] tg3 0000:06:00.0 eno1: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x00001f04 
[  322.204747] tg3 0000:06:00.0 eno1: 0x00000610: 0xffffffff, 0x00000000, 0x07c00004, 0x00000000 
[  322.214251] tg3 0000:06:00.0 eno1: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000 
[  322.223756] tg3 0000:06:00.0 eno1: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123 
[  322.233261] tg3 0000:06:00.0 eno1: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123 
[  322.242767] tg3 0000:06:00.0 eno1: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123 
[  322.252271] tg3 0000:06:00.0 eno1: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123 
[  322.261776] tg3 0000:06:00.0 eno1: 0x00000670: 0x5f865437, 0xe4ac62cc, 0x50103a45, 0x36621985 
[  322.271281] tg3 0000:06:00.0 eno1: 0x00000680: 0xbf14c0e8, 0x1bc27a1e, 0x84f4b556, 0x094ea6fe 
[  322.280787] tg3 0000:06:00.0 eno1: 0x00000690: 0x7dda01e7, 0xc04d7481, 0x00000000, 0x00000000 
[  322.290291] tg3 0000:06:00.0 eno1: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000 
[  322.299795] tg3 0000:06:00.0 eno1: 0x00000800: 0x00001392, 0xffffffff, 0x00000000, 0x00000000 
[  322.309300] tg3 0000:06:00.0 eno1: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000 
[  322.318804] tg3 0000:06:00.0 eno1: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000 
[  322.328309] tg3 0000:06:00.0 eno1: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.337814] tg3 0000:06:00.0 eno1: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.347319] tg3 0000:06:00.0 eno1: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.356823] tg3 0000:06:00.0 eno1: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x0000000f 
[  322.366327] tg3 0000:06:00.0 eno1: 0x00000880: 0x00000772, 0x000094ef, 0x00000000, 0x0000000f 
[  322.375833] tg3 0000:06:00.0 eno1: 0x00000890: 0x00000002, 0x00000002, 0x00000000, 0x00000000 
[  322.385338] tg3 0000:06:00.0 eno1: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000 
[  322.394843] tg3 0000:06:00.0 eno1: 0x00000900: 0x3455a452, 0xffffffff, 0x00000000, 0x00000000 
[  322.404348] tg3 0000:06:00.0 eno1: 0x00000910: 0x00000000, 0xffffffff, 0x00000000, 0x00000000 
[  322.413853] tg3 0000:06:00.0 eno1: 0x00000920: 0x00000000, 0x00000000, 0xffffffff, 0x00000000 
[  322.423358] tg3 0000:06:00.0 eno1: 0x00000930: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.432863] tg3 0000:06:00.0 eno1: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.442368] tg3 0000:06:00.0 eno1: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff 
[  322.451872] tg3 0000:06:00.0 eno1: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x005bc1ad 
[  322.461377] tg3 0000:06:00.0 eno1: 0x00000970: 0x00000009, 0x00000004, 0x00000000, 0x00000000 
[  322.470881] tg3 0000:06:00.0 eno1: 0x00000980: 0x00000c96, 0x000094ef, 0x00000000, 0x00127aac 
[  322.480386] tg3 0000:06:00.0 eno1: 0x00000990: 0x000002a4, 0x000001bc, 0x00000000, 0x00000000 
[  322.489891] tg3 0000:06:00.0 eno1: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001 
[  322.499396] tg3 0000:06:00.0 eno1: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x0d000000 
[  322.508900] tg3 0000:06:00.0 eno1: 0x00000c80: 0x0000296d, 0x00000000, 0x00000000, 0x00000000 
[  322.518405] tg3 0000:06:00.0 eno1: 0x00000ce0: 0x25baacb4, 0x00000008, 0x0072004c, 0x000c062c 
[  322.527910] tg3 0000:06:00.0 eno1: 0x00000cf0: 0x00000000, 0x40000010, 0x404c0072, 0x00000000 
[  322.537416] tg3 0000:06:00.0 eno1: 0x00001000: 0x00000002, 0x00000000, 0xa0008a8b, 0x00000000 
[  322.546920] tg3 0000:06:00.0 eno1: 0x00001010: 0x01901901, 0x00018a8b, 0x00000000, 0x00000000 
[  322.556426] tg3 0000:06:00.0 eno1: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000000 
[  322.565942] tg3 0000:06:00.0 eno1: 0x00001440: 0x00000190, 0x00000190, 0x00000190, 0x00000190 
[  322.575447] tg3 0000:06:00.0 eno1: 0x00001450: 0x00000190, 0x00000190, 0x00000190, 0x00000190 
[  322.584952] tg3 0000:06:00.0 eno1: 0x00001460: 0x00000190, 0x00000190, 0x00000190, 0x00000190 
[  322.594457] tg3 0000:06:00.0 eno1: 0x00001470: 0x00000190, 0x00000190, 0x00000190, 0x00000190 
[  322.603961] tg3 0000:06:00.0 eno1: 0x00001480: 0x00001111, 0x00000000, 0x00000000, 0x00000000 
[  322.613466] tg3 0000:06:00.0 eno1: 0x00001800: 0x00000016, 0x00000000, 0x00000190, 0x00000000 
[  322.622981] tg3 0000:06:00.0 eno1: 0x00001830: 0x00000000, 0x00000000, 0x00000000, 0x4ec80000 
[  322.632485] tg3 0000:06:00.0 eno1: 0x00001840: 0x4ec80000, 0x00000008, 0x00000201, 0xc0000000 
[  322.641989] tg3 0000:06:00.0 eno1: 0x00001850: 0x0000001f, 0x00000000, 0x00004010, 0x01900190 
[  322.651493] tg3 0000:06:00.0 eno1: 0x00001860: 0x0f000000, 0x00000000, 0x4ec81810, 0x00000008 
[  322.660997] tg3 0000:06:00.0 eno1: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000 
[  322.670503] tg3 0000:06:00.0 eno1: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000 
[  322.680009] tg3 0000:06:00.0 eno1: 0x00002010: 0x00000181, 0x00000001, 0x00780003, 0x00000000 
[  322.689514] tg3 0000:06:00.0 eno1: 0x00002100: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.699018] tg3 0000:06:00.0 eno1: 0x00002110: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.708523] tg3 0000:06:00.0 eno1: 0x00002120: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.718028] tg3 0000:06:00.0 eno1: 0x00002130: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.727533] tg3 0000:06:00.0 eno1: 0x00002140: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.737037] tg3 0000:06:00.0 eno1: 0x00002150: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.746542] tg3 0000:06:00.0 eno1: 0x00002160: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.756047] tg3 0000:06:00.0 eno1: 0x00002170: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.765552] tg3 0000:06:00.0 eno1: 0x00002180: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.775058] tg3 0000:06:00.0 eno1: 0x00002190: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.784563] tg3 0000:06:00.0 eno1: 0x000021a0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.794069] tg3 0000:06:00.0 eno1: 0x000021b0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.803575] tg3 0000:06:00.0 eno1: 0x000021c0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.813091] tg3 0000:06:00.0 eno1: 0x000021d0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.822595] tg3 0000:06:00.0 eno1: 0x000021e0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.832100] tg3 0000:06:00.0 eno1: 0x000021f0: 0x0009def0, 0x0009def0, 0x00000000, 0x00000000 
[  322.841605] tg3 0000:06:00.0 eno1: 0x00002200: 0x001280ca, 0x00000000, 0x00000000, 0x00000000 
[  322.851109] tg3 0000:06:00.0 eno1: 0x00002400: 0x00010012, 0x00000000, 0x00400001, 0x00000000 
[  322.860614] tg3 0000:06:00.0 eno1: 0x00002410: 0x0000000f, 0x00005d00, 0x00000000, 0x00000000 
[  322.870119] tg3 0000:06:00.0 eno1: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00044400 
[  322.879625] tg3 0000:06:00.0 eno1: 0x00002450: 0x00000008, 0x51560000, 0x08001800, 0x00040000 
[  322.889130] tg3 0000:06:00.0 eno1: 0x00002470: 0x00000000, 0x000003df, 0x00000000, 0x00000000 
[  322.898634] tg3 0000:06:00.0 eno1: 0x00002500: 0x00000000, 0x00000000, 0x00000002, 0x00044800 
[  322.908138] tg3 0000:06:00.0 eno1: 0x00002510: 0x00000000, 0x00000000, 0x00000002, 0x00040400 
[  322.917644] tg3 0000:06:00.0 eno1: 0x00002520: 0x00000000, 0x00000000, 0x00000002, 0x00044c00 
[  322.927148] tg3 0000:06:00.0 eno1: 0x00002530: 0x00000000, 0x00000000, 0x00000002, 0x00040800 
[  322.936654] tg3 0000:06:00.0 eno1: 0x00002540: 0x00000000, 0x00000000, 0x00000002, 0x00045000 
[  322.946159] tg3 0000:06:00.0 eno1: 0x00002550: 0x00000000, 0x00000000, 0x00000002, 0x00040c00 
[  322.955665] tg3 0000:06:00.0 eno1: 0x00002560: 0x00000000, 0x00000000, 0x00000002, 0x00045400 
[  322.965170] tg3 0000:06:00.0 eno1: 0x00002570: 0x00000000, 0x00000000, 0x00000002, 0x00041000 
[  322.974675] tg3 0000:06:00.0 eno1: 0x00002580: 0x00000000, 0x00000000, 0x00000002, 0x00045800 
[  322.984180] tg3 0000:06:00.0 eno1: 0x00002590: 0x00000000, 0x00000000, 0x00000002, 0x00041400 
[  322.993685] tg3 0000:06:00.0 eno1: 0x000025a0: 0x00000000, 0x00000000, 0x00000002, 0x00045c00 
[  323.003190] tg3 0000:06:00.0 eno1: 0x000025b0: 0x00000000, 0x00000000, 0x00000002, 0x00041800 
[  323.012694] tg3 0000:06:00.0 eno1: 0x000025c0: 0x00000000, 0x00000000, 0x00000002, 0x00046000 
[  323.022200] tg3 0000:06:00.0 eno1: 0x000025d0: 0x00000000, 0x00000000, 0x00000002, 0x00041c00 
[  323.031705] tg3 0000:06:00.0 eno1: 0x000025e0: 0x00000000, 0x00000000, 0x00000002, 0x00046400 
[  323.041209] tg3 0000:06:00.0 eno1: 0x000025f0: 0x00000000, 0x00000000, 0x00000002, 0x00042000 
[  323.050714] tg3 0000:06:00.0 eno1: 0x00002600: 0x00000000, 0x00000000, 0x00000002, 0x00046800 
[  323.060219] tg3 0000:06:00.0 eno1: 0x00002610: 0x00000000, 0x00000000, 0x00000002, 0x00042400 
[  323.069724] tg3 0000:06:00.0 eno1: 0x00002620: 0x00000000, 0x00000000, 0x00000002, 0x00046c00 
[  323.079229] tg3 0000:06:00.0 eno1: 0x00002630: 0x00000000, 0x00000000, 0x00000002, 0x00042800 
[  323.088733] tg3 0000:06:00.0 eno1: 0x00002640: 0x00000000, 0x00000000, 0x00000002, 0x00047000 
[  323.098239] tg3 0000:06:00.0 eno1: 0x00002650: 0x00000000, 0x00000000, 0x00000002, 0x00042c00 
[  323.107744] tg3 0000:06:00.0 eno1: 0x00002660: 0x00000000, 0x00000000, 0x00000002, 0x00047400 
[  323.117250] tg3 0000:06:00.0 eno1: 0x00002670: 0x00000000, 0x00000000, 0x00000002, 0x00043000 
[  323.126754] tg3 0000:06:00.0 eno1: 0x00002680: 0x00000000, 0x00000000, 0x00000002, 0x00047800 
[  323.136267] tg3 0000:06:00.0 eno1: 0x00002690: 0x00000000, 0x00000000, 0x00000002, 0x00043400 
[  323.145773] tg3 0000:06:00.0 eno1: 0x000026a0: 0x00000000, 0x00000000, 0x00000002, 0x00047c00 
[  323.155278] tg3 0000:06:00.0 eno1: 0x000026b0: 0x00000000, 0x00000000, 0x00000002, 0x00043800 
[  323.164783] tg3 0000:06:00.0 eno1: 0x000026c0: 0x00000000, 0x00000000, 0x00000002, 0x00048000 
[  323.174289] tg3 0000:06:00.0 eno1: 0x000026d0: 0x00000000, 0x00000000, 0x00000002, 0x00043c00 
[  323.183794] tg3 0000:06:00.0 eno1: 0x000026e0: 0x00000000, 0x00000000, 0x00000002, 0x00048400 
[  323.193299] tg3 0000:06:00.0 eno1: 0x000026f0: 0x00000000, 0x00000000, 0x00000002, 0x00044000 
[  323.202803] tg3 0000:06:00.0 eno1: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000 
[  323.212308] tg3 0000:06:00.0 eno1: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x0000045f 
[  323.221813] tg3 0000:06:00.0 eno1: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c 
[  323.231319] tg3 0000:06:00.0 eno1: 0x00002c20: 0x00000001, 0x00000000, 0x00000000, 0x00000000 
[  323.240824] tg3 0000:06:00.0 eno1: 0x00002d00: 0x00000080, 0x00000040, 0x00000000, 0x00000000 
[  323.250330] tg3 0000:06:00.0 eno1: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x0000045f 
[  323.259835] tg3 0000:06:00.0 eno1: 0x00003400: 0x00000006, 0x00000000, 0x00000000, 0x00000000 
[  323.269340] tg3 0000:06:00.0 eno1: 0x00003600: 0x00000000, 0x00170000, 0x00110000, 0x00000000 
[  323.278846] tg3 0000:06:00.0 eno1: 0x00003610: 0x00130000, 0x00000000, 0x00130000, 0x00000011 
[  323.288351] tg3 0000:06:00.0 eno1: 0x00003620: 0x00110011, 0x00000000, 0x00000000, 0x00032080 
[  323.297855] tg3 0000:06:00.0 eno1: 0x00003630: 0x00000000, 0x87748774, 0x02c01000, 0x00000000 
[  323.307360] tg3 0000:06:00.0 eno1: 0x00003640: 0x00000000, 0x00000000, 0x00000020, 0x00000019 
[  323.316865] tg3 0000:06:00.0 eno1: 0x00003650: 0x00000171, 0x000f03ff, 0x05719001, 0x00000000 
[  323.326370] tg3 0000:06:00.0 eno1: 0x00003660: 0x00000000, 0x00000000, 0x02000000, 0x00000202 
[  323.335875] tg3 0000:06:00.0 eno1: 0x00003670: 0x00000000, 0xfefffff7, 0x00000000, 0x00000000 
[  323.345380] tg3 0000:06:00.0 eno1: 0x00003680: 0x30018010, 0x00000000, 0x00000000, 0x00000000 
[  323.354886] tg3 0000:06:00.0 eno1: 0x000036a0: 0x000001a0, 0x00000000, 0x00000000, 0x00000000 
[  323.364392] tg3 0000:06:00.0 eno1: 0x000036b0: 0x0010034c, 0x07ff07ff, 0x07ff07ff, 0x01000004 
[  323.373897] tg3 0000:06:00.0 eno1: 0x000036c0: 0xffffffff, 0x00000000, 0x00000000, 0x0011357e 
[  323.383403] tg3 0000:06:00.0 eno1: 0x000036d0: 0x0000019d, 0x00000000, 0x00000000, 0x0000493f 
[  323.392907] tg3 0000:06:00.0 eno1: 0x000036f0: 0x00000000, 0x00000000, 0x00000000, 0x00012b01 
[  323.402412] tg3 0000:06:00.0 eno1: 0x00003800: 0x00000001, 0x00000000, 0x0000000e, 0x0516028b 
[  323.411918] tg3 0000:06:00.0 eno1: 0x00003810: 0x00000024, 0x000000ed, 0x00000000, 0x00000000 
[  323.421422] tg3 0000:06:00.0 eno1: 0x00003c00: 0x00000302, 0x00000000, 0x00000000, 0x00000048 
[  323.430927] tg3 0000:06:00.0 eno1: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000 
[  323.440432] tg3 0000:06:00.0 eno1: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000 
[  323.449937] tg3 0000:06:00.0 eno1: 0x00003c30: 0x00000000, 0x00000000, 0x00000008, 0x4ec9b000 
[  323.459441] tg3 0000:06:00.0 eno1: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000 
[  323.468946] tg3 0000:06:00.0 eno1: 0x00003c50: 0x00000000, 0x000003df, 0x00000000, 0x00000000 
[  323.478450] tg3 0000:06:00.0 eno1: 0x00003c80: 0x00000111, 0x00000299, 0x000002dd, 0x00000558 
[  323.487955] tg3 0000:06:00.0 eno1: 0x00003cc0: 0x00000190, 0x00000000, 0x00000000, 0x00000000 
[  323.497460] tg3 0000:06:00.0 eno1: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000 
[  323.506966] tg3 0000:06:00.0 eno1: 0x00003d00: 0x00000008, 0x4ec88000, 0x00000008, 0x4ec89000 
[  323.516471] tg3 0000:06:00.0 eno1: 0x00003d10: 0x00000008, 0x4ec8a000, 0x00000008, 0x4ec8b000 
[  323.525976] tg3 0000:06:00.0 eno1: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000 
[  323.535482] tg3 0000:06:00.0 eno1: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000 
[  323.544987] tg3 0000:06:00.0 eno1: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000 
[  323.554493] tg3 0000:06:00.0 eno1: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000 
[  323.563997] tg3 0000:06:00.0 eno1: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000 
[  323.573501] tg3 0000:06:00.0 eno1: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000 
[  323.583007] tg3 0000:06:00.0 eno1: 0x00003fc0: 0x0003dfa7, 0x00000000, 0x00000000, 0x00000000 
[  323.592514] tg3 0000:06:00.0 eno1: 0x00004000: 0x00000002, 0x00000000, 0x000eb75f, 0x000d88e1 
[  323.602019] tg3 0000:06:00.0 eno1: 0x00004010: 0x00000000, 0x002f2012, 0x00000480, 0x00857c42 
[  323.611524] tg3 0000:06:00.0 eno1: 0x00004020: 0x00000000, 0x00000000, 0x00000010, 0x00000000 
[  323.621028] tg3 0000:06:00.0 eno1: 0x00004030: 0x00000010, 0x00000050, 0x00000000, 0x00000000 
[  323.630534] tg3 0000:06:00.0 eno1: 0x00004040: 0x00000000, 0x00000000, 0x01081e20, 0x00000000 
[  323.640039] tg3 0000:06:00.0 eno1: 0x00004050: 0x00000000, 0x00000000, 0x002dd010, 0x0048b002 
[  323.649542] tg3 0000:06:00.0 eno1: 0x00004060: 0x0048c000, 0x00000000, 0x00000000, 0x00000000 
[  323.659048] tg3 0000:06:00.0 eno1: 0x00004400: 0x80000006, 0x00000000, 0x00010000, 0x0000a000 
[  323.668552] tg3 0000:06:00.0 eno1: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000 
[  323.678057] tg3 0000:06:00.0 eno1: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000 
[  323.687563] tg3 0000:06:00.0 eno1: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x02004c8d 
[  323.697068] tg3 0000:06:00.0 eno1: 0x00004450: 0x0002053f, 0x00f300f4, 0x00000000, 0x00000000 
[  323.706573] tg3 0000:06:00.0 eno1: 0x00004800: 0x180303fe, 0x00000000, 0x00000000, 0x00000020 
[  323.716079] tg3 0000:06:00.0 eno1: 0x00004810: 0x00000000, 0x0000000b, 0x004c9c78, 0x00002a28 
[  323.725594] tg3 0000:06:00.0 eno1: 0x00004820: 0x0000008c, 0x00000000, 0xf0330000, 0x25baad00 
[  323.735100] tg3 0000:06:00.0 eno1: 0x00004830: 0x00000000, 0x0000003d, 0x0000003d, 0x00000000 
[  323.744604] tg3 0000:06:00.0 eno1: 0x00004840: 0x005bb791, 0x005bb791, 0x000e2200, 0x8c42ec00 
[  323.754108] tg3 0000:06:00.0 eno1: 0x00004850: 0x2996dd3b, 0x7779008e, 0x8b8a8b8a, 0x00000000 
[  323.763614] tg3 0000:06:00.0 eno1: 0x00004860: 0x0000008c, 0x112fda93, 0x00000800, 0x7a000000 
[  323.773119] tg3 0000:06:00.0 eno1: 0x00004870: 0x00000080, 0x00000000, 0x00000000, 0x00000000 
[  323.782624] tg3 0000:06:00.0 eno1: 0x00004900: 0x00090404, 0x00305407, 0x00000000, 0x00000000 
[  323.792127] tg3 0000:06:00.0 eno1: 0x00004910: 0x000f001c, 0x00000000, 0x00000000, 0x00000000 
[  323.801632] tg3 0000:06:00.0 eno1: 0x00004a00: 0x180303fe, 0x00200000, 0x00200020, 0x7bc70000 
[  323.811138] tg3 0000:06:00.0 eno1: 0x00004a10: 0x51568b80, 0x008c5914, 0x00200011, 0x00000000 
[  323.820643] tg3 0000:06:00.0 eno1: 0x00004a20: 0x00000000, 0x00000000, 0xf02c0000, 0x51568be0 
[  323.830149] tg3 0000:06:00.0 eno1: 0x00004a30: 0x00000000, 0x0000009c, 0x0000009c, 0x00000000 
[  323.839654] tg3 0000:06:00.0 eno1: 0x00004a40: 0x51568b80, 0x51568b20, 0x51568b40, 0x51568b60 
[  323.849159] tg3 0000:06:00.0 eno1: 0x00004a50: 0x00200020, 0x00200020, 0x00200020, 0x00200020 
[  323.858664] tg3 0000:06:00.0 eno1: 0x00004a70: 0x00090404, 0x00305407, 0x000f001c, 0x00000000 
[  323.868169] tg3 0000:06:00.0 eno1: 0x00004b00: 0x180303fe, 0x04890003, 0x30000000, 0x00560020 
[  323.877676] tg3 0000:06:00.0 eno1: 0x00004b10: 0x00560040, 0x002a0003, 0x0489dc70, 0x00000000 
[  323.887182] tg3 0000:06:00.0 eno1: 0x00004b20: 0x00000091, 0x02480000, 0x51b664c1, 0x0f890088 
[  323.896687] tg3 0000:06:00.0 eno1: 0x00004b30: 0x2f7facb9, 0x0f560058, 0x2f7fe0b9, 0x0f560058 
[  323.906192] tg3 0000:06:00.0 eno1: 0x00004b40: 0x503e3801, 0x0f2a0030, 0xc0c07e7e, 0xa2a2ecec 
[  323.915696] tg3 0000:06:00.0 eno1: 0x00004b50: 0xf0330000, 0x51b66548, 0xaf490000, 0x91900090 
[  323.925201] tg3 0000:06:00.0 eno1: 0x00004b60: 0xf0330000, 0x2f7fad10, 0xaf060000, 0x40000195 
[  323.934705] tg3 0000:06:00.0 eno1: 0x00004b70: 0xf0330000, 0x2f7fe110, 0xaf060000, 0x000000ff 
[  323.944210] tg3 0000:06:00.0 eno1: 0x00004b80: 0x00000091, 0x112fda93, 0x00000800, 0x7a000000 
[  323.953715] tg3 0000:06:00.0 eno1: 0x00004b90: 0x00000080, 0x00090404, 0x00305407, 0x000f001c 
[  323.963221] tg3 0000:06:00.0 eno1: 0x00004ba0: 0x00f00024, 0x00000000, 0x00000000, 0x00000000 
[  323.972725] tg3 0000:06:00.0 eno1: 0x00004bb0: 0x51b662c1, 0x2f7fccb1, 0x2f7fccb9, 0x521e4cc1 
[  323.982231] tg3 0000:06:00.0 eno1: 0x00004bc0: 0x51b660ba, 0x2f7facba, 0x2f7fe0ba, 0x503e3802 
[  323.991736] tg3 0000:06:00.0 eno1: 0x00004bd0: 0x521d60ce, 0x2f7fccb2, 0x2f7fccba, 0x521e48ba 
[  324.001240] tg3 0000:06:00.0 eno1: 0x00004be0: 0x00560489, 0x002a0056, 0x005e0042, 0x05ea0056 
[  324.010745] tg3 0000:06:00.0 eno1: 0x00004bf0: 0xf0330000, 0x503e3830, 0xaf030000, 0x0000ffff 
[  324.020251] tg3 0000:06:00.0 eno1: 0x00004c00: 0x200003fe, 0x00000000, 0x00000000, 0x00000000 
[  324.029755] tg3 0000:06:00.0 eno1: 0x00004c10: 0x00000015, 0x00000000, 0x00000006, 0x00000000 
[  324.039261] tg3 0000:06:00.0 eno1: 0x00004c20: 0x00000000, 0x00000000, 0x00000000, 0x00000006 
[  324.048765] tg3 0000:06:00.0 eno1: 0x00004c30: 0x00000000, 0x00a64000, 0x000000fb, 0x000000fb 
[  324.058269] tg3 0000:06:00.0 eno1: 0x00004c40: 0x00000000, 0x51a22200, 0x001d0020, 0x00140020 
[  324.067775] tg3 0000:06:00.0 eno1: 0x00004c50: 0xdc298110, 0x00110557, 0x582dd299, 0x005f6b6b 
[  324.077281] tg3 0000:06:00.0 eno1: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000 
[  324.086785] tg3 0000:06:00.0 eno1: 0x00005000: 0x00009800, 0x80004000, 0x00000000, 0x00000000 
[  324.096289] tg3 0000:06:00.0 eno1: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08002178 
[  324.105794] tg3 0000:06:00.0 eno1: 0x00005020: 0xafa40014, 0x00000000, 0x00000000, 0x40000020 
[  324.115298] tg3 0000:06:00.0 eno1: 0x00005030: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.124803] tg3 0000:06:00.0 eno1: 0x00005040: 0x00000000, 0x00000000, 0x080021be, 0x00000000 
[  324.134307] tg3 0000:06:00.0 eno1: 0x00005080: 0x00009800, 0x80004000, 0x00000000, 0x00000000 
[  324.143813] tg3 0000:06:00.0 eno1: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x08002110 
[  324.153318] tg3 0000:06:00.0 eno1: 0x000050a0: 0xaea02e48, 0x00000000, 0x00000000, 0x40000020 
[  324.162823] tg3 0000:06:00.0 eno1: 0x000050b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.172329] tg3 0000:06:00.0 eno1: 0x000050c0: 0x00000000, 0x00000000, 0x08002194, 0x00000000 
[  324.181834] tg3 0000:06:00.0 eno1: 0x00005100: 0x00009800, 0x80004000, 0x00000000, 0x00000000 
[  324.191338] tg3 0000:06:00.0 eno1: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08001bbc 
[  324.200843] tg3 0000:06:00.0 eno1: 0x00005120: 0xaea02e48, 0x00000000, 0x00000000, 0x40000020 
[  324.210347] tg3 0000:06:00.0 eno1: 0x00005130: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.219852] tg3 0000:06:00.0 eno1: 0x00005140: 0x00000000, 0x00000000, 0x08001bde, 0x00000000 
[  324.229357] tg3 0000:06:00.0 eno1: 0x00005180: 0x00009800, 0x80008000, 0x00000000, 0x00000000 
[  324.238862] tg3 0000:06:00.0 eno1: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x08002dc8 
[  324.248365] tg3 0000:06:00.0 eno1: 0x000051a0: 0x14600011, 0x00000000, 0x00000000, 0x40000020 
[  324.257871] tg3 0000:06:00.0 eno1: 0x000051b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.267375] tg3 0000:06:00.0 eno1: 0x000051c0: 0x00000000, 0x00000000, 0x08001bde, 0x00000000 
[  324.276880] tg3 0000:06:00.0 eno1: 0x00005200: 0x00000000, 0x00000000, 0x02c9f5c5, 0x146001a5 
[  324.286386] tg3 0000:06:00.0 eno1: 0x00005210: 0x00000000, 0x00000000, 0x08000000, 0x08002e00 
[  324.295890] tg3 0000:06:00.0 eno1: 0x00005220: 0x00000000, 0xb49a89ab, 0x00000000, 0x08006f98 
[  324.305395] tg3 0000:06:00.0 eno1: 0x00005230: 0x06000000, 0x01020000, 0x00000015, 0x00000000 
[  324.314900] tg3 0000:06:00.0 eno1: 0x00005240: 0xffffffff, 0x146001a5, 0x00000000, 0x00000000 
[  324.324406] tg3 0000:06:00.0 eno1: 0x00005250: 0xc0010000, 0x08006f98, 0x08006f80, 0xb49a89ab 
[  324.333910] tg3 0000:06:00.0 eno1: 0x00005260: 0x08000000, 0x02c9f5c5, 0x146001a5, 0x00000000 
[  324.343414] tg3 0000:06:00.0 eno1: 0x00005270: 0x00000000, 0x08000000, 0x08006f90, 0x00000000 
[  324.352919] tg3 0000:06:00.0 eno1: 0x00005280: 0x00009800, 0x80004000, 0x00000000, 0x00000000 
[  324.362425] tg3 0000:06:00.0 eno1: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x08001974 
[  324.371930] tg3 0000:06:00.0 eno1: 0x000052a0: 0x30632000, 0x00000000, 0x00000000, 0x40000020 
[  324.381434] tg3 0000:06:00.0 eno1: 0x000052b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.390939] tg3 0000:06:00.0 eno1: 0x000052c0: 0x00000000, 0x00000000, 0x08002868, 0x00000000 
[  324.400444] tg3 0000:06:00.0 eno1: 0x00005300: 0x00009800, 0x80000000, 0x00000000, 0x00000000 
[  324.409949] tg3 0000:06:00.0 eno1: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x08002110 
[  324.419455] tg3 0000:06:00.0 eno1: 0x00005320: 0x14400005, 0x00000000, 0x00000000, 0x40000020 
[  324.428960] tg3 0000:06:00.0 eno1: 0x00005330: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.438465] tg3 0000:06:00.0 eno1: 0x00005340: 0x00000000, 0x00000000, 0x08001986, 0x00000000 
[  324.447970] tg3 0000:06:00.0 eno1: 0x00005380: 0x00009800, 0x80004000, 0x00000000, 0x00000000 
[  324.457475] tg3 0000:06:00.0 eno1: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x08001ac0 
[  324.466980] tg3 0000:06:00.0 eno1: 0x000053a0: 0x00031f82, 0x00000000, 0x00000000, 0x40000020 
[  324.476485] tg3 0000:06:00.0 eno1: 0x000053b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000 
[  324.485989] tg3 0000:06:00.0 eno1: 0x000053c0: 0x00000000, 0x00000000, 0x08001bde, 0x00000000 
[  324.495496] tg3 0000:06:00.0 eno1: 0x00005800: 0x6b000000, 0x6b000000, 0xc1000000, 0x00000000 
[  324.505001] tg3 0000:06:00.0 eno1: 0x00005810: 0x00000000, 0x00000000, 0x1b000000, 0x00000000 
[  324.514507] tg3 0000:06:00.0 eno1: 0x00005820: 0x60000000, 0x00000000, 0x00000000, 0x00000000 
[  324.524011] tg3 0000:06:00.0 eno1: 0x00005860: 0x00000000, 0x00000000, 0x000004a7, 0x000004a7 
[  324.533515] tg3 0000:06:00.0 eno1: 0x00005880: 0x00000111, 0x00000111, 0x00000299, 0x00000299 
[  324.543020] tg3 0000:06:00.0 eno1: 0x00005890: 0x000002dd, 0x000002dd, 0x00000558, 0x00000558 
[  324.552525] tg3 0000:06:00.0 eno1: 0x00005900: 0x00000190, 0x00000190, 0x00000000, 0x00000000 
[  324.562029] tg3 0000:06:00.0 eno1: 0x00005980: 0x00000190, 0x00000000, 0x00000000, 0x00000000 
[  324.571534] tg3 0000:06:00.0 eno1: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000 
[  324.581039] tg3 0000:06:00.0 eno1: 0x00006000: 0x00010082, 0x00000000, 0x00000000, 0x00000000 
[  324.590545] tg3 0000:06:00.0 eno1: 0x00006400: 0x00000000, 0x00000000, 0x00010091, 0xc0000000 
[  324.600051] tg3 0000:06:00.0 eno1: 0x00006410: 0x0a000064, 0x0a000064, 0x00000000, 0x00000000 
[  324.609555] tg3 0000:06:00.0 eno1: 0x00006430: 0x00000000, 0x14e41657, 0x04911014, 0x01020000 
[  324.619061] tg3 0000:06:00.0 eno1: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000000 
[  324.628566] tg3 0000:06:00.0 eno1: 0x000064c0: 0x00000010, 0x00000004, 0x00001004, 0x00000000 
[  324.638070] tg3 0000:06:00.0 eno1: 0x000064d0: 0x00000000, 0x10008d81, 0x00000000, 0x00315e42 
[  324.647586] tg3 0000:06:00.0 eno1: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000 
[  324.657090] tg3 0000:06:00.0 eno1: 0x000064f0: 0x00000002, 0x00000031, 0x00000000, 0x00000000 
[  324.666594] tg3 0000:06:00.0 eno1: 0x00006500: 0x01e10003, 0x8b5d6758, 0x00006cae, 0x00000003 
[  324.676100] tg3 0000:06:00.0 eno1: 0x00006510: 0x0007812c, 0x00058116, 0x0004610a, 0x00000000 
[  324.685604] tg3 0000:06:00.0 eno1: 0x00006550: 0x00000001, 0x02800000, 0x00000000, 0x00000000 
[  324.695109] tg3 0000:06:00.0 eno1: 0x000065f0: 0x00000000, 0x00000109, 0x00000000, 0x00000000 
[  324.704615] tg3 0000:06:00.0 eno1: 0x00006800: 0x041b0034, 0x20089082, 0x01c09208, 0x12200c45 
[  324.714119] tg3 0000:06:00.0 eno1: 0x00006810: 0x01020000, 0xffffffff, 0x00000000, 0x00000000 
[  324.723624] tg3 0000:06:00.0 eno1: 0x00006830: 0xffffffff, 0xffffffff, 0x00000000, 0x00000000 
[  324.733129] tg3 0000:06:00.0 eno1: 0x00006840: 0x00000000, 0x00000001, 0x00000000, 0x00000000 
[  324.742635] tg3 0000:06:00.0 eno1: 0x00006890: 0x00000000, 0x88003800, 0x00000000, 0x04102040 
[  324.752139] tg3 0000:06:00.0 eno1: 0x000068a0: 0x00000020, 0x00000001, 0x03ff03ff, 0x00000000 
[  324.761645] tg3 0000:06:00.0 eno1: 0x000068b0: 0xe0011514, 0x00000000, 0x00000000, 0x00000000 
[  324.771149] tg3 0000:06:00.0 eno1: 0x000068f0: 0x00ff000e, 0x00ff0000, 0x00000000, 0x04444444 
[  324.780654] tg3 0000:06:00.0 eno1: 0x00006900: 0x895e9870, 0x14138fbf, 0x00000000, 0x00000000 
[  324.790160] tg3 0000:06:00.0 eno1: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000 
[  324.799666] tg3 0000:06:00.0 eno1: 0x00007000: 0x00000008, 0x00000000, 0x00000000, 0x00004a8c 
[  324.809180] tg3 0000:06:00.0 eno1: 0x00007010: 0x9ed97dea, 0x02408003, 0x000500db, 0x03000a00 
[  324.818687] tg3 0000:06:00.0 eno1: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000 
[  324.828192] tg3 0000:06:00.0 eno1: 0x00007030: 0x00020000, 0x00004a90, 0x000d0030, 0x00000000 
[  324.837699] tg3 0000:06:00.0 eno1: 0: Host status block [00000001:0000006b:(0000:03cf:0000):(0000:0190)] 
[  324.848282] tg3 0000:06:00.0 eno1: 0: NAPI info [0000006b:0000006b:(0190:0190:01ff):0000:(04ad:0000:0000:0000)] 
[  324.859531] tg3 0000:06:00.0 eno1: 1: Host status block [00000001:000000c7:(0000:0000:0000):(0117:0000)] 
[  324.870103] tg3 0000:06:00.0 eno1: 1: NAPI info [000000c7:000000c7:(0000:0000:01ff):0117:(0117:0117:0000:0000)] 
[  324.881352] tg3 0000:06:00.0 eno1: 2: Host status block [00000001:00000000:(0299:0000:0000):(0000:0000)] 
[  324.891922] tg3 0000:06:00.0 eno1: 2: NAPI info [00000000:00000000:(0000:0000:01ff):0299:(0299:0299:0000:0000)] 
[  324.903172] tg3 0000:06:00.0 eno1: 3: Host status block [00000001:0000001b:(0000:0000:0000):(0000:0000)] 
[  324.913742] tg3 0000:06:00.0 eno1: 3: NAPI info [0000001b:0000001b:(0000:0000:01ff):02dd:(02dd:02dd:0000:0000)] 
[  324.924991] tg3 0000:06:00.0 eno1: 4: Host status block [00000001:00000060:(0000:0000:0558):(0000:0000)] 
[  324.935561] tg3 0000:06:00.0 eno1: 4: NAPI info [00000060:00000060:(0000:0000:01ff):0558:(0558:0558:0000:0000)]
Comment 6 Ivan Vecera 2015-11-04 18:01:17 UTC 
Btw. it is reproducible only when TSO is enabled.
Comment 7 xmu 2015-11-05 02:11:47 UTC 
Hi Ivan,

BTW, there are some similar bug has been opened before, they all have the watchdog warning:WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()

e1000e (bz1276596),
igb(1276603),
FCoE ixgbe(1263044), 
i40e(bz1250574)
Comment 8 Ivan Vecera 2015-11-05 07:18:57 UTC 
(In reply to xmu from comment #7)
> Hi Ivan,
> 
> BTW, there are some similar bug has been opened before, they all have the
> watchdog warning:WARNING: at net/sched/sch_generic.c:297
> dev_watchdog+0x270/0x280()
> 
> e1000e (bz1276596),
> igb(1276603),
> FCoE ixgbe(1263044), 
> i40e(bz1250574)

This tx watchdog timeouts are drivers specific bugs. The bugs you mentioned have a different root cause and each should be solved separately.
Comment 9 xmu 2015-11-05 08:14:07 UTC 
(In reply to Ivan Vecera from comment #8)
OK, got it.
Thanks
Comment 10 Prashant 2015-11-23 16:00:54 UTC 
Adding Siva to look at tg3 specific TX timeout.
Comment 11 Ivan Vecera 2015-12-11 17:45:33 UTC 
Siva, do you see any useful in the log?
Comment 12 Ivan Vecera 2015-12-11 17:48:09 UTC 
AFAIK, it is reproducible only when TSO is enabled.
Comment 13 Siva Reddy Kallam 2015-12-12 04:43:57 UTC 
Ivan,

We are able to reproduce this and fix is under review. 
Soon, we will share the patch.
Comment 14 Ivan Vecera 2015-12-15 16:06:21 UTC 
Siva,

thanks a lot. Could you please attach here the proposed fix. I would like to look on it.

Ivan
Comment 15 Siva Reddy Kallam 2015-12-17 10:55:53 UTC 
Created attachment 1106691 [details]
This patch will linearize the skb if available descriptors are not sufficient for gso segment

Here the patch for your review. we will be up-streaming in next batch.
Comment 16 Ivan Vecera 2016-01-15 14:18:56 UTC 
(In reply to Siva Reddy Kallam from comment #15)
> Created attachment 1106691 [details]
> This patch will linearize the skb if available descriptors are not
> sufficient for gso segment
> 
> Here the patch for your review. we will be up-streaming in next batch.

Siva, the patch looks good for me. When are you planning to submit it upstream?
Comment 17 Siva Reddy Kallam 2016-01-18 09:19:50 UTC 
(In reply to Ivan Vecera from comment #16)
> (In reply to Siva Reddy Kallam from comment #15)
> > Created attachment 1106691 [details]
> > This patch will linearize the skb if available descriptors are not
> > sufficient for gso segment
> > 
> > Here the patch for your review. we will be up-streaming in next batch.
> 
> Siva, the patch looks good for me. When are you planning to submit it
> upstream?

We will send the patch to netdev in a week time.
Comment 18 Ivan Vecera 2016-03-29 16:28:42 UTC 

*** This bug has been marked as a duplicate of bug 1258728 ***
Note You need to log in before you can comment on or make changes to this bug.