Bug 1337643
| Summary: | softlockups correlating to "qbrXXXXXXX: hw csum failure" and failed checksumming | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Harald Jensås <hjensas> |
| Component: | kernel-rt | Assignee: | Clark Williams <williams> |
| kernel-rt sub component: | Networking | QA Contact: | qding |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | high | CC: | bhu, hjensas, jshortt, lgoncalv, mleitner, pfb29 |
| Version: | 7.2 | ||
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-03 19:52:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1274397 | ||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2584.html |
Description of problem: When running OpenStack testing, a nova compute node is experiencing softlockups that correspond to apparent lockups. The dumped stack and messages are: [69271.149558] qbrc8e8e9e9-65: hw csum failure [69271.149560] CPU: 29 PID: 575 Comm: kworker/29:1 Not tainted 3.10.0-327.10.1.el7.x86_64 #1 [69271.149561] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 08/02/2014 [69271.149564] Workqueue: events fb_flashcursor [69271.149567] ffff880fe8bc2000 00000000aa45dd55 ffff880fffc63a20 ffffffff816352a4 [69271.149569] ffff880fffc63a38 ffffffff8152821a 0000000000000000 ffff880fffc63a68 [69271.149572] ffffffff815201ed 7d792de08111c451 ffff880fe242c700 0000000000000014 [69271.149572] Call Trace: [69271.149575] <IRQ> [<ffffffff816352a4>] dump_stack+0x19/0x1b [69271.149576] [<ffffffff8152821a>] netdev_rx_csum_fault+0x3a/0x40 [69271.149578] [<ffffffff815201ed>] __skb_checksum_complete+0xbd/0xd0 [69271.149580] [<ffffffff815b7d8c>] nf_ip_checksum+0xcc/0x100 [69271.149586] [<ffffffffa074d572>] udp_error+0x122/0x1f0 [nf_conntrack] [69271.149589] [<ffffffff8111c598>] ? handle_irq_event+0x48/0x60 [69271.149592] [<ffffffffa07467d3>] nf_conntrack_in+0xf3/0xae0 [nf_conntrack] [69271.149598] [<ffffffff8163d1ad>] ? common_interrupt+0x6d/0x6d [69271.149600] [<ffffffffa07c83a4>] ipv4_conntrack_in+0x24/0x30 [nf_conntrack_ipv4] [69271.149602] [<ffffffff8155c860>] nf_iterate+0x70/0xb0 [69271.149604] [<ffffffff8155c948>] nf_hook_slow+0xa8/0x110 [69271.149608] [<ffffffffa076b06e>] br_nf_pre_routing+0x61e/0x730 [bridge] [69271.149612] [<ffffffffa076a640>] ? br_nf_pre_routing_finish_ipv6+0x1b0/0x1b0 [bridge] [69271.149613] [<ffffffff8155c860>] nf_iterate+0x70/0xb0 [69271.149615] [<ffffffff8155c948>] nf_hook_slow+0xa8/0x110 [69271.149619] [<ffffffffa076345a>] br_handle_frame+0x20a/0x2d0 [bridge] [69271.149622] [<ffffffffa0762e30>] ? br_handle_local_finish+0xa0/0xa0 [bridge] [69271.149623] [<ffffffff8152a5d2>] __netif_receive_skb_core+0x1e2/0x7d0 [69271.149625] [<ffffffff8152abd8>] __netif_receive_skb+0x18/0x60 [69271.149627] [<ffffffff8152b8ae>] process_backlog+0xae/0x180 [69271.149628] [<ffffffff8152b092>] net_rx_action+0x152/0x240 [69271.149630] [<ffffffff81084aef>] __do_softirq+0xef/0x280 [69271.149632] [<ffffffff8164735c>] call_softirq+0x1c/0x30 [69271.149634] [<ffffffff81016fc5>] do_softirq+0x65/0xa0 [69271.149636] [<ffffffff81084e85>] irq_exit+0x115/0x120 [69271.149637] [<ffffffff81647ef8>] do_IRQ+0x58/0xf0 [69271.149640] [<ffffffff8163d1ad>] common_interrupt+0x6d/0x6d [69271.149642] <EOI> [<ffffffff81301396>] ? memcpy+0x6/0x110 [69271.149647] [<ffffffffa01b0483>] ? mga_dirty_update+0x193/0x310 [mgag200] [69271.149651] [<ffffffffa01b062f>] mga_imageblit+0x2f/0x40 [mgag200] [69271.149653] [<ffffffff81356ae4>] soft_cursor+0x1b4/0x250 [69271.149654] [<ffffffff81356283>] bit_cursor+0x653/0x690 [69271.149657] [<ffffffff810bb655>] ? sched_clock_cpu+0x85/0xc0 [69271.149659] [<ffffffff813504a6>] fb_flashcursor+0x136/0x140 [69271.149660] [<ffffffff81355c30>] ? bit_clear+0x120/0x120 [69271.149662] [<ffffffff8109d5db>] process_one_work+0x17b/0x470 [69271.149664] [<ffffffff8109e3ab>] worker_thread+0x11b/0x400 [69271.149665] [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400 [69271.149667] [<ffffffff810a5acf>] kthread+0xcf/0xe0 [69271.149670] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 [69271.149672] [<ffffffff81645998>] ret_from_fork+0x58/0x90 [69271.149674] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 Version-Release number of selected component (if applicable): kernel-3.10.0-327.10.1.el7.x86_64 How reproducible: Frequently in customer OVS on OpenStack environment, we see this on all compute nodes. Actual results: qbrc8e8e9e9-65: hw csum failure Expected results: No checksum failiure. Additional info: Releated bug: 1321674 We need this fix: commit 2c26d34bbcc0b3f30385d5587aa232289e2eed8e Author: Jay Vosburgh <jay.vosburgh> Date: Fri Dec 19 15:32:00 2014 -0800 net/core: Handle csum for CHECKSUM_COMPLETE VXLAN forwarding When using VXLAN tunnels and a sky2 device, I have experienced checksum failures of the following type: [ 4297.761899] eth0: hw csum failure [...] [ 4297.765223] Call Trace: [ 4297.765224] <IRQ> [<ffffffff8172f026>] dump_stack+0x46/0x58 [ 4297.765235] [<ffffffff8162ba52>] netdev_rx_csum_fault+0x42/0x50 [ 4297.765238] [<ffffffff8161c1a0>] ? skb_push+0x40/0x40 [ 4297.765240] [<ffffffff8162325c>] __skb_checksum_complete+0xbc/0xd0 [ 4297.765243] [<ffffffff8168c602>] tcp_v4_rcv+0x2e2/0x950 [ 4297.765246] [<ffffffff81666ca0>] ? ip_rcv_finish+0x360/0x360 These are reliably reproduced in a network topology of: container:eth0 == host(OVS VXLAN on VLAN) == bond0 == eth0 (sky2) -> switch When VXLAN encapsulated traffic is received from a similarly configured peer, the above warning is generated in the receive processing of the encapsulated packet. Note that the warning is associated with the container eth0. The skbs from sky2 have ip_summed set to CHECKSUM_COMPLETE, and because the packet is an encapsulated Ethernet frame, the checksum generated by the hardware includes the inner protocol and Ethernet headers. The receive code is careful to update the skb->csum, except in __dev_forward_skb, as called by dev_forward_skb. __dev_forward_skb calls eth_type_trans, which in turn calls skb_pull_inline(skb, ETH_HLEN) to skip over the Ethernet header, but does not update skb->csum when doing so. This patch resolves the problem by adding a call to skb_postpull_rcsum to update the skb->csum after the call to eth_type_trans. Signed-off-by: Jay Vosburgh <jay.vosburgh> Signed-off-by: David S. Miller <davem> We plan to switch to use the real-time kernel soon, and would like to have at least a test build with the above fix included.