Red Hat Bugzilla – Bug 1558328
Kernel data path test with OVS 2.9 + DPDK 17.11 fails with low throughput
Last modified: 2018-10-30 04:50:08 EDT
Description of problem: The kernel datapath tests fail to achieve required performance numbers Version-Release number of selected component (if applicable): RHEL 7.5 snapshot 5 How reproducible: Always Steps to Reproduce: Run either of following tests: 1. ovs_perf for kernel datapath 2. VSPerf, sub type kernel datapath Actual results: # 64 Byte OVS Kernel PVP test result: 0.000 # # 1500 Byte OVS Kernel PVP test result: 0.000 # Expected results: PASS Additional info:
Created attachment 1410221 [details] ethtool statistics
With RHEL 7.4 we saw packet drops for kernel datapath tests... - For VSPerf test failure, we collected ethtool statistic diffs as attached - packet drops are due to checksum errors - we believe its a transmit side issue With RHEL 7.5 we observed same failure... - we need to collect the ethtool statistics to determine if its the same issue
The complete logs with RHEL 7.4 (vsperf_results_BB40G_2018-02-09_101038.tar.gz) are uploaded to ftp://10.16.187.188/pub/RHEL_NIC_QUALIFICATION_FastLinQ_test_results. The same logs are also uploaded to Cavium SFT location.
Ran test with Xena traffic generator using very simple traffic profile. Will attach details on traffic being sent. Running just a phy2phy ovs (no vm) test after 3 packets the qede driver will start dropping packets and cause the system to slow to a crawl. My network stack appears to go unresponsive on the system. Until the qede driver has decided to drop(process) all the packets that were sent then my system will come back to normal behaviour. messages log shows lots of errors No other card has this issue that I've seen. Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet Mar 20 07:48:14 netqe22 kernel: [qede_rx_process_cqe:1115(p7p2)]CQE has error, flags = 449, dropping incoming packet This is not a traffic generator issue from my opinion and is a driver problem. Assigning back to qlogic.
Created attachment 1410443 [details] traffic profile
NIC statistics: 0: rcv_pkts: 3 0: rx_hw_errors: 0 0: rx_alloc_errors: 0 0: rx_ip_frags: 0 0: xmit_pkts: 2 0: stopped_cnt: 0 1: rcv_pkts: 0 1: rx_hw_errors: 0 1: rx_alloc_errors: 0 1: rx_ip_frags: 0 1: xmit_pkts: 0 1: stopped_cnt: 0 2: rcv_pkts: 0 2: rx_hw_errors: 0 2: rx_alloc_errors: 0 2: rx_ip_frags: 0 2: xmit_pkts: 0 2: stopped_cnt: 0 3: rcv_pkts: 0 3: rx_hw_errors: 0 3: rx_alloc_errors: 0 3: rx_ip_frags: 0 3: xmit_pkts: 0 3: stopped_cnt: 0 4: rcv_pkts: 0 4: rx_hw_errors: 0 4: rx_alloc_errors: 0 4: rx_ip_frags: 0 4: xmit_pkts: 0 4: stopped_cnt: 0 5: rcv_pkts: 0 5: rx_hw_errors: 0 5: rx_alloc_errors: 0 5: rx_ip_frags: 0 5: xmit_pkts: 0 5: stopped_cnt: 0 6: rcv_pkts: 0 6: rx_hw_errors: 0 6: rx_alloc_errors: 0 6: rx_ip_frags: 0 6: xmit_pkts: 1 6: stopped_cnt: 0 7: rcv_pkts: 0 7: rx_hw_errors: 0 7: rx_alloc_errors: 0 7: rx_ip_frags: 0 7: xmit_pkts: 0 7: stopped_cnt: 0 rx_ucast_bytes: 192 rx_mcast_bytes: 0 rx_bcast_bytes: 0 rx_ucast_pkts: 3 rx_mcast_pkts: 0 rx_bcast_pkts: 0 tx_ucast_bytes: 192 tx_mcast_bytes: 0 tx_bcast_bytes: 0 tx_ucast_pkts: 3 tx_mcast_pkts: 0 tx_bcast_pkts: 0 rx_64_byte_packets: 3 rx_65_to_127_byte_packets: 0 rx_128_to_255_byte_packets: 0 rx_256_to_511_byte_packets: 0 rx_512_to_1023_byte_packets: 0 rx_1024_to_1518_byte_packets: 0 rx_1519_to_max_byte_packets: 0 tx_64_byte_packets: 3 tx_65_to_127_byte_packets: 36 tx_128_to_255_byte_packets: 0 tx_256_to_511_byte_packets: 0 tx_512_to_1023_byte_packets: 0 tx_1024_to_1518_byte_packets: 0 rx_1519_to_max_byte_packets: 0 tx_64_byte_packets: 3 tx_65_to_127_byte_packets: 36 tx_128_to_255_byte_packets: 0 tx_256_to_511_byte_packets: 0 tx_512_to_1023_byte_packets: 0 tx_1024_to_1518_byte_packets: 0 tx_1519_to_max_byte_packets: 0 rx_mac_crtl_frames: 0 tx_mac_ctrl_frames: 0 rx_pause_frames: 0 tx_pause_frames: 0 rx_pfc_frames: 0 tx_pfc_frames: 0 rx_crc_errors: 0 rx_align_errors: 0 rx_carrier_errors: 0 rx_oversize_packets: 0 rx_jabbers: 0 rx_undersize_packets: 0 rx_fragments: 0 brb_truncates: 0 brb_discards: 0 no_buff_discards: 0 mftag_filter_discards: 0 mac_filter_discards: 0 tx_err_drop_pkts: 0 ttl0_discard: 0 packet_too_big_discard: 0 coalesced_pkts: 0 coalesced_events: 0 coalesced_aborts_num: 0 non_coalesced_pkts: 0 coalesced_bytes: 0
Last thing I notice is running at 1pps that sometimes a packet will get processed and go through the flow tables correctly to return to the traffic generator. But almost all of them are dropped. ovs-ofctl dump-ports br0 OFPST_PORT reply (xid=0x2): 3 ports port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=0, bytes=0, drop=0, errs=0, coll=0 port p7p1: rx pkts=6, bytes=384, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=3, bytes=192, drop=0, errs=0, coll=0 port p7p2: rx pkts=9186, bytes=587904, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=6, bytes=384, drop=0, errs=0, coll=0
The wire trace captured shows that the traffic generator is sending packets with incorrect UDP checksum. The driver drops received packets which are marked with invalid checksum by the device. The attached patch is a potential fix for the problem. It changes the behavior by not dropping such packets instead passing it to the stack with CHECKSUM_NONE indication in skb and let stack handle the packet.
Created attachment 1411471 [details] Potential fix
Patch resolves the issue. I was able to get >200kkps with 64 bytes at 0.002 loss rates.
Upstream commit details for this fix/patch. commit 58f101bf87e32753342a6924772c6ebb0fbde24a Author: Manish Chopra <manish.chopra@cavium.com> Date: Wed Mar 28 03:35:52 2018 -0700 qede: Do not drop rx-checksum invalidated packets. Today, driver drops received packets which are indicated as invalid checksum by the device. Instead of dropping such packets, pass them to the stack with CHECKSUM_NONE indication in skb. Signed-off-by: Ariel Elior <ariel.elior@cavium.com> Signed-off-by: Manish Chopra <manish.chopra@cavium.com> Signed-off-by: David S. Miller <davem@davemloft.net>
Harish, Please submit this patch to RHEL
*** Bug 1568523 has been marked as a duplicate of this bug. ***
(In reply to Ameen Rahman from comment #17) > Harish, Please submit this patch to RHEL The patch has been posted.
Adding comment to flag for 7.5 z-stream for this commit: 58f101bf87e32753342a6924772c6ebb0fbde24a qede: Do not drop rx-checksum invalidated packets.
Patch(es) committed on kernel repository and an interim kernel build is undergoing testing
Patch(es) available on kernel-3.10.0-898.el7
This has been verified running VSperf kernel testing on kernel-3.10.0-898.el7 with Q Logic cards. Issue is not present.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:3083