Description of problem: OVS-DPDK briges created by TripleO should be down, in order to avoid packet drops due to OVS-DPDK sending packets to the local bridge via a tun/tap (syscall). If the bridge is DOWN, the PMD will drop the packet and avoid this issue. Right now, the packet are sent over the tun/tap and then dropped in the kernel. The example below shows that we spent 139.803 us in kernel per packet sent (to be dropped, see the kfree_skb() call) to kernel. Version-Release number of selected component (if applicable): RHOSP10 and onward How reproducible/Actual results: All OVS-DPDK are affected, just deploy, "ip link" and check the status of all your OVS-DPDK bridges, for instance: [root@overcloud-compute-0 ~]# ip link | grep br- 9: br-admin: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000 10: br-link0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000 23: br-int: <BROADCAST,PROMISC> mtu 9000 qdisc noop state DOWN mode DEFAULT group default qlen 1000 24: br-tun: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 br-admin and br-link0 should be DOWN, as br-in and br-tun, which are DOWN. Additional info: extract from my compute.yaml: - type: ovs_user_bridge name: br-admin use_dhcp: false addresses: - ip_netmask: {get_param: TenantIpSubnet} members: - type: ovs_dpdk_port name: dpdk0 members: - type: interface name: eno2 - type: ovs_user_bridge name: br-link0 use_dhcp: false OVS PMD is running over CPU1, and ftrace, with br-admin UP shows: 1) | SyS_write() { 1) 0.226 us | fget_light(); 1) | vfs_write() { 1) | rw_verify_area() { 1) | security_file_permission() { 1) | selinux_file_permission() { 1) | __inode_security_revalidate() { 1) 0.052 us | _cond_resched(); 1) 0.639 us | } 1) 0.050 us | avc_policy_seqno(); 1) 1.746 us | } 1) 2.377 us | } 1) 2.974 us | } 1) | do_sync_write() { 1) | tun_chr_aio_write [tun]() { 1) | tun_get_user [tun]() { 1) | sock_alloc_send_pskb() { 1) | alloc_skb_with_frags() { 1) | __alloc_skb() { 1) | kmem_cache_alloc_node() { 1) 0.043 us | _cond_resched(); 1) | /* kmem_cache_alloc_node: call_site=ffffffffa2329365 ptr=ffff8c6caacb2d00 bytes_req=256 bytes_alloc=256 gfp_flags=GFP_KERNEL|GFP_REPEAT node=-1 */ 1) 1.067 us | } 1) | __kmalloc_reserve.isra.32() { 1) | __kmalloc_node_track_caller() { 1) 0.065 us | kmalloc_slab(); 1) 0.047 us | _cond_resched(); 1) | /* kmalloc_node: call_site=ffffffffa2329365 ptr=ffff8c6cac324400 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NOMEMALLOC node=-1 */ 1) 1.486 us | } 1) 2.027 us | } 1) 0.239 us | ksize(); 1) 5.263 us | } 1) 5.772 us | } 1) 6.381 us | } 1) 0.048 us | skb_put(); 1) | skb_copy_datagram_from_iovec() { 1) 0.178 us | memcpy_fromiovecend(); 1) 0.686 us | } 1) 0.058 us | eth_type_trans(); 1) | __skb_flow_dissect() { 1) 0.198 us | __skb_flow_get_ports(); 1) 1.947 us | } 1) | __skb_get_hash_symmetric() { 1) | __skb_flow_dissect() { 1) 0.059 us | __skb_flow_get_ports(); 1) 0.659 us | } 1) 1.675 us | } 1) 0.048 us | local_bh_disable(); 1) | netif_receive_skb() { 1) | /* netif_receive_skb_entry: dev=br-admin napi_id=0x0 queue_mapping=0 skbaddr=ffff8c6caacb2d00 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x86dd ip_summed=0 hash=0x00000000 l4_hash=0 len=96 data_len=0 truesize=768 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0x0 */ 1) | netif_receive_skb_internal() { 1) | ktime_get_real() { 1) | getnstimeofday64() { 1) 0.146 us | __getnstimeofday64(); 1) 0.649 us | } 1) 1.182 us | } 1) | skb_defer_rx_timestamp() { 1) 0.085 us | classify(); 1) 0.603 us | } 1) | __netif_receive_skb() { 1) | __netif_receive_skb_core() { 1) | /* netif_receive_skb: dev=br-admin skbaddr=ffff8c6caacb2d00 len=96 */ 1) | ipv6_rcv() { 1) | sock_wfree() { 1) 0.169 us | tun_sock_write_space [tun](); 1) 0.840 us | } 1) | nf_hook_slow() { 1) | nf_iterate() { 1) 0.057 us | ip_sabotage_in [br_netfilter](); 1) | ipv6_defrag [nf_defrag_ipv6]() { 1) | nf_ct_frag6_gather [nf_defrag_ipv6]() { 1) 0.047 us | ipv6_ext_hdr(); 1) 0.961 us | } 1) 1.596 us | } 1) | ipv6_conntrack_in [nf_conntrack_ipv6]() { 1) | nf_conntrack_in [nf_conntrack]() { 1) | ipv6_get_l4proto [nf_conntrack_ipv6]() { 1) 0.110 us | skb_copy_bits(); 1) 0.177 us | ipv6_skip_exthdr(); 1) 1.313 us | } 1) 0.430 us | __nf_ct_l4proto_find [nf_conntrack](); 1) | nf_ct_get_tuple [nf_conntrack]() { 1) 0.071 us | ipv6_pkt_to_tuple [nf_conntrack_ipv6](); 1) 0.173 us | generic_pkt_to_tuple [nf_conntrack](); 1) 1.858 us | } 1) 0.073 us | hash_conntrack_raw [nf_conntrack](); 1) | __nf_conntrack_find_get [nf_conntrack]() { 1) 0.046 us | local_bh_disable(); 1) | local_bh_enable() { 1) 0.051 us | __local_bh_enable_ip(); 1) 0.551 us | } 1) 1.935 us | } 1) | init_conntrack [nf_conntrack]() { 1) | nf_ct_invert_tuple [nf_conntrack]() { 1) 0.044 us | ipv6_invert_tuple [nf_conntrack_ipv6](); 1) 0.053 us | generic_invert_tuple [nf_conntrack](); 1) 1.301 us | } 1) | __nf_conntrack_alloc [nf_conntrack]() { 1) | kmem_cache_alloc() { 1) | /* kmem_cache_alloc: call_site=ffffffffa2329365 ptr=ffff8c6c94ba4140 bytes_req=312 bytes_alloc=320 gfp_flags=GFP_ATOMIC */ 1) 0.620 us | } 1) | init_timer_key() { 1) | /* timer_init: timer=ffff8c6c94ba41d0 */ 1) 0.454 us | } 1) | __nf_ct_ext_add_length [nf_conntrack]() { 1) | __kmalloc() { 1) 0.087 us | kmalloc_slab(); 1) | /* kmalloc: call_site=ffffffffa2329365 ptr=ffff8c6c936ad240 bytes_req=80 bytes_alloc=96 gfp_flags=GFP_ATOMIC|GFP_ZERO */ 1) 1.109 us | } 1) 1.944 us | } 1) 4.965 us | } 1) 0.049 us | generic_get_timeouts [nf_conntrack](); 1) 0.049 us | generic_new [nf_conntrack](); 1) | __nf_ct_ext_add_length [nf_conntrack]() { 1) | __krealloc() { 1) 0.169 us | ksize(); 1) 0.697 us | } 1) 1.531 us | } 1) | __nf_ct_ext_add_length [nf_conntrack]() { 1) | __krealloc() { 1) 0.054 us | ksize(); 1) | __kmalloc_track_caller() { 1) 0.049 us | kmalloc_slab(); 1) | /* kmalloc: call_site=ffffffffa2329365 ptr=ffff8c6c8571cc00 bytes_req=168 bytes_alloc=192 gfp_flags=GFP_ATOMIC */ 1) 1.205 us | } 1) 2.607 us | } 1) | kfree_call_rcu() { 1) | __call_rcu() { 1) | __call_rcu_nocb_enqueue() { 1) | __wake_up() { 1) 0.058 us | _raw_spin_lock_irqsave(); 1) | __wake_up_common() { 1) | autoremove_wake_function() { 1) | default_wake_function() { 1) | try_to_wake_up() { 1) 0.150 us | _raw_spin_lock_irqsave(); 1) 0.284 us | task_waking_fair(); 1) 0.149 us | select_task_rq_fair(); 1) | /* sched_wake_idle_without_ipi: cpu=54 */ 1) 0.060 us | _raw_spin_unlock_irqrestore(); 1) 3.602 us | } 1) 4.069 us | } 1) 4.641 us | } 1) 5.383 us | } 1) 0.052 us | _raw_spin_unlock_irqrestore(); 1) 6.949 us | } 1) 7.462 us | } 1) 8.323 us | } 1) 8.891 us | } 1) + 12.785 us | } 1) 0.046 us | local_bh_disable(); 1) 0.350 us | __nf_ct_try_assign_helper [nf_conntrack](); 1) 0.141 us | _raw_spin_lock(); 1) | local_bh_enable() { 1) 0.054 us | __local_bh_enable_ip(); 1) 0.658 us | } 1) + 27.459 us | } 1) 0.045 us | generic_get_timeouts [nf_conntrack](); 1) | generic_packet [nf_conntrack]() { 1) 0.051 us | __nf_ct_refresh_acct [nf_conntrack](); 1) 0.562 us | } 1) + 38.549 us | } 1) + 39.147 us | } 1) + 43.987 us | } 1) + 44.508 us | } 1) | ip6_rcv_finish() { 1) | ip6_route_input() { 1) | ip6_route_input_lookup.isra.44() { 1) 0.055 us | __ipv6_addr_type(); 1) | fib6_rule_lookup() { 1) | fib_rules_lookup() { 1) 0.179 us | fib6_rule_match(); 1) | fib6_rule_action() { 1) 0.797 us | fib6_get_table(); 1) | ip6_pol_route_input() { 1) | ip6_pol_route.isra.43() { 1) 0.050 us | _raw_read_lock_bh(); 1) 0.686 us | fib6_lookup(); 1) 0.113 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.195 us | rt6_score_route(); 1) 0.900 us | } 1) 0.113 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.209 us | rt6_score_route(); 1) 0.708 us | } 1) 0.113 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.138 us | rt6_score_route(); 1) 0.687 us | } 1) 0.180 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.274 us | rt6_score_route(); 1) 0.791 us | } 1) 0.200 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.231 us | rt6_score_route(); 1) 0.717 us | } 1) 0.191 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.280 us | rt6_score_route(); 1) 0.762 us | } 1) 0.209 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.598 us | rt6_score_route(); 1) 1.084 us | } 1) 0.175 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.221 us | rt6_score_route(); 1) 0.815 us | } 1) 0.171 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.245 us | rt6_score_route(); 1) 0.762 us | } 1) 0.178 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.210 us | rt6_score_route(); 1) 0.687 us | } 1) 0.187 us | rt6_check_expired(); 1) | find_match.part.40() { 1) 0.202 us | rt6_score_route(); 1) 0.741 us | } 1) | _raw_read_unlock_bh() { 1) 0.051 us | __local_bh_enable_ip(); 1) 0.597 us | } 1) + 28.408 us | } 1) + 29.298 us | } 1) + 31.486 us | } 1) + 33.232 us | } 1) + 34.001 us | } 1) + 35.190 us | } 1) + 35.969 us | } 1) | ip6_mc_input() { 1) | ipv6_chk_mcast_addr() { 1) 0.065 us | _raw_read_lock_bh(); 1) | _raw_read_unlock_bh() { 1) 0.049 us | __local_bh_enable_ip(); 1) 0.533 us | } 1) 2.126 us | } 1) | kfree_skb() { 1) | /* kfree_skb: skbaddr=ffff8c6caacb2d00 protocol=34525 location=ffffffffa2329365 */ 1) | skb_release_all() { 1) | skb_release_head_state() { 1) 0.055 us | dst_release(); 1) | nf_conntrack_destroy() { 1) | destroy_conntrack [nf_conntrack]() { 1) 0.048 us | __nf_ct_l4proto_find [nf_conntrack](); 1) 0.047 us | local_bh_disable(); 1) 0.172 us | nf_ct_remove_expectations [nf_conntrack](); 1) | nf_ct_del_from_dying_or_unconfirmed_list [nf_conntrack]() { 1) 0.050 us | _raw_spin_lock(); 1) 0.622 us | } 1) | local_bh_enable() { 1) 0.047 us | __local_bh_enable_ip(); 1) 0.529 us | } 1) | nf_conntrack_free [nf_conntrack]() { 1) 0.150 us | __nf_ct_ext_destroy [nf_conntrack](); 1) | kfree_call_rcu() { 1) | __call_rcu() { 1) | __call_rcu_nocb_enqueue() { 1) | __wake_up() { 1) 0.057 us | _raw_spin_lock_irqsave(); 1) 0.064 us | __wake_up_common(); 1) 0.059 us | _raw_spin_unlock_irqrestore(); 1) 1.528 us | } 1) 2.182 us | } 1) 2.942 us | } 1) 3.526 us | } 1) | kmem_cache_free() { 1) | /* kmem_cache_free: call_site=ffffffffa2329365 ptr=ffff8c6c94ba4140 */ 1) 0.486 us | } 1) 0.043 us | __smp_mb__before_atomic(); 1) 6.600 us | } 1) + 11.610 us | } 1) + 12.230 us | } 1) + 13.354 us | } 1) | skb_release_data() { 1) | kfree() { 1) | /* kfree: call_site=ffffffffa2329365 ptr=ffff8c6cac324400 */ 1) 0.332 us | } 1) 0.823 us | } 1) + 15.113 us | } 1) | kfree_skbmem() { 1) | kmem_cache_free() { 1) | /* kmem_cache_free: call_site=ffffffffa2329365 ptr=ffff8c6caacb2d00 */ 1) 0.308 us | } 1) 0.873 us | } 1) + 18.027 us | } 1) + 21.521 us | } 1) + 58.974 us | } 1) ! 106.748 us | } 1) ! 108.229 us | } 1) ! 108.802 us | } 1) ! 112.092 us | } 1) ! 113.447 us | } 1) | local_bh_enable() { 1) 0.189 us | __local_bh_enable_ip(); 1) 0.669 us | } 1) ! 131.337 us | } 1) ! 132.077 us | } 1) ! 132.768 us | } 1) 0.057 us | __fsnotify_parent(); 1) 0.081 us | fsnotify(); 1) ! 137.944 us | } 1) 0.049 us | fput(); 1) ! 139.803 us | } 1) | syscall_trace_leave() { 1) | context_tracking_user_exit() { 1) 0.062 us | context_tracking_exit(); 1) 0.609 us | }
This is caused by ifup-eth script at https://github.com/fedora-sysv/initscripts/blob/master/network-scripts/ifup-eth#L241 Need to see if any ifcfg config available to control it or could be added to control it. May be a workaround could be to make it down after ifup in os-net-config if it is not possible to alter in ifup-eth scripts.
RHEL BZ on init-scripts to get a proper fix - https://bugzilla.redhat.com/show_bug.cgi?id=1555001
https://github.com/fedora-sysv/initscripts/pull/175 Once the RHEL BZ #1555001 fix is available, os-net-config need to add "LINKSTATUS=down" on the ifcfg files of OVS bridges.
While bringing the bridge down is enough for vlan networks, vxlan tunnels won't work as ovs-dpdk relies on the LOCAL interface to handle ARP, and on netlink to set the appropriate IP and routes. IP address for the local endpoint - ovs-dpdk needs the local IP to be set by netlink - the route for the network doesn't exists as the bridge interface is down --> can be fixed by adding the appropriate route in OVS only, maybe with os-net-config ? ARP: - an ARP responder flow needs to be created by neutron - The linux stack answers to all arp requests on all interfaces, even when the interface is down. without network isolation: if br-admin is down, em1 will answer to the ARP request, which will bring down the vxlan tunnel as the packets will have the dst_mac set the em1 instead of br-admin (the linux stack isn't aware of the ARP responder implemented in OVS) --> even if it is a corner case as we strongly suggest to use network isolation, we can: - set "sysctl -w net.ipv4.conf.all.arp_ignore=1" to allow an arp response only from the appropriate interfaces - have ovs-dpdk in a container (but this is not a solution for OSP10) All these steps can be done with the following script, with the assumption that the bridge holding the vxlan IP is "br-admin". BRIDGE=br-admin PRIORITY=10 # Set the bridge down first or the IP retreived will be the link local ipv6 addr ... ip link set $BRIDGE down MAC=$(ip addr show dev $BRIDGE | awk '/ether/ {print $2}') MAC_HEX=${MAC//:/} IP=$(ip addr show dev $BRIDGE | awk '/inet/ {print $2}' | cut -d'/' -f1) IP_HEX=$(printf '%02X' ${IP//./ }) sysctl -w net.ipv4.conf.all.arp_ignore=1 ovs-ofctl add-flow $BRIDGE "priority=$PRIORITY,arp,arp_op=1,arp_tpa=$IP,actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:$MAC,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0x$MAC_HEX->NXM_NX_ARP_SHA[],load:0x$IP_HEX->NXM_OF_ARP_SPA[],in_port"
*** Bug 1734368 has been marked as a duplicate of this bug. ***
Why this BZ should be fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1734368
As we move to OVN, we won't pursue in fixing this bug, and we will re-open it if OVN requires tap interfaces to be up, in the future.