Bug 1854192
| Summary: | Ingress/Egress OVN hybrid-overlay integration with F5 BIG-IP doesn't work after upgrading from 4.4.7 to 4.5.0-rc.5 | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Marius Cornea <mcornea> | |
| Component: | Networking | Assignee: | Ben Bennett <bbennett> | |
| Networking sub component: | ovn-kubernetes | QA Contact: | Marius Cornea <mcornea> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | urgent | |||
| Priority: | unspecified | CC: | achernet, agurenko, bbennett, sgordon, xtian, zzhao | |
| Version: | 4.5 | Keywords: | TestBlocker | |
| Target Milestone: | --- | |||
| Target Release: | 4.6.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1854244 (view as bug list) | Environment: | ||
| Last Closed: | 2020-10-27 16:12:24 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1854244 | |||
I created a pod on the nat46 namespace and the route over .3 address is set:
[kni@ocp-edge06 ~]$ oc -n nat46 describe pod test-hybrid-overlay [49/1950]
Name: test-hybrid-overlay
Namespace: nat46
Priority: 0
Node: openshift-worker-1/10.46.29.133
Start Time: Tue, 07 Jul 2020 17:42:04 +0300
Labels: <none>
Annotations: k8s.ovn.org/pod-networks:
{"default":{"ip_addresses":["10.128.2.13/23"],"mac_address":"5a:49:a4:80:02:0e","gateway_ips":["10.128.2.3"],
So the hybrid overlay code seems to run, need to figure out why the flows are not added to br-ext.
I'm in the process of upgrading a cluster from 4.4 to latest 4.5 RC: [ricky@localhost openshift-installer]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True True 12m Working towards 4.5.0-rc.6: 79% complete Will report back later if flows disappear. It seems the cluster keeps the flows during the upgrade but at some retry it misses them: cookie=0xa82003a, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.58 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:3b,output:[118/1814] cookie=0xa820005, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.5 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:06,output:ext cookie=0xa820017, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.23 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:18,output:ext cookie=0xa82002a, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.42 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:2b,output:ext cookie=0xa820035, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.53 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:36,output:ext cookie=0xa820011, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.17 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:12,output:ext cookie=0xa820010, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.16 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:11,output:ext cookie=0xa820015, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.21 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:16,output:ext cookie=0xa820024, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.36 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:25,output:ext cookie=0xa820037, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.55 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:38,output:ext cookie=0xa82002c, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.44 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:2d,output:ext cookie=0xa82003b, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.59 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:3c,output:ext cookie=0xa820012, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.18 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:13,output:ext cookie=0xa820021, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.33 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:22,output:ext cookie=0xa820019, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.25 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:1a,output:ext cookie=0xa820004, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.4 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:05,output:ext cookie=0xa820013, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.19 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:14,output:ext cookie=0xa820031, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.49 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:32,output:ext cookie=0xa820039, duration=800.527s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.57 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:3a,output:ext cookie=0xa820018, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.24 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:19,output:ext cookie=0xa82001c, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.28 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:1d,output:ext cookie=0xa82000d, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.13 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:0e,output:ext cookie=0xa82001b, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.27 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:1c,output:ext cookie=0xa82000c, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.12 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:0d,output:ext cookie=0xa82002b, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.43 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:2c,output:ext cookie=0xa82001d, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.29 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:1e,output:ext cookie=0xa82000b, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.11 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:0c,output:ext cookie=0xa820030, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.48 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:31,output:ext cookie=0xa820034, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.52 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:35,output:ext cookie=0xa82000f, duration=749.774s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.15 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:f2:bf:20:82:00:10,output:ext cookie=0x0, duration=800.526s, table=10, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x0, duration=800.526s, table=20, n_packets=0, n_bytes=0, priority=0 actions=drop [ricky@localhost openshift-installer]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True True 25m Working towards 4.5.0-rc.6: 76% complete [ricky@localhost openshift-installer]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True True 25m Working towards 4.5.0-rc.6: 76% complete ... ... [ricky@localhost openshift-installer]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True True 31m Working towards 4.5.0-rc.6: 28% complete [ricky@localhost openshift-installer]$ oc -n openshift-ovn-kubernetes rsh -c ovnkube-node ovnkube-node-6vtbw ovs-ofctl dump-flows br-ext Any idea how we proceed here folks? As this in effect a regression for the customer using this functionality they are asking when they should expect resolution (or implicitly, when should they plan to upgrade to 4.5 as this would block them from doing so). I missed to put update on this, just put on the clone: < snip > I was pointed about https://github.com/openshift/cluster-network-operator/pull/697 . This prevents from doing upgrades, as the hybrid overlay is not executed. Need to monitor the PR, per Dan Williams this will only land for 4.5.1 . < /snip > TL;DR there's no issue on the hybrid overlay code, it's just the daemonset on 4.5 is broken and doesn't enable it on the script thus upgrades are broken. Fix landed on 4.5.2 and performed an upgrade to make sure it works: [ricky@localhost openshift-installer]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.13 True True 46m Working towards 4.5.2: 87% complete [ricky@localhost openshift-installer]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.2 True False 7m54s Cluster version is 4.5.2 [ricky@localhost openshift-installer]$ oc -n openshift-ovn-kubernetes rsh -c ovnkube-node ovnkube-node-d8zqc ovs-ofctl dump-flows br-ext | head -n10 cookie=0x0, duration=16.513s, table=0, n_packets=0, n_bytes=0, priority=100,arp,in_port=ext,arp_tpa=10.130.0.3,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:0a:58:0a:82:00:03,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xa580a820003->NXM_NX_ARP_SHA[],load:0xa820003->NXM_OF_ARP_SPA[],IN_PORT,resubmit(,1) cookie=0x0, duration=1466.149s, table=0, n_packets=0, n_bytes=0, priority=100,ip,in_port="ext-vxlan",dl_dst=0a:58:0a:82:00:03,nw_dst=10.130.0.0/23 actions=resubmit(,10) cookie=0x0, duration=1466.149s, table=0, n_packets=0, n_bytes=0, priority=10,arp,in_port="ext-vxlan",arp_tpa=10.130.0.0/23,arp_op=1 actions=resubmit(,2) cookie=0x0, duration=1466.149s, table=0, n_packets=2, n_bytes=240, priority=0 actions=drop cookie=0x0, duration=1466.149s, table=1, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x0, duration=16.513s, table=2, n_packets=0, n_bytes=0, priority=100,arp,in_port="ext-vxlan",arp_tpa=10.130.0.0/23,arp_op=1 actions=move:NXM_NX_TUN_IPV4_SRC[]->NXM_NX_TUN_IPV4_DST[],load:0x1001->NXM_NX_TUN_ID[0..31],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:0a:58:0a:82:00:03,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],load:0xa580a820003->NXM_NX_ARP_SHA[],move:NXM_OF_ARP_TPA[]->NXM_NX_REG0[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],move:NXM_NX_REG0[]->NXM_OF_ARP_SPA[],IN_PORT cookie=0x0, duration=1466.149s, table=2, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xa820008, duration=1465.979s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.8 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:ea:58:76:82:00:09,output:ext cookie=0xa820004, duration=1465.979s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.4 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:ea:58:76:82:00:05,output:ext cookie=0xa820006, duration=1465.706s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.130.0.6 actions=mod_dl_src:0a:58:0a:82:00:03,mod_dl_dst:ea:58:76:82:00:07,output:ext Unassigning as I'm on long vacation. @Marius Cornea could you help verify this issue? Verified on 4.5.2 [root@worker-0-0 core]# ovs-ofctl dump-flows br-ext cookie=0xa800207, duration=1364.356s, table=0, n_packets=0, n_bytes=0, priority=10000,ip,in_port=ext,nw_src=10.128.2.7 actions=resubmit(,20) cookie=0xa800208, duration=1364.236s, table=0, n_packets=0, n_bytes=0, priority=10000,ip,in_port=ext,nw_src=10.128.2.8 actions=resubmit(,20) cookie=0xa2e1db9, duration=1364.236s, table=0, n_packets=0, n_bytes=0, priority=120,arp,tun_src=10.46.29.185,in_port="ext-vxlan",arp_spa=192.168.15.100 actions=learn(table=20,priority=50,cookie=0xa800208,eth_type=0x800,ip_src=10.128.2.8,load:NXM_NX_ARP_SHA[]->NXM_OF_ETH_DST[],load:0xa580a800203->NXM_OF_ETH_SRC[],load:0x1001->NXM_NX_TUN_ID[0..31],load:0xa2e1db9->NXM_NX_TUN_IPV4_DST[],output:NXM_OF_IN_PORT[]),learn(table=20,priority=50,cookie=0xa800207,eth_type=0x800,ip_src=10.128.2.7,load:NXM_NX_ARP_SHA[]->NXM_OF_ETH_DST[],load:0xa580a800203->NXM_OF_ETH_SRC[],load:0x1001->NXM_NX_TUN_ID[0..31],load:0xa2e1db9->NXM_NX_TUN_IPV4_DST[],output:NXM_OF_IN_PORT[]),resubmit(,2) cookie=0x0, duration=29.268s, table=0, n_packets=0, n_bytes=0, priority=100,arp,in_port=ext,arp_tpa=10.128.2.3,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:0a:58:0a:80:02:03,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xa580a800203->NXM_NX_ARP_SHA[],load:0xa800203->NXM_OF_ARP_SPA[],IN_PORT,resubmit(,1) cookie=0x0, duration=1392.333s, table=0, n_packets=0, n_bytes=0, priority=100,ip,in_port="ext-vxlan",dl_dst=0a:58:0a:80:02:03,nw_dst=10.128.2.0/23 actions=resubmit(,10) cookie=0x0, duration=1392.333s, table=0, n_packets=0, n_bytes=0, priority=10,arp,in_port="ext-vxlan",arp_tpa=10.128.2.0/23,arp_op=1 actions=resubmit(,2) cookie=0x0, duration=1392.333s, table=0, n_packets=2, n_bytes=240, priority=0 actions=drop cookie=0xa800207, duration=29.268s, table=1, n_packets=0, n_bytes=0, priority=10,arp,arp_tpa=10.128.2.7 actions=mod_dl_dst:ff:ff:ff:ff:ff:ff,mod_dl_src:0a:58:0a:80:02:03,load:0x1->NXM_OF_ARP_OP[],load:0xa580a800203->NXM_NX_ARP_SHA[],load:0xa800203->NXM_OF_ARP_SPA[],load:0xc0a80f64->NXM_OF_ARP_TPA[],load:0->NXM_NX_ARP_THA[],load:0x1001->NXM_NX_TUN_ID[0..31],load:0xa2e1db9->NXM_NX_TUN_IPV4_DST[],output:"ext-vxlan" cookie=0xa800208, duration=29.268s, table=1, n_packets=0, n_bytes=0, priority=10,arp,arp_tpa=10.128.2.8 actions=mod_dl_dst:ff:ff:ff:ff:ff:ff,mod_dl_src:0a:58:0a:80:02:03,load:0x1->NXM_OF_ARP_OP[],load:0xa580a800203->NXM_NX_ARP_SHA[],load:0xa800203->NXM_OF_ARP_SPA[],load:0xc0a80f64->NXM_OF_ARP_TPA[],load:0->NXM_NX_ARP_THA[],load:0x1001->NXM_NX_TUN_ID[0..31],load:0xa2e1db9->NXM_NX_TUN_IPV4_DST[],output:"ext-vxlan" cookie=0x0, duration=1392.333s, table=1, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x0, duration=29.268s, table=2, n_packets=0, n_bytes=0, priority=100,arp,in_port="ext-vxlan",arp_tpa=10.128.2.0/23,arp_op=1 actions=move:NXM_NX_TUN_IPV4_SRC[]->NXM_NX_TUN_IPV4_DST[],load:0x1001->NXM_NX_TUN_ID[0..31],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:0a:58:0a:80:02:03,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],load:0xa580a800203->NXM_NX_ARP_SHA[],move:NXM_OF_ARP_TPA[]->NXM_NX_REG0[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],move:NXM_NX_REG0[]->NXM_OF_ARP_SPA[],IN_PORT cookie=0x0, duration=1392.333s, table=2, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xa800204, duration=1392.290s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.4 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:05,output:ext cookie=0xa800206, duration=1392.242s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.6 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:07,output:ext cookie=0xa800207, duration=1364.356s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.7 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:08,output:ext cookie=0xa800208, duration=1364.236s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.8 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:09,output:ext cookie=0xa800209, duration=1363.831s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.9 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:0a,output:ext cookie=0xa80020a, duration=1363.431s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.10 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:0b,output:ext cookie=0xa800210, duration=1363.194s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.16 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:11,output:ext cookie=0xa80020b, duration=1363.194s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.11 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:0c,output:ext cookie=0xa80020d, duration=1363.194s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.13 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:0e,output:ext cookie=0xa80020c, duration=1363.194s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.12 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:0d,output:ext cookie=0xa80020e, duration=1363.194s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.14 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:0f,output:ext cookie=0xa800211, duration=1351.201s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.17 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:12,output:ext cookie=0xa800212, duration=1351.035s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.18 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:d6:e0:89:80:02:13,output:ext cookie=0x0, duration=29.268s, table=10, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.2.5 actions=mod_dl_src:0a:58:0a:80:02:03,mod_dl_dst:52:86:2c:af:6b:62,output:ext cookie=0x0, duration=1392.333s, table=10, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x0, duration=1392.333s, table=20, n_packets=0, n_bytes=0, priority=0 actions=drop Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |
Description of problem: Ingress/Egress OVN hybrid-overlay integration with F5 big-ip doesn't work after upgrading from 4.4.7 to 4.5.0-rc.5 Version-Release number of selected component (if applicable): 4.5.0-rc.5 How reproducible: On a single environment so far Steps to Reproduce: 1. Deploy OCP on bare metal via IPI process 2. Set up integration with F5 BIG-IP with the OVN hybrid overlay 3. Create namespace apiVersion: v1 kind: Namespace metadata: name: nat46 annotations: k8s.ovn.org/hybrid-overlay-external-gw: '192.168.15.100' k8s.ovn.org/hybrid-overlay-vtep: '10.46.29.185' 4. Create test pods --- apiVersion: apps/v1 kind: Deployment metadata: name: f5-hello-world namespace: nat46 spec: replicas: 1 selector: matchLabels: app: f5-hello-world template: metadata: labels: app: f5-hello-world annotations: k8s.v1.cni.cncf.io/networks: '[{"name": "workercnfens1f1vf0", "namespace": "nat46", "mac": "20:04:0f:f1:88:06","ips": ["10.46.48.6/24", "2620:52:0:2e30::6/64"]}]' spec: nodeSelector: kubernetes.io/hostname: openshift-worker-0 containers: - env: - name: service_name value: f5-hello-world image: registry.ocp-edge.lab.eng.tlv2.redhat.com:5000/rhscl/httpd-24-rhel7:latest imagePullPolicy: Always name: f5-hello-world ports: - containerPort: 8080 protocol: TCP --- apiVersion: apps/v1 kind: Deployment metadata: name: f5-hello-world-https namespace: nat46 spec: replicas: 1 selector: matchLabels: app: f5-hello-world-https template: metadata: labels: app: f5-hello-world-https spec: nodeSelector: kubernetes.io/hostname: openshift-worker-1 containers: - env: - name: service_name value: f5-hello-world-https image: registry.ocp-edge.lab.eng.tlv2.redhat.com:5000/rhscl/httpd-24-rhel7:latest imagePullPolicy: Always name: f5-hello-world-https ports: - containerPort: 8080 protocol: TCP 5. Validate egress curl requests originated from the pods reach the F5 BIG-IP 6. Validate ingress requests originated from outside the cluster reach the pods via the F5 BIG-IP 7. Upgrade OCP cluster to 4.5.0-rc.5 Actual results: After the upgrade, steps 5 and 6 cannot be validated any longer. Checking br-ext flows on the worker nodes they seem to be empty: [root@openshift-worker-0 core]# ovs-ofctl dump-flows br-ext [root@openshift-worker-1 core]# ovs-ofctl dump-flows br-ext Expected results: Connectivity is not interrupted by the upgrade process. Additional info: Link to must-gather: http://10.46.29.134:8080/images/must-gather.local.3469774449681029720/registry-ocp-edge-lab-eng-tlv2-redhat-com-5000-localimages-local-release-image-sha256-adac6646f9be8893b57a346ba62fca3d8f75dfa7059422d30a341f49b531a69b/ oc get network.operator -o yaml apiVersion: v1 items: - apiVersion: operator.openshift.io/v1 kind: Network metadata: creationTimestamp: "2020-05-14T21:30:11Z" generation: 5 name: cluster resourceVersion: "13203811" selfLink: /apis/operator.openshift.io/v1/networks/cluster uid: 663a97c5-81d9-46df-a1b2-0ab6f6bb11eb spec: additionalNetworks: - name: dummy-dhcp-network simpleMacvlanConfig: ipamConfig: type: dhcp master: eth0 mode: bridge mtu: 1500 type: SimpleMacvlan - name: bigip-mgmt namespace: f5-lb rawCNIConfig: '{ "cniVersion": "0.3.1", "type": "bridge", "bridge": "bigip-mgmt", "ipMasq": false, "isGateway": false, "isDefaultGateway": false, "forceAddress": false, "hairpinMode": false,"promiscMode":false }' type: Raw - name: bigip-ha namespace: f5-lb rawCNIConfig: '{ "cniVersion": "0.3.1", "type": "bridge", "bridge": "bigip-ha", "ipMasq": false, "isGateway": false, "isDefaultGateway": false, "forceAddress": false, "hairpinMode": false,"promiscMode":false }' type: Raw clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 defaultNetwork: ovnKubernetesConfig: hybridOverlayConfig: hybridClusterNetwork: - cidr: 172.255.0.0/16 hostPrefix: 23 type: OVNKubernetes logLevel: "" serviceNetwork: - 172.30.0.0/16 status: {} kind: List metadata: resourceVersion: "" selfLink: ""