Bug 2027854
| Summary: | Traffic in ovn not reaching correct pods due to duplicate route | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Dirk Porter <dporter> |
| Component: | Networking | Assignee: | Surya Seetharaman <surya> |
| Networking sub component: | ovn-kubernetes | QA Contact: | Anurag saxena <anusaxen> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | high | ||
| Priority: | unspecified | CC: | bpickard, bzvonar, dacarpen, surya |
| Version: | 4.7 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-12-03 10:11:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hi Surya ~ It looks like this is a 4.7.24 cluster. With regards to your question; "However this error seen on the logs in itself should not cause any traffic disruption or pod creation failures. Even if you see ErrorAddingLogicalPort, the pod should be up and running and those logs can be ignored. Is that the case or are you seeing pods not getting created correctly because of this error?" I don't think this is what is happening, when the errors are in the logs ~ the pods would simply not come up. It appears the latter is occurring, "...seeing pods not getting created correctly because of this error." |
Description of problem: Traffic from coredns pods in the spk-coredns namespace dont reach tmm pod in spk-dns46 namespace. The dns46 namespace has OVN events that indicate route failed because of a duplication, however, the route exists in the OVN. 2021-11-24T16:45:18.214099024Z I1124 16:45:18.214065 1 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"spk-dns46", Name:"f5-tmm-68997cb668-65cpt", UID:"3345d0d2-90d0-4f63-954d-df934eaec2db", APIVersion:"v1", ResourceVersion:"168405204", FieldPath:""}): type: 'Warning' reason: 'ErrorAddingLogicalPort' failed to handle external GW check: unable to add src-ip route to GR router, stderr:"ovn-nbctl: duplicate nexthop for the same ECMP route\n", err:OVN command '/usr/bin/ovn-nbctl --timeout=15 --may-exist --policy=src-ip --ecmp-symmetric-reply lr-route-add GR_master-2 192.168.0.17/32 198.19.16.3' failed: exit status 1 I believe this may be linked to https://bugzilla.redhat.com/show_bug.cgi?id=2005913 To fix the issue: (1) oc delete ns spk-coredns (2) use the "patch pipeline" to recreate the "spk-coredns" namespace and everything inside it (3) delete the pods inside spk-app-01 namespace and confirm that their .spec.dnsConfig.nameservers correctly point to the IP address of the newly-recreated service inside namespace spk-coredns (4) problem cannot be duplicated after doing the above (from digging from inside spk-app-01 pods) : $ oc -n spk-coredns get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE spk-coredns-svc ClusterIP 198.223.90.94 <none> 53/UDP,53/TCP,9154/TCP 14m $ oc -n spk-app-01 exec -it mt-deploy-667fd45fdd-mqp5d -- /bin/bash bash-5.0# cat /etc/resolv.conf search spk-app-01.svc.cluster.local svc.cluster.local cluster.local nameserver 198.223.90.94 options ndots:5 bash-5.0# dig kub12labtest-1.5gc.vzimstest.com +short 151.144.128.1 bash-5.0# dig kub12labtest-1.5gc.vzimstest.com aaaa +short 2607:f160:a:a857::a