Description of problem: When enabling BFD with IPv6 we hit issues with IPv6 address quoting failed to list routes for rtoe-GR_master-0-0, stderr: "fd2e:6f44:5dd8::89", (OVN command '/usr/bin/ovn-nbctl --timeout=15 --format=csv --data=bare --no-heading --columns=bfd find Logical_Router_Static_Route output_port=rtoe-GR_master-0-0 nexthop=fd2e:6f44:5dd8::89 bfd!=[]' failed: exit status 1) %!(EXTRA string=ovn-nbctl: fd2e:6f44:5dd8::89: unexpected ":" parsing string Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-08-12-174317 How reproducible: Always Steps to Reproduce: 1. oc new-project t1 2. annotate with IPv6 gws oc annotate --overwrite ns t1 k8s.ovn.org/routing-external-gws=fd2e:6f44:5dd8::89,fd2e:6f44:5dd8::76 3. annotate with bfd enable oc annotate --overwrite ns t1 k8s.ovn.org/bfd-enabled="" 4. check the logs for the error Actual results: failed to list routes for rtoe-GR_master-0-0, stderr: "fd2e:6f44:5dd8::89", (OVN command '/usr/bin/ovn-nbctl --timeout=15 --format=csv --data=bare --no-heading --columns=bfd find Logical_Router_Static_Route output_port=rtoe-GR_master-0-0 nexthop=fd2e:6f44:5dd8::89 bfd!=[]' failed: exit status 1)%!(EXTRA string=ovn-nbctl: fd2e:6f44:5dd8::89: unexpected ":" parsing string Expected results: Quotes around the IPv6 address sh-4.4# /usr/bin/ovn-nbctl --timeout=15 --format=csv --data=bare --no-heading --columns=bfd find Logical_Router_Static_Route output_port=rtoe-GR_master-0-2 nexthop='"fd2e:6f44:5dd8::76"' bfd!=[] 61eadeb2-686d-4598-a9f3-f837501ddc38 61eadeb2-686d-4598-a9f3-f837501ddc38 61eadeb2-686d-4598-a9f3-f837501ddc38 Additional info:
Verified on 4.9.0-0.nightly-2021-08-25-061033 I0826 03:24:46.016095 1 namespace.go:278] [t1] updating namespace 2021-08-26T03:24:46.021Z|15696|unixctl|DBG|received request run["--if-exists","--policy=src-ip","--","lr-route-del","GR_master-0-0","fd01:0:0:3::9b/128","fd2e:6f44:5dd8::96"], id=0 2021-08-26T03:24:46.022Z|15697|ovn_dbctl|INFO|Running command run --if-exists --policy=src-ip -- lr-route-del GR_master-0-0 fd01:0:0:3::9b/128 fd2e:6f44:5dd8::96 2021-08-26T03:24:46.027Z|15698|unixctl|DBG|replying with success, id=0: "" 2021-08-26T03:24:46.035Z|15699|unixctl|DBG|received request run["--format=csv","--data=bare","--no-headings","--columns=bfd","--","find","Logical_Router_Static_Route","output_port=rtoe-GR_master-0-0","nexthop=\"fd2e:6f44:5dd8::96\"","bfd!=[]"], id=0 2021-08-26T03:24:46.035Z|15700|ovn_dbctl|DBG|Running command run --format=csv --data=bare --no-headings --columns=bfd -- find Logical_Router_Static_Route output_port=rtoe-GR_master-0-0 "nexthop=\"fd2e:6f44:5dd8::96\"" bfd!=[] 2021-08-26T03:24:46.035Z|15701|unixctl|DBG|replying with success, id=0: "" 2021-08-26T03:24:46.044Z|15702|unixctl|DBG|received request run["--format=csv","--data=bare","--no-headings","--columns=_uuid","--","find","BFD","logical_port=rtoe-GR_master-0-0","dst_ip=\"fd2e:6f44:5dd8::96\""], id=0 2021-08-26T03:24:46.044Z|15703|ovn_dbctl|DBG|Running command run --format=csv --data=bare --no-headings --columns=_uuid -- find BFD logical_port=rtoe-GR_master-0-0 "dst_ip=\"fd2e:6f44:5dd8::96\"" 2021-08-26T03:24:46.044Z|15704|unixctl|DBG|replying with success, id=0: "6196e252-4d96-4c8b-9c7c-4d520321fc4a " 2021-08-26T03:24:46.048Z|15705|unixctl|DBG|received request run["--if-exists","--","destroy","BFD","6196e252-4d96-4c8b-9c7c-4d520321fc4a] id=0 2021-08-26T03:24:46.048Z|15706|ovn_dbctl|INFO|Running command run --if-exists -- destroy BFD 6196e252-4d96-4c8b-9c7c-4d520321fc4a 2021-08-26T03:24:46.050Z|15707|unixctl|DBG|replying with success, id=0: "" 2021-08-26T03:24:46.067Z|15708|unixctl|DBG|received request run["--if-exists","--policy=src-ip","--","lr-route-del","GR_master-0-0","fd01:0:0:3::9b/128","fd2e:6f44:5dd8::8e"], id=0 2021-08-26T03:24:46.067Z|15709|ovn_dbctl|INFO|Running command run --if-exists --policy=src-ip -- lr-route-del GR_master-0-0 fd01:0:0:3::9b/128 fd2e:6f44:5dd8::8e 2021-08-26T03:24:46.070Z|15710|unixctl|DBG|replying with success, id=0: "" 2021-08-26T03:24:46.075Z|15711|unixctl|DBG|received request run["--format=csv","--data=bare","--no-headings","--columns=bfd","--","find","Logical_Router_Static_Route","output_port=rtoe-GR_master-0-0","nexthop=\"fd2e:6f44:5dd8::8e\"","bfd!=[]"], id=0 2021-08-26T03:24:46.075Z|15712|ovn_dbctl|DBG|Running command run --format=csv --data=bare --no-headings --columns=bfd -- find Logical_Router_Static_Route output_port=rtoe-GR_master-0-0 "nexthop=\"fd2e:6f44:5dd8::8e\"" bfd!=[] 2021-08-26T03:24:46.075Z|15713|unixctl|DBG|replying with success, id=0: "" 2021-08-26T03:24:46.080Z|15714|unixctl|DBG|received request run["--format=csv","--data=bare","--no-headings","--columns=_uuid","--","find","BFD","logical_port=rtoe-GR_master-0-0","dst_ip=\"fd2e:6f44:5dd8::8e\""], id=0 2021-08-26T03:24:46.080Z|15715|ovn_dbctl|DBG|Running command run --format=csv --data=bare --no-headings --columns=_uuid -- find BFD logical_port=rtoe-GR_master-0-0 "dst_ip=\"fd2e:6f44:5dd8::8e\"" 2021-08-26T03:24:46.080Z|15716|unixctl|DBG|replying with success, id=0: "dba2ced8-5acb-4e93-9788-d28be2d596c9 " 2021-08-26T03:24:46.094Z|15717|unixctl|DBG|received request run["--if-exists","--","destroy","BFD","dba2ced8-5acb-4e93-9788-d28be2d596c9] id=0 2021-08-26T03:24:46.094Z|15718|ovn_dbctl|INFO|Running command run --if-exists -- destroy BFD dba2ced8-5acb-4e93-9788-d28be2d596c9 2021-08-26T03:24:46.098Z|15719|unixctl|DBG|replying with success, id=0: "" 2021-08-26T03:24:46.102Z|15720|unixctl|DBG|received request run["--may-exist","--policy=src-ip","--ecmp-symmetric-reply","--","lr-route-add","GR_master-0-0"], id=0 2021-08-26T03:24:46.102Z|15721|ovn_dbctl|INFO|Running command run --may-exist --policy=src-ip --ecmp-symmetric-reply -- lr-route-add GR_master-0-0 fd01:0:0:3::9b/128 fd2e:6f44:5dd8::96 rtoe-GR_master-0-0 2021-08-26T03:24:46.107Z|15722|unixctl|DBG|replying with success, id=0: "" 2021-08-26T03:24:46.111Z|15723|unixctl|DBG|received request run["--may-exist","--policy=src-ip","--ecmp-symmetric-reply","--","lr-route-add","GR_master-0-0"], id=0 2021-08-26T03:24:46.111Z|15724|ovn_dbctl|INFO|Running command run --may-exist --policy=src-ip --ecmp-symmetric-reply -- lr-route-add GR_master-0-0 fd01:0:0:3::9b/128 fd2e:6f44:5dd8::8e rtoe-GR_master-0-0 2021-08-26T03:24:46.121Z|15725|unixctl|DBG|replying with success, id=0: "" @surya the ovn-kubernetes code has a comment // cleanUpBFDEntry checks if the BFD table entry related to the associated // gw router / port / gateway ip is referenced by other routing rules, and if // not removes the entry to avoid having dangling BFD entries. // This is temporary and can be safely removed when we consume an ovn version // that includes http://patchwork.ozlabs.org/project/ovn/patch/3c39dc96a36a3445cfa8485a67de79f9f3d5651b.1614602770.git.lorenzo.bianconi@redhat.com/ https://github.com/openshift/ovn-kubernetes/blob/0cbb5365eed472814508210b919befc9181e9eba/go-controller/pkg/ovn/egressgw.go#L524 We just updated OVN, do we have that patch now?
Hi Ross, yea I think master deff has that ovn patch. I'll verify and make sure we update code to remove that check. Nevertheless its a good call to have this fix ^ with the check to see if its referenced before removing since its easier for backports.
On a second round of inspection, it seems that OVN PR never got merged: http://patchwork.ozlabs.org/project/ovn/patch/3c39dc96a36a3445cfa8485a67de79f9f3d5651b.1614602770.git.lorenzo.bianconi@redhat.com/#2654565 So that comment in OVN-K is wrong and can be ignored. I can open a trivial/nit PR to update that to avoid confusions.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759