Bug 1994647

Summary:	[ipv6] ovn-nbctl calls to find with nexthop= need quotes for IPv6
Product:	OpenShift Container Platform	Reporter:	Ross Brattain <rbrattai>
Component:	Networking	Assignee:	Surya Seetharaman <surya>
Networking sub component:	ovn-kubernetes	QA Contact:	Ross Brattain <rbrattai>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	mifiedle, surya
Version:	4.8
Target Milestone:	---
Target Release:	4.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-10-18 17:47:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1996965

Description Ross Brattain 2021-08-17 15:49:07 UTC

Description of problem:


When enabling BFD with IPv6 we hit issues with IPv6 address quoting

failed to list routes for rtoe-GR_master-0-0, stderr: "fd2e:6f44:5dd8::89",

(OVN command '/usr/bin/ovn-nbctl --timeout=15 --format=csv --data=bare --no-heading --columns=bfd find Logical_Router_Static_Route output_port=rtoe-GR_master-0-0 nexthop=fd2e:6f44:5dd8::89 bfd!=[]' failed: exit status 1)
%!(EXTRA string=ovn-nbctl: fd2e:6f44:5dd8::89: unexpected ":" parsing string


Version-Release number of selected component (if applicable):

4.8.0-0.nightly-2021-08-12-174317


How reproducible:

Always

Steps to Reproduce:
1. oc new-project t1
2. annotate with IPv6 gws
oc annotate --overwrite ns t1  k8s.ovn.org/routing-external-gws=fd2e:6f44:5dd8::89,fd2e:6f44:5dd8::76
3. annotate with bfd enable
oc annotate --overwrite ns t1  k8s.ovn.org/bfd-enabled=""
4. check the logs for the error

Actual results:

failed to list routes for rtoe-GR_master-0-0, stderr: "fd2e:6f44:5dd8::89", (OVN command '/usr/bin/ovn-nbctl --timeout=15 --format=csv --data=bare --no-heading --columns=bfd find Logical_Router_Static_Route output_port=rtoe-GR_master-0-0 nexthop=fd2e:6f44:5dd8::89 bfd!=[]' failed: exit status 1)%!(EXTRA string=ovn-nbctl: fd2e:6f44:5dd8::89: unexpected ":" parsing string


Expected results:

Quotes around the IPv6 address

sh-4.4# /usr/bin/ovn-nbctl --timeout=15 --format=csv --data=bare --no-heading --columns=bfd find Logical_Router_Static_Route output_port=rtoe-GR_master-0-2 nexthop='"fd2e:6f44:5dd8::76"' bfd!=[]
61eadeb2-686d-4598-a9f3-f837501ddc38
61eadeb2-686d-4598-a9f3-f837501ddc38
61eadeb2-686d-4598-a9f3-f837501ddc38

Additional info:

Comment 3 Ross Brattain 2021-08-26 03:30:20 UTC

Verified on 4.9.0-0.nightly-2021-08-25-061033

I0826 03:24:46.016095       1 namespace.go:278] [t1] updating namespace
2021-08-26T03:24:46.021Z|15696|unixctl|DBG|received request run["--if-exists","--policy=src-ip","--","lr-route-del","GR_master-0-0","fd01:0:0:3::9b/128","fd2e:6f44:5dd8::96"], id=0
2021-08-26T03:24:46.022Z|15697|ovn_dbctl|INFO|Running command run --if-exists --policy=src-ip -- lr-route-del GR_master-0-0 fd01:0:0:3::9b/128 fd2e:6f44:5dd8::96
2021-08-26T03:24:46.027Z|15698|unixctl|DBG|replying with success, id=0: ""
2021-08-26T03:24:46.035Z|15699|unixctl|DBG|received request run["--format=csv","--data=bare","--no-headings","--columns=bfd","--","find","Logical_Router_Static_Route","output_port=rtoe-GR_master-0-0","nexthop=\"fd2e:6f44:5dd8::96\"","bfd!=[]"], id=0
2021-08-26T03:24:46.035Z|15700|ovn_dbctl|DBG|Running command run --format=csv --data=bare --no-headings --columns=bfd -- find Logical_Router_Static_Route output_port=rtoe-GR_master-0-0 "nexthop=\"fd2e:6f44:5dd8::96\"" bfd!=[]

2021-08-26T03:24:46.035Z|15701|unixctl|DBG|replying with success, id=0: ""
2021-08-26T03:24:46.044Z|15702|unixctl|DBG|received request run["--format=csv","--data=bare","--no-headings","--columns=_uuid","--","find","BFD","logical_port=rtoe-GR_master-0-0","dst_ip=\"fd2e:6f44:5dd8::96\""], id=0
2021-08-26T03:24:46.044Z|15703|ovn_dbctl|DBG|Running command run --format=csv --data=bare --no-headings --columns=_uuid -- find BFD logical_port=rtoe-GR_master-0-0 "dst_ip=\"fd2e:6f44:5dd8::96\""
2021-08-26T03:24:46.044Z|15704|unixctl|DBG|replying with success, id=0: "6196e252-4d96-4c8b-9c7c-4d520321fc4a
"
2021-08-26T03:24:46.048Z|15705|unixctl|DBG|received request run["--if-exists","--","destroy","BFD","6196e252-4d96-4c8b-9c7c-4d520321fc4a] id=0
2021-08-26T03:24:46.048Z|15706|ovn_dbctl|INFO|Running command run --if-exists -- destroy BFD 6196e252-4d96-4c8b-9c7c-4d520321fc4a
2021-08-26T03:24:46.050Z|15707|unixctl|DBG|replying with success, id=0: ""
2021-08-26T03:24:46.067Z|15708|unixctl|DBG|received request run["--if-exists","--policy=src-ip","--","lr-route-del","GR_master-0-0","fd01:0:0:3::9b/128","fd2e:6f44:5dd8::8e"], id=0
2021-08-26T03:24:46.067Z|15709|ovn_dbctl|INFO|Running command run --if-exists --policy=src-ip -- lr-route-del GR_master-0-0 fd01:0:0:3::9b/128 fd2e:6f44:5dd8::8e
2021-08-26T03:24:46.070Z|15710|unixctl|DBG|replying with success, id=0: ""
2021-08-26T03:24:46.075Z|15711|unixctl|DBG|received request run["--format=csv","--data=bare","--no-headings","--columns=bfd","--","find","Logical_Router_Static_Route","output_port=rtoe-GR_master-0-0","nexthop=\"fd2e:6f44:5dd8::8e\"","bfd!=[]"], id=0
2021-08-26T03:24:46.075Z|15712|ovn_dbctl|DBG|Running command run --format=csv --data=bare --no-headings --columns=bfd -- find Logical_Router_Static_Route output_port=rtoe-GR_master-0-0 "nexthop=\"fd2e:6f44:5dd8::8e\"" bfd!=[]
2021-08-26T03:24:46.075Z|15713|unixctl|DBG|replying with success, id=0: ""
2021-08-26T03:24:46.080Z|15714|unixctl|DBG|received request run["--format=csv","--data=bare","--no-headings","--columns=_uuid","--","find","BFD","logical_port=rtoe-GR_master-0-0","dst_ip=\"fd2e:6f44:5dd8::8e\""], id=0
2021-08-26T03:24:46.080Z|15715|ovn_dbctl|DBG|Running command run --format=csv --data=bare --no-headings --columns=_uuid -- find BFD logical_port=rtoe-GR_master-0-0 "dst_ip=\"fd2e:6f44:5dd8::8e\""
2021-08-26T03:24:46.080Z|15716|unixctl|DBG|replying with success, id=0: "dba2ced8-5acb-4e93-9788-d28be2d596c9
"
2021-08-26T03:24:46.094Z|15717|unixctl|DBG|received request run["--if-exists","--","destroy","BFD","dba2ced8-5acb-4e93-9788-d28be2d596c9] id=0
2021-08-26T03:24:46.094Z|15718|ovn_dbctl|INFO|Running command run --if-exists -- destroy BFD dba2ced8-5acb-4e93-9788-d28be2d596c9
2021-08-26T03:24:46.098Z|15719|unixctl|DBG|replying with success, id=0: ""
2021-08-26T03:24:46.102Z|15720|unixctl|DBG|received request run["--may-exist","--policy=src-ip","--ecmp-symmetric-reply","--","lr-route-add","GR_master-0-0"], id=0
2021-08-26T03:24:46.102Z|15721|ovn_dbctl|INFO|Running command run --may-exist --policy=src-ip --ecmp-symmetric-reply -- lr-route-add GR_master-0-0 fd01:0:0:3::9b/128 fd2e:6f44:5dd8::96 rtoe-GR_master-0-0
2021-08-26T03:24:46.107Z|15722|unixctl|DBG|replying with success, id=0: ""
2021-08-26T03:24:46.111Z|15723|unixctl|DBG|received request run["--may-exist","--policy=src-ip","--ecmp-symmetric-reply","--","lr-route-add","GR_master-0-0"], id=0
2021-08-26T03:24:46.111Z|15724|ovn_dbctl|INFO|Running command run --may-exist --policy=src-ip --ecmp-symmetric-reply -- lr-route-add GR_master-0-0 fd01:0:0:3::9b/128 fd2e:6f44:5dd8::8e rtoe-GR_master-0-0
2021-08-26T03:24:46.121Z|15725|unixctl|DBG|replying with success, id=0: ""



@surya the ovn-kubernetes code has a comment 

// cleanUpBFDEntry checks if the BFD table entry related to the associated
// gw router / port / gateway ip is referenced by other routing rules, and if
// not removes the entry to avoid having dangling BFD entries.
// This is temporary and can be safely removed when we consume an ovn version
// that includes http://patchwork.ozlabs.org/project/ovn/patch/3c39dc96a36a3445cfa8485a67de79f9f3d5651b.1614602770.git.lorenzo.bianconi@redhat.com/

https://github.com/openshift/ovn-kubernetes/blob/0cbb5365eed472814508210b919befc9181e9eba/go-controller/pkg/ovn/egressgw.go#L524

We just updated OVN, do we have that patch now?

Comment 4 Surya Seetharaman 2021-09-02 08:56:37 UTC

Hi Ross, yea I think master deff has that ovn patch. I'll verify and make sure we update code to remove that check. Nevertheless its a good call to have this fix ^ with the check to see if its referenced before removing since its easier for backports.

Comment 5 Surya Seetharaman 2021-09-02 09:04:40 UTC

On a second round of inspection, it seems that OVN PR never got merged:

http://patchwork.ozlabs.org/project/ovn/patch/3c39dc96a36a3445cfa8485a67de79f9f3d5651b.1614602770.git.lorenzo.bianconi@redhat.com/#2654565

So that comment in OVN-K is wrong and can be ignored. I can open a trivial/nit PR to update that to avoid confusions.

Comment 8 errata-xmlrpc 2021-10-18 17:47:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759