Bug 1894194

Summary: KuryrPorts leftovers from 4.6 GA need to be deleted
Product: OpenShift Container Platform Reporter: Michał Dulko <mdulko>
Component: NetworkingAssignee: Michał Dulko <mdulko>
Networking sub component: kuryr QA Contact: Itzik Brown <itbrown>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: low CC: juriarte, rlobillo
Version: 4.7   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Due to https://bugzilla.redhat.com/show_bug.cgi?id=1886871 unnecessary Neutron ports were created for hostNetworking pods. Consequence: Deployments installed with that bug existing would keep a number of unnecessary Neutron ports. Fix: On startup Kuryr makes sure to delete any unnecessary ports created for hostNetworking pods. Result: After upgrading to a version with this bug fix the ports should get removed.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:30:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1894408    

Description Michał Dulko 2020-11-03 17:48:12 UTC 
Description of problem:
Anybody installing 4.6 GA with Kuryr will get KuryrPorts and Neutron ports created for host-networking pods due to bug https://bugzilla.redhat.com/show_bug.cgi?id=1886871, which is totally unnecessary and only uses the quota. This bug is about making sure an upgrade to 4.7 or 4.6.z deletes the KuryrPorts, at least allowing them to be used for other pods in the namespace.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Install 4.6 GA.
2. Upgrade kuryr pods to 4.7 or 4.6.z with the fix, wait a bit for Kuryr to finish deleting.
3. Check number of KuryrPorts created for host-networking pods using method from https://bugzilla.redhat.com/show_bug.cgi?id=1886871.

Actual results:
There will be KuryrPorts (probably 64).

Expected results:
The KuryrPorts should be gone.

Additional info:
This will not delete Neutron ports, we'll address that in a future change that'll configure port pools to actually delete Neutron ports.
Comment 3 rlobillo 2020-11-11 16:32:08 UTC 
Verified on OCP4.7.0-0.nightly-2020-11-11-055454 over OSP16.1 with OVN-Octavia (RHOS-16.1-RHEL-8-20201021.n.0)

Upgrade from OCP4.6.0 experiencing the issue to the latest 4.7 nightly results on the deletion of the unnecessary kuryr ports.

# Issue replicated on OCP4.6.0:

	(overcloud) [stack@undercloud-0 ~]$ oc get clusterversion
	NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
	version   4.6.0     True        False         18h     Cluster version is 4.6.0
	(overcloud) [stack@undercloud-0 ~]$ cat core_puddle_version 
	RHOS-16.1-RHEL-8-20201021.n.0
	(overcloud) [stack@undercloud-0 ~]$ oc get nodes
	NAME                          STATUS   ROLES    AGE   VERSION
	ostest-dhsb6-master-0         Ready    master   19h   v1.19.0+d59ce34
	ostest-dhsb6-master-1         Ready    master   19h   v1.19.0+d59ce34
	ostest-dhsb6-master-2         Ready    master   19h   v1.19.0+d59ce34
	ostest-dhsb6-worker-0-h9hpn   Ready    worker   18h   v1.19.0+d59ce34
	ostest-dhsb6-worker-0-q8qmf   Ready    worker   19h   v1.19.0+d59ce34
	ostest-dhsb6-worker-0-v6f9f   Ready    worker   18h   v1.19.0+d59ce34
	(overcloud) [stack@undercloud-0 ~]$ oc get pods -n openshift-kuryr
	NAME                               READY   STATUS    RESTARTS   AGE
	kuryr-cni-bmrrh                    1/1     Running   0          19h
	kuryr-cni-fqz44                    1/1     Running   1          19h
	kuryr-cni-k4v64                    1/1     Running   2          18h
	kuryr-cni-l77np                    1/1     Running   0          19h
	kuryr-cni-n7mmr                    1/1     Running   0          18h
	kuryr-cni-vj75s                    1/1     Running   0          19h
	kuryr-controller-77bc4f8cf-mcgn9   1/1     Running   1          19h

	(overcloud) [stack@undercloud-0 ~]$  oc get pods -A -o wide | grep 10.196 | awk '{print $2}' > host-networking-pods
	(overcloud) [stack@undercloud-0 ~]$ oc get kp -A | grep -f host-networking-pods | wc -l
	64


# Performing the upgrade to latest 4.7 nightly:


	(overcloud) [stack@undercloud-0 ~]$ oc adm upgrade --to-image="registry.svc.ci.openshift.org/ocp/release:4.7.0-0.nightly-2020-11-11-055454" --allow-explicit-upgrade --force=true
	warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
	warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
	warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
	Updating to release image registry.svc.ci.openshift.org/ocp/release:4.7.0-0.nightly-2020-11-11-055454
	(overcloud) [stack@undercloud-0 ~]$ oc get clusterversion
	NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
	version   4.6.0     True        True          109s    Working towards 4.7.0-0.nightly-2020-11-11-055454: 11% complete

# Once openshift-kuryr pods are upgraded, unnecesary kuryr-ports are removed:

	(shiftstack) [stack@undercloud-0 ~]$ oc get clusterversion
	NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
	version   4.6.0     True        True          58m     Working towards 4.7.0-0.nightly-2020-11-11-055454: 84% complete
	(shiftstack) [stack@undercloud-0 ~]$  oc get pods -A -o wide | grep 10.196 | awk '{print $2}' > host-networking-pods.after_upgrade
	(shiftstack) [stack@undercloud-0 ~]$ wc -l host-networking-pods host-networking-pods.after_upgrade 
	  81 host-networking-pods
	  81 host-networking-pods.after_upgrade
	 162 total
	(shiftstack) [stack@undercloud-0 ~]$ oc get kp -A | grep -f host-networking-pods | wc -l
	0
	(shiftstack) [stack@undercloud-0 ~]$ oc get kp -A | grep -f host-networking-pods.after_upgrade | wc -l
	0
	(shiftstack) [stack@undercloud-0 ~]$ oc get pods -n openshift-kuryr
	NAME                                READY   STATUS    RESTARTS   AGE
	kuryr-cni-28vzw                     1/1     Running   0          22m
	kuryr-cni-6dklb                     1/1     Running   0          24m
	kuryr-cni-6qnxf                     1/1     Running   0          23m
	kuryr-cni-7wddz                     1/1     Running   0          26m
	kuryr-cni-hdmxc                     1/1     Running   0          19m
	kuryr-cni-rcvbn                     1/1     Running   0          21m
	kuryr-controller-79f8cc85f8-w7qlx   1/1     Running   1          28m
	(shiftstack) [stack@undercloud-0 ~]$
Comment 6 errata-xmlrpc 2021-02-24 15:30:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633