Bug 1832305

Summary:	Old SG rules created by CNO on Kuryr bootstrap not removed on upgrade
Product:	OpenShift Container Platform	Reporter:	Michał Dulko <mdulko>
Component:	Networking	Assignee:	Michał Dulko <mdulko>
Networking sub component:	kuryr	QA Contact:	Jon Uriarte <juriarte>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	jshepherd, juriarte, ltomasbo, rlobillo
Version:	4.4	Keywords:	AutomationBackLog, UpcomingSprint
Target Milestone:	---
Target Release:	4.5.0
Hardware:	All
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: cluster-network-operator on Kuryr bootstrapping had no logic to remove deprecated security group rules when they get replaced by new ones. Consequence: On OCP upgrade the old SG rules were left on the SGs meaning that tightening them to increase security was not done on environments upgraded from 4.3 to 4.4. Fix: The fix is to make sure CNO is removing old security group rules. Result: The SG rules get removed, on 4.3->4.4 upgrade pods are correctly getting the access to host VMs restricted.	Story Points:	---
Clone Of:
Clones:	1832899 (view as bug list)		Environment:
Last Closed:	2020-07-13 17:35:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1832899

Description Michał Dulko 2020-05-06 13:56:24 UTC

Description of problem:
In 4.4 the set of rules created on master and workers SG for Kuryr to work by CNO was updated to be more strict. The problem is that on upgrade old rules are not removed, so any "tightening" doesn't have affect as, less "tight" rules still exist.

Version-Release number of selected component (if applicable):


How reproducible:
Always on upgrade

Steps to Reproduce:
1. Deploy 4.3 with Kuryr
2. Upgrade to 4.4

Actual results:
On masters and workers SG there will be rules opening traffic from pod subnet on all ports.

Expected results:
Those rules should be gone, replaced by rules only opening specific ports for traffic from pod subnet.

Additional info:
The workaround is to manually remove offending rules on upgrade.

Comment 3 Jon Uriarte 2020-05-08 14:00:10 UTC

Verified in 4.5.0-0.nightly-2020-05-08-060557 on top of OSP 13 2020-04-01.3 puddle.

In order to verify this BZ the cno image has been updated to a 4.5 one that includes the fix.

$ oc scale deploy -n openshift-cluster-version cluster-version-operator --replicas 0
$ 4.5.0-0.nightly-2020-05-08-060557/openshift-install version
4.5.0-0.nightly-2020-05-08-060557/openshift-install 4.5.0-0.nightly-2020-05-08-060557
built from commit fca033874fd4cd5b0d184cccf981dbd3187c1f6c
release image registry.svc.ci.openshift.org/ocp/release@sha256:9281b80d1a5579887ad8bc432f9111f411675153d78b12d015b44d5bab2881ac

$ docker pull registry.svc.ci.openshift.org/ocp/release@sha256:9281b80d1a5579887ad8bc432f9111f411675153d78b12d015b44d5bab2881ac
$ docker run -it registry.svc.ci.openshift.org/ocp/release@sha256:9281b80d1a5579887ad8bc432f9111f411675153d78b12d015b44d5bab2881ac image cluster-network-operator               
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5836903c789e6c45ee611ecd0afb312e4bf19f502fa77f24b4dfe6fe223709a3


$ openstack subnet pool list
+--------------------------------------+-----------------------------------+---------------+
| ID                                   | Name                              | Prefixes      |
+--------------------------------------+-----------------------------------+---------------+
| 3970b6e8-d16c-4299-be1b-219001987b06 | ostest-xdr85-kuryr-pod-subnetpool | 10.128.0.0/14 |
+--------------------------------------+-----------------------------------+---------------+

$ openstack subnet list | grep ostest
| 352dd4ac-b006-4501-b5c1-02bf05e84b3d | ostest-xdr85-nodes                                              | 126ecc35-401e-411e-8453-5ed9e2ab18e2 | 10.196.0.0/16   |
| b1ec6731-50bc-47d9-b1ce-cc2a456d3977 | ostest-xdr85-kuryr-service-subnet                               | 28236289-99ce-4d31-819e-26475c8f50c5 | 172.30.0.0/15   |

$ openstack security group list
+--------------------------------------+----------------------------------------+------------------------+----------------------------------+
| ID                                   | Name                                   | Description            | Project                          |
+--------------------------------------+----------------------------------------+------------------------+----------------------------------+
| 744ed27d-5789-46b1-9c96-0838704190b3 | ostest-xdr85-worker                    |                        | 371b4b309c8b47d4aad01cee1430016b |
| d9bd119a-a64b-43cc-8c8e-b260a8f73d21 | ostest-xdr85-master                    |                        | 371b4b309c8b47d4aad01cee1430016b |
| f15f847d-e007-4845-8446-876e8d1d75bb | ostest-xdr85-kuryr-pods-security-group |                        | 371b4b309c8b47d4aad01cee1430016b |
| fc97c9aa-d354-44c6-950c-6a118a47d382 | default                                | Default security group | 371b4b309c8b47d4aad01cee1430016b |
+--------------------------------------+----------------------------------------+------------------------+----------------------------------+


Run the CNO from 4.5 image:
$ oc -n openshift-network-operator edit deploy network-operator

     image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5836903c789e6c45ee611ecd0afb312e4bf19f502fa77f24b4dfe6fe223709a3  <<<<<<<

$ oc -n openshift-network-operator get pods
NAME                                READY   STATUS    RESTARTS   AGE
network-operator-77fb74dc58-95n8p   1/1     Running   0          32m


$ oc -n openshift-network-operator logs -f network-operator-77fb74dc58-95n8p
...                                                                                 
2020/05/08 13:13:21 Found master nodes security group d9bd119a-a64b-43cc-8c8e-b260a8f73d21
2020/05/08 13:13:22 Found worker nodes security group 744ed27d-5789-46b1-9c96-0838704190b3
2020/05/08 13:13:22 Ensuring pods security group
2020/05/08 13:13:22 Pods security group f15f847d-e007-4845-8446-876e8d1d75bb present
2020/05/08 13:13:22 Allowing required traffic
2020/05/08 13:13:32 All requried traffic allowed
2020/05/08 13:13:32 Removing old SG rules
2020/05/08 13:13:33 Removing decommisioned rule 795e9c3c-c4f7-4e48-8699-08862baa26cd (10.128.0.0/14, 0, 0, ) from SG 744ed27d-5789-46b1-9c96-0838704190b3
2020/05/08 13:13:33 Removing decommisioned rule 97056ecd-7434-4764-b2b1-e39aa17740c4 (172.30.0.0/15, 2379, 2380, tcp) from SG d9bd119a-a64b-43cc-8c8e-b260a8f73d21
2020/05/08 13:13:33 Removing decommisioned rule eb3e8eba-a51c-4628-b3d5-c9e343457004 (10.128.0.0/14, 0, 0, ) from SG d9bd119a-a64b-43cc-8c8e-b260a8f73d21
2020/05/08 13:13:34 Removing decommisioned rule 9e747d94-79f9-4aa8-b87a-e1b5580efc5d (10.196.0.0/16, 0, 0, ) from SG f15f847d-e007-4845-8446-876e8d1d75bb
2020/05/08 13:13:34 All old SG rules removed
...

The CNO log shows the security rules that have been deleted from the worker's, master's and kuryr pods security groups.


$ openstack security group rule list --ingress f15f847d-e007-4845-8446-876e8d1d75bb                                                                                             
+--------------------------------------+-------------+-----------+------------+-----------------------+
| ID                                   | IP Protocol | IP Range  | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+------------+-----------------------+
| 5c2561eb-299c-4ab9-a5d9-06e2115c5020 | None        | 0.0.0.0/0 |            | None                  |
+--------------------------------------+-------------+-----------+------------+-----------------------+

$ openstack security group rule list --ingress d9bd119a-a64b-43cc-8c8e-b260a8f73d21                                                                                             
+--------------------------------------+-------------+---------------+-------------+-----------------------+
| ID                                   | IP Protocol | IP Range      | Port Range  | Remote Security Group |
+--------------------------------------+-------------+---------------+-------------+-----------------------+
| 07af3300-9e07-4792-9352-6f16f6d949a6 | tcp         | 10.196.0.0/16 | 10257:10257 | None                  |
| 17dfb658-6561-47b3-af1c-5cd3ee01c833 | tcp         | 172.30.0.0/15 | 6443:6443   | None                  |
| 20b008c2-b7e1-402e-92e8-b6dcc9446806 | icmp        | 0.0.0.0/0     |             | None                  |
| 20dd5885-3610-4b4e-976c-5ac810b80765 | tcp         | 10.196.0.0/16 | 30000:32767 | None                  |
| 25b75364-1c05-45e1-ba25-bfc7918e290d | tcp         | 0.0.0.0/0     | 22:22       | None                  |
| 2632a855-f2cf-4856-aed2-bd0919069ae9 | tcp         | 0.0.0.0/0     | 6443:6443   | None                  |
| 2eab3e89-5442-4782-8599-43223b33bac2 | tcp         | 10.128.0.0/14 | 10250:10250 | None                  |
| 2ed03a55-3a12-46b5-82f1-bec8d29f96c9 | udp         | 10.128.0.0/14 | 9000:9999   | None                  |
| 313b7c5c-da79-49d4-9291-d1dd853709be | tcp         | 10.196.0.0/16 | 10259:10259 | None                  |
| 34ae6608-e33b-40a1-afe1-df6abdebb4d3 | udp         | 10.196.0.0/16 | 5353:5353   | None                  |
| 358682fa-8847-4146-9f22-c051c9d7a1fa | tcp         | 10.196.0.0/16 | 9000:9999   | None                  |
| 3bb088aa-7dde-47d4-9f47-2cefc70cbc9d | udp         | 10.128.0.0/14 | 53:53       | None                  |
| 3daf295b-41a2-4a6f-b91a-119f565d63e3 | tcp         | 10.128.0.0/14 | 10257:10257 | None                  |
| 3fe37653-c632-4929-830a-7f65fc8e05f1 | udp         | 10.196.0.0/16 | 30000:32767 | None                  |
| 42b675ec-f0d3-4967-afb2-20026930336d | tcp         | 10.128.0.0/14 |             | None                  |
| 787ca8e6-85a5-4154-b55f-ae0c87e9fbcf | tcp         | 10.196.0.0/16 | 2379:2380   | None                  |
| 830375c2-75a5-4bf8-acf9-a4e94f2ac217 | udp         | 10.196.0.0/16 | 53:53       | None                  |
| 9cb446ff-2ce8-4c98-9afe-1714abf6ee2c | tcp         | 10.128.0.0/14 | 53:53       | None                  |
| ab93c6cf-bc2c-4e32-9e31-c7e222829371 | tcp         | 10.196.0.0/16 | 6641:6642   | None                  |
| ada7a367-eb6b-4189-8cb3-87af45897280 | tcp         | 172.30.0.0/15 | 2379:2379   | None                  |
| c127a2da-b9cf-4f13-8b8a-ab4b693948c8 | udp         | 10.196.0.0/16 | 9000:9999   | None                  |
| c3c13297-4791-45d9-8d6a-de59e8d978ca | vrrp        | 10.196.0.0/16 |             | None                  |
| caa03cd7-08e7-4d39-b648-9fa68b566e07 | tcp         | 10.128.0.0/14 | 10259:10259 | None                  |
| e296a1c0-a0a4-45bf-ae7c-84ffb39d396f | tcp         | 10.196.0.0/16 | 22623:22623 | None                  |
| e7b5a37f-efaa-49fb-ac01-4b65cee6e54d | tcp         | 10.128.0.0/14 | 2379:2379   | None                  |
| efee2c5e-3b75-4bce-9dd1-87e048d425aa | udp         | 10.196.0.0/16 | 4789:4789   | None                  |
| f02880ff-458d-4ee8-bfd2-276e51831ba8 | tcp         | 10.196.0.0/16 | 53:53       | None                  |
| f88e69e0-a415-4aa7-857e-7af3e4f29c1d | tcp         | 10.128.0.0/14 | 9000:9999   | None                  |
| f938026c-0366-49fd-ae83-e578190ee5c8 | udp         | 10.196.0.0/16 | 6081:6081   | None                  |
| fdb964c7-9a21-4d38-a780-0d1fcfe8b868 | tcp         | 10.196.0.0/16 | 10250:10250 | None                  |
+--------------------------------------+-------------+---------------+-------------+-----------------------+

$ openstack security group rule list --ingress 744ed27d-5789-46b1-9c96-0838704190b3                                                                                             
+--------------------------------------+-------------+---------------+-------------+-----------------------+
| ID                                   | IP Protocol | IP Range      | Port Range  | Remote Security Group |
+--------------------------------------+-------------+---------------+-------------+-----------------------+
| 0bec190f-df98-4f72-94e5-8ea6ef8eb95d | udp         | 10.128.0.0/14 | 9000:9999   | None                  |
| 0c36070f-0121-4721-a9ab-accabf9d3658 | udp         | 10.196.0.0/16 | 6081:6081   | None                  |
| 264df5f6-9c2d-48e3-88e2-cad5e20b2738 | tcp         | 10.128.0.0/14 | 9000:9999   | None                  |
| 2cdc62c4-a77a-4f6e-a391-fe01cf40fb92 | udp         | 10.196.0.0/16 | 9000:9999   | None                  |
| 2d6f6918-b031-47bd-83b6-7cd4fe380d43 | tcp         | 10.196.0.0/16 | 10250:10250 | None                  |
| 2d766718-76fe-4807-bd93-1be613295de8 | tcp         | 0.0.0.0/0     | 80:80       | None                  |
| 353369ae-b999-4917-bff6-c907e386e285 | udp         | 10.196.0.0/16 | 4789:4789   | None                  |
| 3c7db47b-1de1-4f01-8eea-791f61869379 | tcp         | 10.128.0.0/14 | 10250:10250 | None                  |
| 61d227e8-905d-44c0-b83b-be315b2c92f2 | tcp         | 10.196.0.0/16 | 9000:9999   | None                  |
| 6726b0f0-fa5a-40df-b26e-fa89cc3f2b9e | tcp         | 0.0.0.0/0     | 22:22       | None                  |
| 69c87281-3f1d-4589-a705-02c51d46a80a | tcp         | 0.0.0.0/0     | 443:443     | None                  |
| 82ace9e8-3874-48c3-a715-a4ac26673c3d | udp         | 10.196.0.0/16 | 30000:32767 | None                  |
| 961b1410-7512-4dbe-8549-c929e23fb44f | udp         | 10.196.0.0/16 | 5353:5353   | None                  |
| a52e4374-e50b-456c-855e-d0a86682a357 | tcp         | 10.196.0.0/16 | 30000:32767 | None                  |
| bf2b81c4-0117-4b08-b41d-5f30bb8e950d | tcp         | 10.196.0.0/16 | 1936:1936   | None                  |
| cbd3af0c-dc16-4aef-8be8-fc4516aa5769 | tcp         | 10.128.0.0/14 | 53:53       | None                  |
| d36cff55-fb70-4aed-b086-63c17055e9ac | icmp        | 0.0.0.0/0     |             | None                  |
| d708963a-a869-49cd-8db0-2a6529682ae1 | vrrp        | 10.196.0.0/16 |             | None                  |
| de0fe90d-51c3-469e-83f3-9922408c6b67 | udp         | 10.128.0.0/14 | 53:53       | None                  |
| e939d765-e63e-42e1-871f-2c5404268fc5 | tcp         | 10.128.0.0/14 | 1936:1936   | None                  |
+--------------------------------------+-------------+---------------+-------------+-----------------------+

Comment 4 errata-xmlrpc 2020-07-13 17:35:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409