Bug 1832899

Summary:	Old SG rules created by CNO on Kuryr bootstrap not removed on upgrade
Product:	OpenShift Container Platform	Reporter:	Michał Dulko <mdulko>
Component:	Networking	Assignee:	Michał Dulko <mdulko>
Networking sub component:	kuryr	QA Contact:	Jon Uriarte <juriarte>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	urgent	CC:	gcheresh, juriarte, ltomasbo
Version:	4.4
Target Milestone:	---
Target Release:	4.4.z
Hardware:	All
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: cluster-network-operator on Kuryr bootstrapping had no logic to remove deprecated security group rules when they get replaced by new ones. Consequence: On OCP upgrade the old SG rules were left on the SGs meaning that tightening them to increase security was not done on environments upgraded from 4.3 to 4.4. Fix: The fix is to make sure CNO is removing old security group rules. Result: The SG rules get removed, on 4.3->4.4 upgrade pods are correctly getting the access to host VMs restricted.	Story Points:	---
Clone Of:	1832305
Clones:	1834858 (view as bug list)		Environment:
Last Closed:	2020-05-18 13:35:03 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1832305
Bug Blocks:	1834858

Description Michał Dulko 2020-05-07 12:57:17 UTC

+++ This bug was initially created as a clone of Bug #1832305 +++

Description of problem:
In 4.4 the set of rules created on master and workers SG for Kuryr to work by CNO was updated to be more strict. The problem is that on upgrade old rules are not removed, so any "tightening" doesn't have affect as, less "tight" rules still exist.

Version-Release number of selected component (if applicable):


How reproducible:
Always on upgrade

Steps to Reproduce:
1. Deploy 4.3 with Kuryr
2. Upgrade to 4.4

Actual results:
On masters and workers SG there will be rules opening traffic from pod subnet on all ports.

Expected results:
Those rules should be gone, replaced by rules only opening specific ports for traffic from pod subnet.

Additional info:
The workaround is to manually remove offending rules on upgrade.

Comment 3 Jon Uriarte 2020-05-11 18:01:34 UTC

Verified in 4.4.0-0.nightly-2020-05-08-224132 on top of OSP 16 compose RHOS_TRUNK-16.0-RHEL-8-20200506.n.2 (with OVS and amphora-driver).

In order to verify this BZ the cno image has been updated to a 4.4 one that includes the fix.

$ oc scale deploy -n openshift-cluster-version cluster-version-operator --replicas 0

$ 4.4.0-0.nightly-2020-05-08-224132/openshift-install version
4.4.0-0.nightly-2020-05-08-224132/openshift-install 4.4.0-0.nightly-2020-05-08-224132
built from commit 0c7f28d8642b8e9bf330038423120c5cf6a4bf91
release image registry.svc.ci.openshift.org/ocp/release@sha256:eec18530cd4777dcd9965083b1114a826ccaf1966bbbfb324afed79b0f8c6182

$ sudo podman pull --authfile secret.json registry.svc.ci.openshift.org/ocp/release@sha256:eec18530cd4777dcd9965083b1114a826ccaf1966bbbfb324afed79b0f8c6182
$ sudo podman run --network none --rm -it registry.svc.ci.openshift.org/ocp/release@sha256:eec18530cd4777dcd9965083b1114a826ccaf1966bbbfb324afed79b0f8c6182 image cluster-network-operator
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d04a1399063458596b789e5a7219b4613870e1b285a647539d93d6ebaf6542d7



$ openstack subnet pool list
+--------------------------------------+-----------------------------------+---------------+
| ID                                   | Name                              | Prefixes      |
+--------------------------------------+-----------------------------------+---------------+
| cb2bcfbc-e8e3-4cf9-9254-435ce010b9f2 | ostest-4bwnq-kuryr-pod-subnetpool | 10.128.0.0/14 |
+--------------------------------------+-----------------------------------+---------------+

$ openstack subnet list | grep ostest
| c89265bb-0e9a-4015-9c1a-7ab9b96f48f3 | ostest-4bwnq-kuryr-service-subnet                               | 1154692a-353b-4f86-ace7-aafe203d5a4f | 172.30.0.0/15   |
| e02edda1-62f8-42b6-8eac-6f036c57da8e | ostest-4bwnq-nodes                                              | 958c0181-7edf-47a5-8349-229cbedbf97a | 10.196.0.0/16   |


$ openstack security group list
+--------------------------------------+----------------------------------------+------------------------+----------------------------------+-------------------------------------+
| ID                                   | Name                                   | Description            | Project                          | Tags                                |
+--------------------------------------+----------------------------------------+------------------------+----------------------------------+-------------------------------------+
| 17335149-dbc1-48db-ac70-de10ed4b2528 | default                                | Default security group | 74e7ae11d5614856b968999e8dfcd0f1 | []                                  |
| 2d1b2f73-8ab5-4ebb-9e7c-10b79025af06 | ostest-4bwnq-kuryr-pods-security-group |                        | 74e7ae11d5614856b968999e8dfcd0f1 | ['openshiftClusterID=ostest-4bwnq'] |
| 4ca131eb-232f-46a5-9561-e78f5eba84fc | ostest-4bwnq-master                    |                        | 74e7ae11d5614856b968999e8dfcd0f1 | ['openshiftClusterID=ostest-4bwnq'] |
| ac5d2812-e4d1-4bf4-b7a4-d2dbd34d1caf | ostest-4bwnq-worker                    |                        | 74e7ae11d5614856b968999e8dfcd0f1 | ['openshiftClusterID=ostest-4bwnq'] |
+--------------------------------------+----------------------------------------+------------------------+----------------------------------+-------------------------------------+


Run the CNO from 4.4 image:
$ oc -n openshift-network-operator edit deploy network-operator

     image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d04a1399063458596b789e5a7219b4613870e1b285a647539d93d6ebaf6542d7  <<<<<<<


$ oc -n openshift-network-operator get pods
NAME                                READY   STATUS    RESTARTS   AGE
network-operator-59b69dd7c-ljd4v   1/1     Running   0          60s


$ oc -n openshift-network-operator logs -f network-operator-59b69dd7c-ljd4v
...                                                                                 
2020/05/11 17:51:55 Found master nodes security group 4ca131eb-232f-46a5-9561-e78f5eba84fc
2020/05/11 17:51:55 Found worker nodes security group ac5d2812-e4d1-4bf4-b7a4-d2dbd34d1caf
2020/05/11 17:51:55 Ensuring pods security group
2020/05/11 17:51:55 Pods security group 2d1b2f73-8ab5-4ebb-9e7c-10b79025af06 present
2020/05/11 17:51:55 Allowing required traffic
2020/05/11 17:51:59 All requried traffic allowed
2020/05/11 17:51:59 Removing old SG rules
2020/05/11 17:51:59 Removing decommisioned rule 7c4f4248-57a2-41fd-a669-8cd394d2618d (10.196.0.0/16, 0, 0, ) from SG 2d1b2f73-8ab5-4ebb-9e7c-10b79025af06
2020/05/11 17:51:59 Removing decommisioned rule 32958782-6ccf-44f5-98ad-3d871cfde878 (172.30.0.0/15, 2379, 2380, tcp) from SG 4ca131eb-232f-46a5-9561-e78f5eba84fc
2020/05/11 17:51:59 Removing decommisioned rule 8a39f5cb-a06a-4452-a9a6-9f5860b143a5 (10.128.0.0/14, 0, 0, ) from SG 4ca131eb-232f-46a5-9561-e78f5eba84fc
2020/05/11 17:51:59 Removing decommisioned rule da384772-745b-4c59-b9d3-66a54aac5deb (10.128.0.0/14, 0, 0, ) from SG ac5d2812-e4d1-4bf4-b7a4-d2dbd34d1caf
2020/05/11 17:51:59 All old SG rules removed
...

The CNO log shows the security rules that have been deleted from the worker's, master's and kuryr pods security groups.


$ openstack security group rule list --ingress 2d1b2f73-8ab5-4ebb-9e7c-10b79025af06
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
| ID                                   | IP Protocol | Ethertype | IP Range  | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
| f88d2421-12df-4046-94f4-892590a39876 | None        | IPv4      | 0.0.0.0/0 |            | None                  |
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+


$ openstack security group rule list --ingress 4ca131eb-232f-46a5-9561-e78f5eba84fc
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
| ID                                   | IP Protocol | Ethertype | IP Range      | Port Range  | Remote Security Group |
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
| 04f28411-a9b1-4670-86e9-fb3e6a4c9d1a | tcp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
| 0d2eddc0-6181-4341-abfb-e993c361d344 | udp         | IPv4      | 10.196.0.0/16 | 4789:4789   | None                  |
| 144f4a13-d366-471b-8d46-7bd5c3e44bbc | tcp         | IPv4      | 0.0.0.0/0     | 6443:6443   | None                  |
| 1656a0c5-cb35-412c-ac49-f7d05fa99786 | tcp         | IPv4      | 0.0.0.0/0     | 22:22       | None                  |
| 277d3894-a535-4907-a813-14654fef5e2e | tcp         | IPv4      | 10.196.0.0/16 | 2379:2380   | None                  |
| 2a52efff-3627-46a8-b5ea-d2070bd09d21 | udp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
| 3af23d20-778f-43b6-bed6-fd702839b6ab | tcp         | IPv4      | 10.196.0.0/16 | 6641:6642   | None                  |
| 3b958552-266f-4dc4-83e6-12996126ae86 | tcp         | IPv4      | 172.30.0.0/15 | 2379:2379   | None                  |
| 3e64e938-798b-46b4-a341-2216d482a2cd | tcp         | IPv4      | 10.196.0.0/16 | 10257:10257 | None                  |
| 417b46cf-26dc-4a1e-b53d-45386ae7d071 | tcp         | IPv4      | 10.128.0.0/14 | 2379:2379   | None                  |
| 58403550-1189-4a7d-b761-304374352236 | udp         | IPv4      | 10.196.0.0/16 | 53:53       | None                  |
| 61cf968d-b805-4237-825a-ebef744b1cf5 | tcp         | IPv4      | 172.30.0.0/15 | 6443:6443   | None                  |
| 701fd3f8-6a85-4259-b19c-ee6b33af4f67 | tcp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
| 76e6a478-9e56-4b84-9b6a-058c8cbc26b0 | tcp         | IPv4      | 10.196.0.0/16 | 22623:22623 | None                  |
| 7b53002d-b3c1-4d39-8803-76730ddaa8af | tcp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
| 7ca3c623-fcc2-4b56-93bf-cb4eeb7c4876 | tcp         | IPv4      | 10.128.0.0/14 | 10259:10259 | None                  |
| 977be9f8-4d3d-4368-b217-b46f3f626b2e | udp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
| 99fb3b76-055d-42db-80e2-2a03b87ed461 | tcp         | IPv4      | 10.128.0.0/14 | 10257:10257 | None                  |
| 9f18066d-8150-47af-95c2-c12535461bf1 | udp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
| aa4e263d-1d00-4845-8711-2dfce41b1a82 | icmp        | IPv4      | 0.0.0.0/0     |             | None                  |
| b0b76dcf-8fd1-406a-981a-b94342ae3b8f | udp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
| b49ace68-7c93-428b-9450-f6b237954184 | tcp         | IPv4      | 10.196.0.0/16 | 53:53       | None                  |
| b9fd6157-3155-43c4-8581-fa9927f13bdc | udp         | IPv4      | 10.196.0.0/16 | 5353:5353   | None                  |
| c0a0bc0e-456f-408b-b43d-3bbc7c978cd5 | udp         | IPv4      | 10.196.0.0/16 | 6081:6081   | None                  |
| c61cfcc6-59e8-4ec3-939a-08dbe438d73f | tcp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
| e7f57f3c-608a-41d5-b10a-e8231c435e0d | vrrp        | IPv4      | 10.196.0.0/16 |             | None                  |
| eb0c84f7-d282-45e0-806a-02e982bfe7ce | tcp         | IPv4      | 10.128.0.0/14 | 10250:10250 | None                  |
| f286d560-deab-4536-b06b-1db96d3d0ab0 | tcp         | IPv4      | 10.196.0.0/16 | 10250:10250 | None                  |
| ffe5cc7e-068e-43c9-9490-b66c29e7a5a5 | tcp         | IPv4      | 10.196.0.0/16 | 10259:10259 | None                  |
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+


$ openstack security group rule list --ingress ac5d2812-e4d1-4bf4-b7a4-d2dbd34d1caf
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
| ID                                   | IP Protocol | Ethertype | IP Range      | Port Range  | Remote Security Group |
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
| 195dfb59-d162-4bc5-80c2-533c0730b93e | tcp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
| 2a203e59-6c8f-4152-a253-77579b48eea2 | udp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
| 3baa6055-4b15-4cd6-a304-413644e47cd4 | tcp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
| 4e6b43b5-2b85-47b6-94a6-5913029612f2 | tcp         | IPv4      | 10.128.0.0/14 | 10250:10250 | None                  |
| 5c6a921b-e271-45c3-960a-1b8359c02d73 | tcp         | IPv4      | 0.0.0.0/0     | 443:443     | None                  |
| 62275688-1726-43a6-8614-a6298d353b42 | tcp         | IPv4      | 0.0.0.0/0     | 80:80       | None                  |
| 80fa5323-7b3c-4460-af23-628f5afdc0c7 | tcp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
| 8771bdae-6ec5-4073-b138-6f039e8dac0e | udp         | IPv4      | 10.196.0.0/16 | 4789:4789   | None                  |
| 8908c07b-a56c-4b00-b166-2a45529592aa | udp         | IPv4      | 10.196.0.0/16 | 5353:5353   | None                  |
| 91b5f72e-c5c3-4be1-8d75-34c4eeee9832 | tcp         | IPv4      | 0.0.0.0/0     | 22:22       | None                  |
| 9b9fda61-8790-4aa0-a9fa-44dc84233a31 | udp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
| ad9a0c03-fff3-4e82-8115-170d31092517 | udp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
| b3cbc01f-99f1-4b4c-8b90-1a3ff61cf1eb | tcp         | IPv4      | 10.128.0.0/14 | 1936:1936   | None                  |
| c00508df-c11e-4813-9609-9814d1cbf067 | tcp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
| cb51c86d-ee2a-4749-b042-cd31a2875bc3 | udp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
| d55132e3-8e3e-4556-965e-cc599afa97f4 | tcp         | IPv4      | 10.196.0.0/16 | 1936:1936   | None                  |
| e4367427-dadd-4c2c-8d31-3001bf7022ca | icmp        | IPv4      | 0.0.0.0/0     |             | None                  |
| e46ba636-82ad-4142-93a4-6e82b431c11f | vrrp        | IPv4      | 10.196.0.0/16 |             | None                  |
| ea42710e-4475-4f0a-9b27-4fde8ddc1edd | tcp         | IPv4      | 10.196.0.0/16 | 10250:10250 | None                  |
| ee114ca9-87de-4ac5-8fee-472e7b733941 | udp         | IPv4      | 10.196.0.0/16 | 6081:6081   | None                  |
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+

Comment 5 errata-xmlrpc 2020-05-18 13:35:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2133