1832899 – Old SG rules created by CNO on Kuryr bootstrap not removed on upgrade

Bug 1832899 - Old SG rules created by CNO on Kuryr bootstrap not removed on upgrade

Summary: Old SG rules created by CNO on Kuryr bootstrap not removed on upgrade

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.4
Hardware:	All
OS:	Unspecified
Priority:	urgent
Severity:	high
Target Milestone:	---
Target Release:	4.4.z
Assignee:	Michał Dulko
QA Contact:	Jon Uriarte
Docs Contact:
URL:
Whiteboard:
Depends On:	1832305
Blocks:	1834858
TreeView+	depends on / blocked

Reported:	2020-05-07 12:57 UTC by Michał Dulko
Modified:	2020-05-20 19:57 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: cluster-network-operator on Kuryr bootstrapping had no logic to remove deprecated security group rules when they get replaced by new ones. Consequence: On OCP upgrade the old SG rules were left on the SGs meaning that tightening them to increase security was not done on environments upgraded from 4.3 to 4.4. Fix: The fix is to make sure CNO is removing old security group rules. Result: The SG rules get removed, on 4.3->4.4 upgrade pods are correctly getting the access to host VMs restricted.
Clone Of:	1832305
Clones:	1834858 (view as bug list)
Environment:
Last Closed:	2020-05-18 13:35:03 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift cluster-network-operator pull 623	0	None	closed	Bug 1832899: Kuryr: Remove old SG rules on upgrade	2020-05-18 08:00:45 UTC
Red Hat Product Errata	RHBA-2020:2133	0	None	None	None	2020-05-18 13:35:19 UTC

Description Michał Dulko 2020-05-07 12:57:17 UTC

+++ This bug was initially created as a clone of Bug #1832305 +++

Description of problem:
In 4.4 the set of rules created on master and workers SG for Kuryr to work by CNO was updated to be more strict. The problem is that on upgrade old rules are not removed, so any "tightening" doesn't have affect as, less "tight" rules still exist.

Version-Release number of selected component (if applicable):


How reproducible:
Always on upgrade

Steps to Reproduce:
1. Deploy 4.3 with Kuryr
2. Upgrade to 4.4

Actual results:
On masters and workers SG there will be rules opening traffic from pod subnet on all ports.

Expected results:
Those rules should be gone, replaced by rules only opening specific ports for traffic from pod subnet.

Additional info:
The workaround is to manually remove offending rules on upgrade.

Comment 3 Jon Uriarte 2020-05-11 18:01:34 UTC

Verified in 4.4.0-0.nightly-2020-05-08-224132 on top of OSP 16 compose RHOS_TRUNK-16.0-RHEL-8-20200506.n.2 (with OVS and amphora-driver).

In order to verify this BZ the cno image has been updated to a 4.4 one that includes the fix.

$ oc scale deploy -n openshift-cluster-version cluster-version-operator --replicas 0

$ 4.4.0-0.nightly-2020-05-08-224132/openshift-install version
4.4.0-0.nightly-2020-05-08-224132/openshift-install 4.4.0-0.nightly-2020-05-08-224132
built from commit 0c7f28d8642b8e9bf330038423120c5cf6a4bf91
release image registry.svc.ci.openshift.org/ocp/release@sha256:eec18530cd4777dcd9965083b1114a826ccaf1966bbbfb324afed79b0f8c6182

$ sudo podman pull --authfile secret.json registry.svc.ci.openshift.org/ocp/release@sha256:eec18530cd4777dcd9965083b1114a826ccaf1966bbbfb324afed79b0f8c6182
$ sudo podman run --network none --rm -it registry.svc.ci.openshift.org/ocp/release@sha256:eec18530cd4777dcd9965083b1114a826ccaf1966bbbfb324afed79b0f8c6182 image cluster-network-operator
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d04a1399063458596b789e5a7219b4613870e1b285a647539d93d6ebaf6542d7



$ openstack subnet pool list
+--------------------------------------+-----------------------------------+---------------+
| ID                                   | Name                              | Prefixes      |
+--------------------------------------+-----------------------------------+---------------+
| cb2bcfbc-e8e3-4cf9-9254-435ce010b9f2 | ostest-4bwnq-kuryr-pod-subnetpool | 10.128.0.0/14 |
+--------------------------------------+-----------------------------------+---------------+

$ openstack subnet list | grep ostest
| c89265bb-0e9a-4015-9c1a-7ab9b96f48f3 | ostest-4bwnq-kuryr-service-subnet                               | 1154692a-353b-4f86-ace7-aafe203d5a4f | 172.30.0.0/15   |
| e02edda1-62f8-42b6-8eac-6f036c57da8e | ostest-4bwnq-nodes                                              | 958c0181-7edf-47a5-8349-229cbedbf97a | 10.196.0.0/16   |


$ openstack security group list
+--------------------------------------+----------------------------------------+------------------------+----------------------------------+-------------------------------------+
| ID                                   | Name                                   | Description            | Project                          | Tags                                |
+--------------------------------------+----------------------------------------+------------------------+----------------------------------+-------------------------------------+
| 17335149-dbc1-48db-ac70-de10ed4b2528 | default                                | Default security group | 74e7ae11d5614856b968999e8dfcd0f1 | []                                  |
| 2d1b2f73-8ab5-4ebb-9e7c-10b79025af06 | ostest-4bwnq-kuryr-pods-security-group |                        | 74e7ae11d5614856b968999e8dfcd0f1 | ['openshiftClusterID=ostest-4bwnq'] |
| 4ca131eb-232f-46a5-9561-e78f5eba84fc | ostest-4bwnq-master                    |                        | 74e7ae11d5614856b968999e8dfcd0f1 | ['openshiftClusterID=ostest-4bwnq'] |
| ac5d2812-e4d1-4bf4-b7a4-d2dbd34d1caf | ostest-4bwnq-worker                    |                        | 74e7ae11d5614856b968999e8dfcd0f1 | ['openshiftClusterID=ostest-4bwnq'] |
+--------------------------------------+----------------------------------------+------------------------+----------------------------------+-------------------------------------+


Run the CNO from 4.4 image:
$ oc -n openshift-network-operator edit deploy network-operator

     image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d04a1399063458596b789e5a7219b4613870e1b285a647539d93d6ebaf6542d7  <<<<<<<


$ oc -n openshift-network-operator get pods
NAME                                READY   STATUS    RESTARTS   AGE
network-operator-59b69dd7c-ljd4v   1/1     Running   0          60s


$ oc -n openshift-network-operator logs -f network-operator-59b69dd7c-ljd4v
...                                                                                 
2020/05/11 17:51:55 Found master nodes security group 4ca131eb-232f-46a5-9561-e78f5eba84fc
2020/05/11 17:51:55 Found worker nodes security group ac5d2812-e4d1-4bf4-b7a4-d2dbd34d1caf
2020/05/11 17:51:55 Ensuring pods security group
2020/05/11 17:51:55 Pods security group 2d1b2f73-8ab5-4ebb-9e7c-10b79025af06 present
2020/05/11 17:51:55 Allowing required traffic
2020/05/11 17:51:59 All requried traffic allowed
2020/05/11 17:51:59 Removing old SG rules
2020/05/11 17:51:59 Removing decommisioned rule 7c4f4248-57a2-41fd-a669-8cd394d2618d (10.196.0.0/16, 0, 0, ) from SG 2d1b2f73-8ab5-4ebb-9e7c-10b79025af06
2020/05/11 17:51:59 Removing decommisioned rule 32958782-6ccf-44f5-98ad-3d871cfde878 (172.30.0.0/15, 2379, 2380, tcp) from SG 4ca131eb-232f-46a5-9561-e78f5eba84fc
2020/05/11 17:51:59 Removing decommisioned rule 8a39f5cb-a06a-4452-a9a6-9f5860b143a5 (10.128.0.0/14, 0, 0, ) from SG 4ca131eb-232f-46a5-9561-e78f5eba84fc
2020/05/11 17:51:59 Removing decommisioned rule da384772-745b-4c59-b9d3-66a54aac5deb (10.128.0.0/14, 0, 0, ) from SG ac5d2812-e4d1-4bf4-b7a4-d2dbd34d1caf
2020/05/11 17:51:59 All old SG rules removed
...

The CNO log shows the security rules that have been deleted from the worker's, master's and kuryr pods security groups.


$ openstack security group rule list --ingress 2d1b2f73-8ab5-4ebb-9e7c-10b79025af06
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
| ID                                   | IP Protocol | Ethertype | IP Range  | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
| f88d2421-12df-4046-94f4-892590a39876 | None        | IPv4      | 0.0.0.0/0 |            | None                  |
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+


$ openstack security group rule list --ingress 4ca131eb-232f-46a5-9561-e78f5eba84fc
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
| ID                                   | IP Protocol | Ethertype | IP Range      | Port Range  | Remote Security Group |
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
| 04f28411-a9b1-4670-86e9-fb3e6a4c9d1a | tcp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
| 0d2eddc0-6181-4341-abfb-e993c361d344 | udp         | IPv4      | 10.196.0.0/16 | 4789:4789   | None                  |
| 144f4a13-d366-471b-8d46-7bd5c3e44bbc | tcp         | IPv4      | 0.0.0.0/0     | 6443:6443   | None                  |
| 1656a0c5-cb35-412c-ac49-f7d05fa99786 | tcp         | IPv4      | 0.0.0.0/0     | 22:22       | None                  |
| 277d3894-a535-4907-a813-14654fef5e2e | tcp         | IPv4      | 10.196.0.0/16 | 2379:2380   | None                  |
| 2a52efff-3627-46a8-b5ea-d2070bd09d21 | udp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
| 3af23d20-778f-43b6-bed6-fd702839b6ab | tcp         | IPv4      | 10.196.0.0/16 | 6641:6642   | None                  |
| 3b958552-266f-4dc4-83e6-12996126ae86 | tcp         | IPv4      | 172.30.0.0/15 | 2379:2379   | None                  |
| 3e64e938-798b-46b4-a341-2216d482a2cd | tcp         | IPv4      | 10.196.0.0/16 | 10257:10257 | None                  |
| 417b46cf-26dc-4a1e-b53d-45386ae7d071 | tcp         | IPv4      | 10.128.0.0/14 | 2379:2379   | None                  |
| 58403550-1189-4a7d-b761-304374352236 | udp         | IPv4      | 10.196.0.0/16 | 53:53       | None                  |
| 61cf968d-b805-4237-825a-ebef744b1cf5 | tcp         | IPv4      | 172.30.0.0/15 | 6443:6443   | None                  |
| 701fd3f8-6a85-4259-b19c-ee6b33af4f67 | tcp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
| 76e6a478-9e56-4b84-9b6a-058c8cbc26b0 | tcp         | IPv4      | 10.196.0.0/16 | 22623:22623 | None                  |
| 7b53002d-b3c1-4d39-8803-76730ddaa8af | tcp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
| 7ca3c623-fcc2-4b56-93bf-cb4eeb7c4876 | tcp         | IPv4      | 10.128.0.0/14 | 10259:10259 | None                  |
| 977be9f8-4d3d-4368-b217-b46f3f626b2e | udp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
| 99fb3b76-055d-42db-80e2-2a03b87ed461 | tcp         | IPv4      | 10.128.0.0/14 | 10257:10257 | None                  |
| 9f18066d-8150-47af-95c2-c12535461bf1 | udp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
| aa4e263d-1d00-4845-8711-2dfce41b1a82 | icmp        | IPv4      | 0.0.0.0/0     |             | None                  |
| b0b76dcf-8fd1-406a-981a-b94342ae3b8f | udp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
| b49ace68-7c93-428b-9450-f6b237954184 | tcp         | IPv4      | 10.196.0.0/16 | 53:53       | None                  |
| b9fd6157-3155-43c4-8581-fa9927f13bdc | udp         | IPv4      | 10.196.0.0/16 | 5353:5353   | None                  |
| c0a0bc0e-456f-408b-b43d-3bbc7c978cd5 | udp         | IPv4      | 10.196.0.0/16 | 6081:6081   | None                  |
| c61cfcc6-59e8-4ec3-939a-08dbe438d73f | tcp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
| e7f57f3c-608a-41d5-b10a-e8231c435e0d | vrrp        | IPv4      | 10.196.0.0/16 |             | None                  |
| eb0c84f7-d282-45e0-806a-02e982bfe7ce | tcp         | IPv4      | 10.128.0.0/14 | 10250:10250 | None                  |
| f286d560-deab-4536-b06b-1db96d3d0ab0 | tcp         | IPv4      | 10.196.0.0/16 | 10250:10250 | None                  |
| ffe5cc7e-068e-43c9-9490-b66c29e7a5a5 | tcp         | IPv4      | 10.196.0.0/16 | 10259:10259 | None                  |
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+


$ openstack security group rule list --ingress ac5d2812-e4d1-4bf4-b7a4-d2dbd34d1caf
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
| ID                                   | IP Protocol | Ethertype | IP Range      | Port Range  | Remote Security Group |
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
| 195dfb59-d162-4bc5-80c2-533c0730b93e | tcp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
| 2a203e59-6c8f-4152-a253-77579b48eea2 | udp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
| 3baa6055-4b15-4cd6-a304-413644e47cd4 | tcp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
| 4e6b43b5-2b85-47b6-94a6-5913029612f2 | tcp         | IPv4      | 10.128.0.0/14 | 10250:10250 | None                  |
| 5c6a921b-e271-45c3-960a-1b8359c02d73 | tcp         | IPv4      | 0.0.0.0/0     | 443:443     | None                  |
| 62275688-1726-43a6-8614-a6298d353b42 | tcp         | IPv4      | 0.0.0.0/0     | 80:80       | None                  |
| 80fa5323-7b3c-4460-af23-628f5afdc0c7 | tcp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
| 8771bdae-6ec5-4073-b138-6f039e8dac0e | udp         | IPv4      | 10.196.0.0/16 | 4789:4789   | None                  |
| 8908c07b-a56c-4b00-b166-2a45529592aa | udp         | IPv4      | 10.196.0.0/16 | 5353:5353   | None                  |
| 91b5f72e-c5c3-4be1-8d75-34c4eeee9832 | tcp         | IPv4      | 0.0.0.0/0     | 22:22       | None                  |
| 9b9fda61-8790-4aa0-a9fa-44dc84233a31 | udp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
| ad9a0c03-fff3-4e82-8115-170d31092517 | udp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
| b3cbc01f-99f1-4b4c-8b90-1a3ff61cf1eb | tcp         | IPv4      | 10.128.0.0/14 | 1936:1936   | None                  |
| c00508df-c11e-4813-9609-9814d1cbf067 | tcp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
| cb51c86d-ee2a-4749-b042-cd31a2875bc3 | udp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
| d55132e3-8e3e-4556-965e-cc599afa97f4 | tcp         | IPv4      | 10.196.0.0/16 | 1936:1936   | None                  |
| e4367427-dadd-4c2c-8d31-3001bf7022ca | icmp        | IPv4      | 0.0.0.0/0     |             | None                  |
| e46ba636-82ad-4142-93a4-6e82b431c11f | vrrp        | IPv4      | 10.196.0.0/16 |             | None                  |
| ea42710e-4475-4f0a-9b27-4fde8ddc1edd | tcp         | IPv4      | 10.196.0.0/16 | 10250:10250 | None                  |
| ee114ca9-87de-4ac5-8fee-472e7b733941 | udp         | IPv4      | 10.196.0.0/16 | 6081:6081   | None                  |
+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+

Comment 5 errata-xmlrpc 2020-05-18 13:35:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2133

Note You need to log in before you can comment on or make changes to this bug.