Bug 1834858

Summary: Old SG rules created by CNO on Kuryr bootstrap not removed on upgrade
Product: OpenShift Container Platform Reporter: Michał Dulko <mdulko>
Component: NetworkingAssignee: Michał Dulko <mdulko>
Networking sub component: kuryr QA Contact: GenadiC <gcheresh>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: urgent CC: gcheresh, juriarte, ltomasbo, rlobillo
Version: 4.3.zKeywords: UpcomingSprint
Target Milestone: ---   
Target Release: 4.3.z   
Hardware: All   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: cluster-network-operator on Kuryr bootstrapping had no logic to remove deprecated security group rules when they get replaced by new ones. Consequence: On OCP upgrade the old SG rules were left on the SGs meaning that tightening them to increase security was not done on environments upgraded between 4.3.z releases. Fix: The fix is to make sure CNO is removing old security group rules. Result: The SG rules get removed, on 4.3.z upgrade pods are correctly getting the access to host VMs restricted.
 Story Points: ---
Clone Of: 1832899 Environment:
Last Closed: 2020-05-27 17:00:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1832899    
Bug Blocks:    

Comment 3 rlobillo 2020-05-20 10:02:44 UTC 
Verified in 4.3.0-0.nightly-2020-05-18-043018 on top of OSP 16 compose RHOS_TRUNK-16.0-RHEL-8-20200506.n.2

In order to verify this BZ OCP4.3.0-0.nightly-2020-05-13-181621 was installed and CNO image was updated to the one included on OCP4.3.0-0.nightly-2020-05-18-043018, which includes the fix.

[stack@undercloud-0 4.3.0-0.nightly-2020-05-18-043018]$ openshift-install version
openshift-install 4.3.0-0.nightly-2020-05-18-043018
built from commit 18f1578ec61b15ef5b19f91ae14af2f30d1251ae
release image registry.svc.ci.openshift.org/ocp/release@sha256:67ac76aea1c0072c0c03c0649e04c108a75a86db78b0ab4167ac0cac4ad8a224
[stack@undercloud-0 ~]$ sudo podman run --network none --rm -it registry.svc.ci.openshift.org/ocp/release@sha256:67ac76aea1c0072c0c03c0649e04c108a75a86db78b0ab4167ac0cac4ad8a224 image cluster-network-operator
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d10fb4d4b22b6520439aa9a681e7a888749bfbbbb35e977a96f350cca8f1718


Before updating the image:

	(shiftstack) [stack@undercloud-0 ~]$ openstack security group rule list --ingress ostest-ndmb6-master
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	| ID                                   | IP Protocol | Ethertype | IP Range      | Port Range  | Remote Security Group |
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	| 0305ff84-fd76-4dad-af34-0810f4dfc479 | udp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
	| 0fa5ab9f-5b55-4b15-bed7-112c6e475d3b | tcp         | IPv4      | 10.196.0.0/16 | 22623:22623 | None                  |
	| 1fa1da64-9724-42b4-97fc-fe27a67e91fd | tcp         | IPv4      | 10.196.0.0/16 | 10259:10259 | None                  |
	| 20c73e8d-516f-47f6-ab98-566c9ef69cae | None        | IPv4      | 10.128.0.0/14 |             | None                  |
	| 3ac791fc-9f85-4c8c-b286-ebb71164adf2 | udp         | IPv4      | 10.196.0.0/16 | 53:53       | None                  |
	| 6b6e6b29-bc84-41be-8b90-4f527fae2aa0 | tcp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
	| 6bdea740-aee0-4bcb-8b89-00e6d0e37364 | icmp        | IPv4      | 0.0.0.0/0     |             | None                  |
	| 6c916169-070f-4703-9538-72461cb5ef1d | tcp         | IPv4      | 10.196.0.0/16 | 2379:2380   | None                  |
	| 85088d85-0ee4-4e38-84f1-581de18312c2 | tcp         | IPv4      | 10.196.0.0/16 | 53:53       | None                  |
	| 87e2c968-020f-45ea-be35-3988bd5e4530 | vrrp        | IPv4      | 10.196.0.0/16 |             | None                  |
	| 9a906ad1-108b-4993-b903-13351ffab575 | tcp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
	| 9bc5dc7c-8e59-44d5-9fdc-d19ed4ee43bb | tcp         | IPv4      | 0.0.0.0/0     | 22:22       | None                  |
	| ab28c297-d62e-4e6c-abd0-46f1878771ea | tcp         | IPv4      | 0.0.0.0/0     | 6443:6443   | None                  |
	| b19cd4ed-108b-4a14-a7ca-2910460ada0b | udp         | IPv4      | 10.196.0.0/16 | 4789:4789   | None                  |
	| b403b1f7-02e0-4152-b6bb-8337adf38f40 | udp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
	| ba4129b2-a6da-4b4a-802d-7e9b1994f316 | udp         | IPv4      | 10.196.0.0/16 | 6081:6081   | None                  |
	| c8c3bd30-b1df-476d-8f53-d09778906f61 | tcp         | IPv4      | 10.196.0.0/16 | 10257:10257 | None                  |
	| cafb0ab0-067f-4ece-a416-98aae8f2724a | tcp         | IPv4      | 10.196.0.0/16 | 10250:10250 | None                  |
	| ceab2d9a-c1b9-4bac-bf62-492043418272 | tcp         | IPv4      | 172.30.0.0/15 | 2379:2380   | None                  |
	| d5224a88-1abf-4085-8e71-6fd7fa69dab2 | tcp         | IPv4      | 172.30.0.0/15 | 6443:6443   | None                  |
	| f83fc835-8853-4dfc-a79c-ad60b25c0f61 | tcp         | IPv4      | 10.196.0.0/16 | 6641:6642   | None                  |
	| fcdaa145-6dae-4153-88bb-91c8f0fe025f | udp         | IPv4      | 10.196.0.0/16 | 5353:5353   | None                  |
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	(shiftstack) [stack@undercloud-0 ~]$ openstack security group rule list --ingress ostest-ndmb6-worker
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	| ID                                   | IP Protocol | Ethertype | IP Range      | Port Range  | Remote Security Group |
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	| 0837b2e1-e069-4a83-ba85-9fd45543d17c | None        | IPv4      | 10.128.0.0/14 |             | None                  |
	| 1e733eb2-3665-4304-85e1-95101702a95a | tcp         | IPv4      | 10.196.0.0/16 | 1936:1936   | None                  |
	| 2d44f38e-65d4-4a36-81da-2e8aee5ae856 | tcp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
	| 37db8fb7-14fa-4c0c-a9cd-48753153106e | icmp        | IPv4      | 0.0.0.0/0     |             | None                  |
	| 38c3ffaf-549b-40b9-ad81-fb68b5089a09 | tcp         | IPv4      | 0.0.0.0/0     | 443:443     | None                  |
	| 3f2a7454-5bd3-4279-a1e8-b59be5b1a389 | tcp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
	| 5b96ac5d-a523-46cc-a389-7c9f5c8b9767 | udp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
	| 5ba30cb4-85a8-47b4-9e35-cf1ae86c43a4 | udp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
	| 739d532e-524d-43c5-8615-27f99571f1db | tcp         | IPv4      | 10.196.0.0/16 | 10250:10250 | None                  |
	| 9262216e-c9df-486d-a028-52921c7cdb83 | udp         | IPv4      | 10.196.0.0/16 | 6081:6081   | None                  |
	| a56a7699-78e1-4d86-9277-91d8b908c0c0 | vrrp        | IPv4      | 10.196.0.0/16 |             | None                  |
	| a8fcb252-4362-4602-93b5-cb4af0078045 | udp         | IPv4      | 10.196.0.0/16 | 5353:5353   | None                  |
	| ae0e7727-ebf9-45b8-ab50-5679e8a91485 | tcp         | IPv4      | 0.0.0.0/0     | 22:22       | None                  |
	| afa07c82-02bc-4164-9dc5-4358046a6928 | tcp         | IPv4      | 0.0.0.0/0     | 80:80       | None                  |
	| f18e0234-4273-4f59-80e9-c0b2eed38a09 | udp         | IPv4      | 10.196.0.0/16 | 4789:4789   | None                  |
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	(shiftstack) [stack@undercloud-0 ~]$ openstack security group rule list --ingress ostest-ndmb6-kuryr-pods-security-group
	+--------------------------------------+-------------+-----------+---------------+------------+-----------------------+
	| ID                                   | IP Protocol | Ethertype | IP Range      | Port Range | Remote Security Group |
	+--------------------------------------+-------------+-----------+---------------+------------+-----------------------+
	| 54f0d3b9-04c4-4e12-aa9b-5b67cae07028 | None        | IPv4      | 0.0.0.0/0     |            | None                  |
	| bc52e0c1-8092-4b87-9544-b3690ea3fc9d | None        | IPv4      | 10.196.0.0/16 |            | None                  |
	+--------------------------------------+-------------+-----------+---------------+------------+-----------------------+

	Connectivity test: 

	(shiftstack) [stack@undercloud-0 ~]$ oc rsh -n test demo1-1-2l558 curl -k --head https://10.196.3.195:22623/config/master
	HTTP/1.1 200 OK
	Content-Length: 150999
	Content-Type: application/json
	Date: Wed, 20 May 2020 08:54:48 GMT


Once updating CNO image to the one included on 4.3.0-0.nightly-2020-05-18-043018:
	$ oc scale deploy -n openshift-cluster-version cluster-version-operator --replicas 0
	$ oc -n openshift-network-operator edit deploy network-operator
	
		image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d10fb4d4b22b6520439aa9a681e7a888749bfbbbb35e977a96f350cca8f1718
	
	(shiftstack) [stack@undercloud-0 ~]$ openstack security group rule list --ingress ostest-ndmb6-master
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	| ID                                   | IP Protocol | Ethertype | IP Range      | Port Range  | Remote Security Group |
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	| 0305ff84-fd76-4dad-af34-0810f4dfc479 | udp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
	| 0fa5ab9f-5b55-4b15-bed7-112c6e475d3b | tcp         | IPv4      | 10.196.0.0/16 | 22623:22623 | None                  |
	| 1fa1da64-9724-42b4-97fc-fe27a67e91fd | tcp         | IPv4      | 10.196.0.0/16 | 10259:10259 | None                  |
	| 3ac791fc-9f85-4c8c-b286-ebb71164adf2 | udp         | IPv4      | 10.196.0.0/16 | 53:53       | None                  |
	| 5af15e32-d744-42e3-8644-54d4412f1ad5 | udp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
	| 5f08a623-a0e2-49fd-919e-d51434213943 | tcp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
	| 6b6e6b29-bc84-41be-8b90-4f527fae2aa0 | tcp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
	| 6bdea740-aee0-4bcb-8b89-00e6d0e37364 | icmp        | IPv4      | 0.0.0.0/0     |             | None                  |
	| 6c916169-070f-4703-9538-72461cb5ef1d | tcp         | IPv4      | 10.196.0.0/16 | 2379:2380   | None                  |
	| 70d31b81-5052-4e0b-80da-8668c341671e | tcp         | IPv4      | 172.30.0.0/15 | 2379:2379   | None                  |
	| 85088d85-0ee4-4e38-84f1-581de18312c2 | tcp         | IPv4      | 10.196.0.0/16 | 53:53       | None                  |
	| 8694350f-f234-420b-8e7b-b13d51b65d91 | udp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
	| 87e2c968-020f-45ea-be35-3988bd5e4530 | vrrp        | IPv4      | 10.196.0.0/16 |             | None                  |
	| 993c533f-acc0-4301-9396-9b8301cf1d2c | tcp         | IPv4      | 10.128.0.0/14 | 10257:10257 | None                  |
	| 9a906ad1-108b-4993-b903-13351ffab575 | tcp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
	| 9bc5dc7c-8e59-44d5-9fdc-d19ed4ee43bb | tcp         | IPv4      | 0.0.0.0/0     | 22:22       | None                  |
	| a13dec9f-83b9-4bb0-93ec-a66a14dd7455 | tcp         | IPv4      | 10.128.0.0/14 | 2379:2379   | None                  |
	| a3d7804f-4a8d-4f3d-b40d-473d3af0a86d | tcp         | IPv4      | 10.128.0.0/14 | 10259:10259 | None                  |
	| ab28c297-d62e-4e6c-abd0-46f1878771ea | tcp         | IPv4      | 0.0.0.0/0     | 6443:6443   | None                  |
	| ab5ce50c-261a-476e-9af1-7ce00b3aea7a | tcp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
	| b19cd4ed-108b-4a14-a7ca-2910460ada0b | udp         | IPv4      | 10.196.0.0/16 | 4789:4789   | None                  |
	| b403b1f7-02e0-4152-b6bb-8337adf38f40 | udp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
	| ba4129b2-a6da-4b4a-802d-7e9b1994f316 | udp         | IPv4      | 10.196.0.0/16 | 6081:6081   | None                  |
	| c8c3bd30-b1df-476d-8f53-d09778906f61 | tcp         | IPv4      | 10.196.0.0/16 | 10257:10257 | None                  |
	| cafb0ab0-067f-4ece-a416-98aae8f2724a | tcp         | IPv4      | 10.196.0.0/16 | 10250:10250 | None                  |
	| d5224a88-1abf-4085-8e71-6fd7fa69dab2 | tcp         | IPv4      | 172.30.0.0/15 | 6443:6443   | None                  |
	| f268bd65-667d-40fa-86af-1441eb83ee2d | tcp         | IPv4      | 10.128.0.0/14 | 10250:10250 | None                  |
	| f83fc835-8853-4dfc-a79c-ad60b25c0f61 | tcp         | IPv4      | 10.196.0.0/16 | 6641:6642   | None                  |
	| fcdaa145-6dae-4153-88bb-91c8f0fe025f | udp         | IPv4      | 10.196.0.0/16 | 5353:5353   | None                  |
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	(shiftstack) [stack@undercloud-0 ~]$ openstack security group rule list --ingress ostest-ndmb6-worker
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	| ID                                   | IP Protocol | Ethertype | IP Range      | Port Range  | Remote Security Group |
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	| 1e733eb2-3665-4304-85e1-95101702a95a | tcp         | IPv4      | 10.196.0.0/16 | 1936:1936   | None                  |
	| 2d44f38e-65d4-4a36-81da-2e8aee5ae856 | tcp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
	| 37db8fb7-14fa-4c0c-a9cd-48753153106e | icmp        | IPv4      | 0.0.0.0/0     |             | None                  |
	| 38c3ffaf-549b-40b9-ad81-fb68b5089a09 | tcp         | IPv4      | 0.0.0.0/0     | 443:443     | None                  |
	| 3f2a7454-5bd3-4279-a1e8-b59be5b1a389 | tcp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
	| 475d9fa5-b84b-4e01-a8a3-9e13d327a586 | tcp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
	| 5b96ac5d-a523-46cc-a389-7c9f5c8b9767 | udp         | IPv4      | 10.196.0.0/16 | 30000:32767 | None                  |
	| 5ba30cb4-85a8-47b4-9e35-cf1ae86c43a4 | udp         | IPv4      | 10.196.0.0/16 | 9000:9999   | None                  |
	| 6f6ffeb9-4191-422c-911c-8d5c35bf262d | udp         | IPv4      | 10.128.0.0/14 | 9000:9999   | None                  |
	| 739d532e-524d-43c5-8615-27f99571f1db | tcp         | IPv4      | 10.196.0.0/16 | 10250:10250 | None                  |
	| 79e589ed-aace-450d-af84-2d6d45335cb5 | tcp         | IPv4      | 10.128.0.0/14 | 1936:1936   | None                  |
	| 9262216e-c9df-486d-a028-52921c7cdb83 | udp         | IPv4      | 10.196.0.0/16 | 6081:6081   | None                  |
	| a56a7699-78e1-4d86-9277-91d8b908c0c0 | vrrp        | IPv4      | 10.196.0.0/16 |             | None                  |
	| a8fcb252-4362-4602-93b5-cb4af0078045 | udp         | IPv4      | 10.196.0.0/16 | 5353:5353   | None                  |
	| ad61fd63-6376-4aaf-826e-046049daabca | tcp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
	| ae0e7727-ebf9-45b8-ab50-5679e8a91485 | tcp         | IPv4      | 0.0.0.0/0     | 22:22       | None                  |
	| afa07c82-02bc-4164-9dc5-4358046a6928 | tcp         | IPv4      | 0.0.0.0/0     | 80:80       | None                  |
	| bbb58f5a-2880-4f86-a4e7-1a642c820e04 | udp         | IPv4      | 10.128.0.0/14 | 53:53       | None                  |
	| c6792ac9-4606-47d0-b8bb-557c4499d3b4 | tcp         | IPv4      | 10.128.0.0/14 | 10250:10250 | None                  |
	| f18e0234-4273-4f59-80e9-c0b2eed38a09 | udp         | IPv4      | 10.196.0.0/16 | 4789:4789   | None                  |
	+--------------------------------------+-------------+-----------+---------------+-------------+-----------------------+
	(shiftstack) [stack@undercloud-0 ~]$ openstack security group rule list --ingress ostest-ndmb6-kuryr-pods-security-group
	+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
	| ID                                   | IP Protocol | Ethertype | IP Range  | Port Range | Remote Security Group |
	+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
	| 54f0d3b9-04c4-4e12-aa9b-5b67cae07028 | None        | IPv4      | 0.0.0.0/0 |            | None                  |
	+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+

	(shiftstack) [stack@undercloud-0 ~]$ oc rsh -n test demo1-1-2l558 curl -k --head https://10.196.3.195:22623/config/master
	curl: (7) Failed to connect to 10.196.3.195 port 22623: Operation timed out
Comment 5 errata-xmlrpc 2020-05-27 17:00:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2184