Bug 1832305
Summary: | Old SG rules created by CNO on Kuryr bootstrap not removed on upgrade | |||
---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Michał Dulko <mdulko> | |
Component: | Networking | Assignee: | Michał Dulko <mdulko> | |
Networking sub component: | kuryr | QA Contact: | Jon Uriarte <juriarte> | |
Status: | CLOSED ERRATA | Docs Contact: | ||
Severity: | high | |||
Priority: | unspecified | CC: | jshepherd, juriarte, ltomasbo, rlobillo | |
Version: | 4.4 | Keywords: | AutomationBackLog, UpcomingSprint | |
Target Milestone: | --- | |||
Target Release: | 4.5.0 | |||
Hardware: | All | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | Bug Fix | ||
Doc Text: |
Cause: cluster-network-operator on Kuryr bootstrapping had no logic to remove deprecated security group rules when they get replaced by new ones.
Consequence: On OCP upgrade the old SG rules were left on the SGs meaning that tightening them to increase security was not done on environments upgraded from 4.3 to 4.4.
Fix: The fix is to make sure CNO is removing old security group rules.
Result: The SG rules get removed, on 4.3->4.4 upgrade pods are correctly getting the access to host VMs restricted.
|
Story Points: | --- | |
Clone Of: | ||||
: | 1832899 (view as bug list) | Environment: | ||
Last Closed: | 2020-07-13 17:35:31 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1832899 |
Description
Michał Dulko
2020-05-06 13:56:24 UTC
Verified in 4.5.0-0.nightly-2020-05-08-060557 on top of OSP 13 2020-04-01.3 puddle. In order to verify this BZ the cno image has been updated to a 4.5 one that includes the fix. $ oc scale deploy -n openshift-cluster-version cluster-version-operator --replicas 0 $ 4.5.0-0.nightly-2020-05-08-060557/openshift-install version 4.5.0-0.nightly-2020-05-08-060557/openshift-install 4.5.0-0.nightly-2020-05-08-060557 built from commit fca033874fd4cd5b0d184cccf981dbd3187c1f6c release image registry.svc.ci.openshift.org/ocp/release@sha256:9281b80d1a5579887ad8bc432f9111f411675153d78b12d015b44d5bab2881ac $ docker pull registry.svc.ci.openshift.org/ocp/release@sha256:9281b80d1a5579887ad8bc432f9111f411675153d78b12d015b44d5bab2881ac $ docker run -it registry.svc.ci.openshift.org/ocp/release@sha256:9281b80d1a5579887ad8bc432f9111f411675153d78b12d015b44d5bab2881ac image cluster-network-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5836903c789e6c45ee611ecd0afb312e4bf19f502fa77f24b4dfe6fe223709a3 $ openstack subnet pool list +--------------------------------------+-----------------------------------+---------------+ | ID | Name | Prefixes | +--------------------------------------+-----------------------------------+---------------+ | 3970b6e8-d16c-4299-be1b-219001987b06 | ostest-xdr85-kuryr-pod-subnetpool | 10.128.0.0/14 | +--------------------------------------+-----------------------------------+---------------+ $ openstack subnet list | grep ostest | 352dd4ac-b006-4501-b5c1-02bf05e84b3d | ostest-xdr85-nodes | 126ecc35-401e-411e-8453-5ed9e2ab18e2 | 10.196.0.0/16 | | b1ec6731-50bc-47d9-b1ce-cc2a456d3977 | ostest-xdr85-kuryr-service-subnet | 28236289-99ce-4d31-819e-26475c8f50c5 | 172.30.0.0/15 | $ openstack security group list +--------------------------------------+----------------------------------------+------------------------+----------------------------------+ | ID | Name | Description | Project | +--------------------------------------+----------------------------------------+------------------------+----------------------------------+ | 744ed27d-5789-46b1-9c96-0838704190b3 | ostest-xdr85-worker | | 371b4b309c8b47d4aad01cee1430016b | | d9bd119a-a64b-43cc-8c8e-b260a8f73d21 | ostest-xdr85-master | | 371b4b309c8b47d4aad01cee1430016b | | f15f847d-e007-4845-8446-876e8d1d75bb | ostest-xdr85-kuryr-pods-security-group | | 371b4b309c8b47d4aad01cee1430016b | | fc97c9aa-d354-44c6-950c-6a118a47d382 | default | Default security group | 371b4b309c8b47d4aad01cee1430016b | +--------------------------------------+----------------------------------------+------------------------+----------------------------------+ Run the CNO from 4.5 image: $ oc -n openshift-network-operator edit deploy network-operator image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5836903c789e6c45ee611ecd0afb312e4bf19f502fa77f24b4dfe6fe223709a3 <<<<<<< $ oc -n openshift-network-operator get pods NAME READY STATUS RESTARTS AGE network-operator-77fb74dc58-95n8p 1/1 Running 0 32m $ oc -n openshift-network-operator logs -f network-operator-77fb74dc58-95n8p ... 2020/05/08 13:13:21 Found master nodes security group d9bd119a-a64b-43cc-8c8e-b260a8f73d21 2020/05/08 13:13:22 Found worker nodes security group 744ed27d-5789-46b1-9c96-0838704190b3 2020/05/08 13:13:22 Ensuring pods security group 2020/05/08 13:13:22 Pods security group f15f847d-e007-4845-8446-876e8d1d75bb present 2020/05/08 13:13:22 Allowing required traffic 2020/05/08 13:13:32 All requried traffic allowed 2020/05/08 13:13:32 Removing old SG rules 2020/05/08 13:13:33 Removing decommisioned rule 795e9c3c-c4f7-4e48-8699-08862baa26cd (10.128.0.0/14, 0, 0, ) from SG 744ed27d-5789-46b1-9c96-0838704190b3 2020/05/08 13:13:33 Removing decommisioned rule 97056ecd-7434-4764-b2b1-e39aa17740c4 (172.30.0.0/15, 2379, 2380, tcp) from SG d9bd119a-a64b-43cc-8c8e-b260a8f73d21 2020/05/08 13:13:33 Removing decommisioned rule eb3e8eba-a51c-4628-b3d5-c9e343457004 (10.128.0.0/14, 0, 0, ) from SG d9bd119a-a64b-43cc-8c8e-b260a8f73d21 2020/05/08 13:13:34 Removing decommisioned rule 9e747d94-79f9-4aa8-b87a-e1b5580efc5d (10.196.0.0/16, 0, 0, ) from SG f15f847d-e007-4845-8446-876e8d1d75bb 2020/05/08 13:13:34 All old SG rules removed ... The CNO log shows the security rules that have been deleted from the worker's, master's and kuryr pods security groups. $ openstack security group rule list --ingress f15f847d-e007-4845-8446-876e8d1d75bb +--------------------------------------+-------------+-----------+------------+-----------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-----------+------------+-----------------------+ | 5c2561eb-299c-4ab9-a5d9-06e2115c5020 | None | 0.0.0.0/0 | | None | +--------------------------------------+-------------+-----------+------------+-----------------------+ $ openstack security group rule list --ingress d9bd119a-a64b-43cc-8c8e-b260a8f73d21 +--------------------------------------+-------------+---------------+-------------+-----------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+---------------+-------------+-----------------------+ | 07af3300-9e07-4792-9352-6f16f6d949a6 | tcp | 10.196.0.0/16 | 10257:10257 | None | | 17dfb658-6561-47b3-af1c-5cd3ee01c833 | tcp | 172.30.0.0/15 | 6443:6443 | None | | 20b008c2-b7e1-402e-92e8-b6dcc9446806 | icmp | 0.0.0.0/0 | | None | | 20dd5885-3610-4b4e-976c-5ac810b80765 | tcp | 10.196.0.0/16 | 30000:32767 | None | | 25b75364-1c05-45e1-ba25-bfc7918e290d | tcp | 0.0.0.0/0 | 22:22 | None | | 2632a855-f2cf-4856-aed2-bd0919069ae9 | tcp | 0.0.0.0/0 | 6443:6443 | None | | 2eab3e89-5442-4782-8599-43223b33bac2 | tcp | 10.128.0.0/14 | 10250:10250 | None | | 2ed03a55-3a12-46b5-82f1-bec8d29f96c9 | udp | 10.128.0.0/14 | 9000:9999 | None | | 313b7c5c-da79-49d4-9291-d1dd853709be | tcp | 10.196.0.0/16 | 10259:10259 | None | | 34ae6608-e33b-40a1-afe1-df6abdebb4d3 | udp | 10.196.0.0/16 | 5353:5353 | None | | 358682fa-8847-4146-9f22-c051c9d7a1fa | tcp | 10.196.0.0/16 | 9000:9999 | None | | 3bb088aa-7dde-47d4-9f47-2cefc70cbc9d | udp | 10.128.0.0/14 | 53:53 | None | | 3daf295b-41a2-4a6f-b91a-119f565d63e3 | tcp | 10.128.0.0/14 | 10257:10257 | None | | 3fe37653-c632-4929-830a-7f65fc8e05f1 | udp | 10.196.0.0/16 | 30000:32767 | None | | 42b675ec-f0d3-4967-afb2-20026930336d | tcp | 10.128.0.0/14 | | None | | 787ca8e6-85a5-4154-b55f-ae0c87e9fbcf | tcp | 10.196.0.0/16 | 2379:2380 | None | | 830375c2-75a5-4bf8-acf9-a4e94f2ac217 | udp | 10.196.0.0/16 | 53:53 | None | | 9cb446ff-2ce8-4c98-9afe-1714abf6ee2c | tcp | 10.128.0.0/14 | 53:53 | None | | ab93c6cf-bc2c-4e32-9e31-c7e222829371 | tcp | 10.196.0.0/16 | 6641:6642 | None | | ada7a367-eb6b-4189-8cb3-87af45897280 | tcp | 172.30.0.0/15 | 2379:2379 | None | | c127a2da-b9cf-4f13-8b8a-ab4b693948c8 | udp | 10.196.0.0/16 | 9000:9999 | None | | c3c13297-4791-45d9-8d6a-de59e8d978ca | vrrp | 10.196.0.0/16 | | None | | caa03cd7-08e7-4d39-b648-9fa68b566e07 | tcp | 10.128.0.0/14 | 10259:10259 | None | | e296a1c0-a0a4-45bf-ae7c-84ffb39d396f | tcp | 10.196.0.0/16 | 22623:22623 | None | | e7b5a37f-efaa-49fb-ac01-4b65cee6e54d | tcp | 10.128.0.0/14 | 2379:2379 | None | | efee2c5e-3b75-4bce-9dd1-87e048d425aa | udp | 10.196.0.0/16 | 4789:4789 | None | | f02880ff-458d-4ee8-bfd2-276e51831ba8 | tcp | 10.196.0.0/16 | 53:53 | None | | f88e69e0-a415-4aa7-857e-7af3e4f29c1d | tcp | 10.128.0.0/14 | 9000:9999 | None | | f938026c-0366-49fd-ae83-e578190ee5c8 | udp | 10.196.0.0/16 | 6081:6081 | None | | fdb964c7-9a21-4d38-a780-0d1fcfe8b868 | tcp | 10.196.0.0/16 | 10250:10250 | None | +--------------------------------------+-------------+---------------+-------------+-----------------------+ $ openstack security group rule list --ingress 744ed27d-5789-46b1-9c96-0838704190b3 +--------------------------------------+-------------+---------------+-------------+-----------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+---------------+-------------+-----------------------+ | 0bec190f-df98-4f72-94e5-8ea6ef8eb95d | udp | 10.128.0.0/14 | 9000:9999 | None | | 0c36070f-0121-4721-a9ab-accabf9d3658 | udp | 10.196.0.0/16 | 6081:6081 | None | | 264df5f6-9c2d-48e3-88e2-cad5e20b2738 | tcp | 10.128.0.0/14 | 9000:9999 | None | | 2cdc62c4-a77a-4f6e-a391-fe01cf40fb92 | udp | 10.196.0.0/16 | 9000:9999 | None | | 2d6f6918-b031-47bd-83b6-7cd4fe380d43 | tcp | 10.196.0.0/16 | 10250:10250 | None | | 2d766718-76fe-4807-bd93-1be613295de8 | tcp | 0.0.0.0/0 | 80:80 | None | | 353369ae-b999-4917-bff6-c907e386e285 | udp | 10.196.0.0/16 | 4789:4789 | None | | 3c7db47b-1de1-4f01-8eea-791f61869379 | tcp | 10.128.0.0/14 | 10250:10250 | None | | 61d227e8-905d-44c0-b83b-be315b2c92f2 | tcp | 10.196.0.0/16 | 9000:9999 | None | | 6726b0f0-fa5a-40df-b26e-fa89cc3f2b9e | tcp | 0.0.0.0/0 | 22:22 | None | | 69c87281-3f1d-4589-a705-02c51d46a80a | tcp | 0.0.0.0/0 | 443:443 | None | | 82ace9e8-3874-48c3-a715-a4ac26673c3d | udp | 10.196.0.0/16 | 30000:32767 | None | | 961b1410-7512-4dbe-8549-c929e23fb44f | udp | 10.196.0.0/16 | 5353:5353 | None | | a52e4374-e50b-456c-855e-d0a86682a357 | tcp | 10.196.0.0/16 | 30000:32767 | None | | bf2b81c4-0117-4b08-b41d-5f30bb8e950d | tcp | 10.196.0.0/16 | 1936:1936 | None | | cbd3af0c-dc16-4aef-8be8-fc4516aa5769 | tcp | 10.128.0.0/14 | 53:53 | None | | d36cff55-fb70-4aed-b086-63c17055e9ac | icmp | 0.0.0.0/0 | | None | | d708963a-a869-49cd-8db0-2a6529682ae1 | vrrp | 10.196.0.0/16 | | None | | de0fe90d-51c3-469e-83f3-9922408c6b67 | udp | 10.128.0.0/14 | 53:53 | None | | e939d765-e63e-42e1-871f-2c5404268fc5 | tcp | 10.128.0.0/14 | 1936:1936 | None | +--------------------------------------+-------------+---------------+-------------+-----------------------+ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409 |