Description of problem: In 4.4 the set of rules created on master and workers SG for Kuryr to work by CNO was updated to be more strict. The problem is that on upgrade old rules are not removed, so any "tightening" doesn't have affect as, less "tight" rules still exist. Version-Release number of selected component (if applicable): How reproducible: Always on upgrade Steps to Reproduce: 1. Deploy 4.3 with Kuryr 2. Upgrade to 4.4 Actual results: On masters and workers SG there will be rules opening traffic from pod subnet on all ports. Expected results: Those rules should be gone, replaced by rules only opening specific ports for traffic from pod subnet. Additional info: The workaround is to manually remove offending rules on upgrade.
Verified in 4.5.0-0.nightly-2020-05-08-060557 on top of OSP 13 2020-04-01.3 puddle. In order to verify this BZ the cno image has been updated to a 4.5 one that includes the fix. $ oc scale deploy -n openshift-cluster-version cluster-version-operator --replicas 0 $ 4.5.0-0.nightly-2020-05-08-060557/openshift-install version 4.5.0-0.nightly-2020-05-08-060557/openshift-install 4.5.0-0.nightly-2020-05-08-060557 built from commit fca033874fd4cd5b0d184cccf981dbd3187c1f6c release image registry.svc.ci.openshift.org/ocp/release@sha256:9281b80d1a5579887ad8bc432f9111f411675153d78b12d015b44d5bab2881ac $ docker pull registry.svc.ci.openshift.org/ocp/release@sha256:9281b80d1a5579887ad8bc432f9111f411675153d78b12d015b44d5bab2881ac $ docker run -it registry.svc.ci.openshift.org/ocp/release@sha256:9281b80d1a5579887ad8bc432f9111f411675153d78b12d015b44d5bab2881ac image cluster-network-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5836903c789e6c45ee611ecd0afb312e4bf19f502fa77f24b4dfe6fe223709a3 $ openstack subnet pool list +--------------------------------------+-----------------------------------+---------------+ | ID | Name | Prefixes | +--------------------------------------+-----------------------------------+---------------+ | 3970b6e8-d16c-4299-be1b-219001987b06 | ostest-xdr85-kuryr-pod-subnetpool | 10.128.0.0/14 | +--------------------------------------+-----------------------------------+---------------+ $ openstack subnet list | grep ostest | 352dd4ac-b006-4501-b5c1-02bf05e84b3d | ostest-xdr85-nodes | 126ecc35-401e-411e-8453-5ed9e2ab18e2 | 10.196.0.0/16 | | b1ec6731-50bc-47d9-b1ce-cc2a456d3977 | ostest-xdr85-kuryr-service-subnet | 28236289-99ce-4d31-819e-26475c8f50c5 | 172.30.0.0/15 | $ openstack security group list +--------------------------------------+----------------------------------------+------------------------+----------------------------------+ | ID | Name | Description | Project | +--------------------------------------+----------------------------------------+------------------------+----------------------------------+ | 744ed27d-5789-46b1-9c96-0838704190b3 | ostest-xdr85-worker | | 371b4b309c8b47d4aad01cee1430016b | | d9bd119a-a64b-43cc-8c8e-b260a8f73d21 | ostest-xdr85-master | | 371b4b309c8b47d4aad01cee1430016b | | f15f847d-e007-4845-8446-876e8d1d75bb | ostest-xdr85-kuryr-pods-security-group | | 371b4b309c8b47d4aad01cee1430016b | | fc97c9aa-d354-44c6-950c-6a118a47d382 | default | Default security group | 371b4b309c8b47d4aad01cee1430016b | +--------------------------------------+----------------------------------------+------------------------+----------------------------------+ Run the CNO from 4.5 image: $ oc -n openshift-network-operator edit deploy network-operator image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5836903c789e6c45ee611ecd0afb312e4bf19f502fa77f24b4dfe6fe223709a3 <<<<<<< $ oc -n openshift-network-operator get pods NAME READY STATUS RESTARTS AGE network-operator-77fb74dc58-95n8p 1/1 Running 0 32m $ oc -n openshift-network-operator logs -f network-operator-77fb74dc58-95n8p ... 2020/05/08 13:13:21 Found master nodes security group d9bd119a-a64b-43cc-8c8e-b260a8f73d21 2020/05/08 13:13:22 Found worker nodes security group 744ed27d-5789-46b1-9c96-0838704190b3 2020/05/08 13:13:22 Ensuring pods security group 2020/05/08 13:13:22 Pods security group f15f847d-e007-4845-8446-876e8d1d75bb present 2020/05/08 13:13:22 Allowing required traffic 2020/05/08 13:13:32 All requried traffic allowed 2020/05/08 13:13:32 Removing old SG rules 2020/05/08 13:13:33 Removing decommisioned rule 795e9c3c-c4f7-4e48-8699-08862baa26cd (10.128.0.0/14, 0, 0, ) from SG 744ed27d-5789-46b1-9c96-0838704190b3 2020/05/08 13:13:33 Removing decommisioned rule 97056ecd-7434-4764-b2b1-e39aa17740c4 (172.30.0.0/15, 2379, 2380, tcp) from SG d9bd119a-a64b-43cc-8c8e-b260a8f73d21 2020/05/08 13:13:33 Removing decommisioned rule eb3e8eba-a51c-4628-b3d5-c9e343457004 (10.128.0.0/14, 0, 0, ) from SG d9bd119a-a64b-43cc-8c8e-b260a8f73d21 2020/05/08 13:13:34 Removing decommisioned rule 9e747d94-79f9-4aa8-b87a-e1b5580efc5d (10.196.0.0/16, 0, 0, ) from SG f15f847d-e007-4845-8446-876e8d1d75bb 2020/05/08 13:13:34 All old SG rules removed ... The CNO log shows the security rules that have been deleted from the worker's, master's and kuryr pods security groups. $ openstack security group rule list --ingress f15f847d-e007-4845-8446-876e8d1d75bb +--------------------------------------+-------------+-----------+------------+-----------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-----------+------------+-----------------------+ | 5c2561eb-299c-4ab9-a5d9-06e2115c5020 | None | 0.0.0.0/0 | | None | +--------------------------------------+-------------+-----------+------------+-----------------------+ $ openstack security group rule list --ingress d9bd119a-a64b-43cc-8c8e-b260a8f73d21 +--------------------------------------+-------------+---------------+-------------+-----------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+---------------+-------------+-----------------------+ | 07af3300-9e07-4792-9352-6f16f6d949a6 | tcp | 10.196.0.0/16 | 10257:10257 | None | | 17dfb658-6561-47b3-af1c-5cd3ee01c833 | tcp | 172.30.0.0/15 | 6443:6443 | None | | 20b008c2-b7e1-402e-92e8-b6dcc9446806 | icmp | 0.0.0.0/0 | | None | | 20dd5885-3610-4b4e-976c-5ac810b80765 | tcp | 10.196.0.0/16 | 30000:32767 | None | | 25b75364-1c05-45e1-ba25-bfc7918e290d | tcp | 0.0.0.0/0 | 22:22 | None | | 2632a855-f2cf-4856-aed2-bd0919069ae9 | tcp | 0.0.0.0/0 | 6443:6443 | None | | 2eab3e89-5442-4782-8599-43223b33bac2 | tcp | 10.128.0.0/14 | 10250:10250 | None | | 2ed03a55-3a12-46b5-82f1-bec8d29f96c9 | udp | 10.128.0.0/14 | 9000:9999 | None | | 313b7c5c-da79-49d4-9291-d1dd853709be | tcp | 10.196.0.0/16 | 10259:10259 | None | | 34ae6608-e33b-40a1-afe1-df6abdebb4d3 | udp | 10.196.0.0/16 | 5353:5353 | None | | 358682fa-8847-4146-9f22-c051c9d7a1fa | tcp | 10.196.0.0/16 | 9000:9999 | None | | 3bb088aa-7dde-47d4-9f47-2cefc70cbc9d | udp | 10.128.0.0/14 | 53:53 | None | | 3daf295b-41a2-4a6f-b91a-119f565d63e3 | tcp | 10.128.0.0/14 | 10257:10257 | None | | 3fe37653-c632-4929-830a-7f65fc8e05f1 | udp | 10.196.0.0/16 | 30000:32767 | None | | 42b675ec-f0d3-4967-afb2-20026930336d | tcp | 10.128.0.0/14 | | None | | 787ca8e6-85a5-4154-b55f-ae0c87e9fbcf | tcp | 10.196.0.0/16 | 2379:2380 | None | | 830375c2-75a5-4bf8-acf9-a4e94f2ac217 | udp | 10.196.0.0/16 | 53:53 | None | | 9cb446ff-2ce8-4c98-9afe-1714abf6ee2c | tcp | 10.128.0.0/14 | 53:53 | None | | ab93c6cf-bc2c-4e32-9e31-c7e222829371 | tcp | 10.196.0.0/16 | 6641:6642 | None | | ada7a367-eb6b-4189-8cb3-87af45897280 | tcp | 172.30.0.0/15 | 2379:2379 | None | | c127a2da-b9cf-4f13-8b8a-ab4b693948c8 | udp | 10.196.0.0/16 | 9000:9999 | None | | c3c13297-4791-45d9-8d6a-de59e8d978ca | vrrp | 10.196.0.0/16 | | None | | caa03cd7-08e7-4d39-b648-9fa68b566e07 | tcp | 10.128.0.0/14 | 10259:10259 | None | | e296a1c0-a0a4-45bf-ae7c-84ffb39d396f | tcp | 10.196.0.0/16 | 22623:22623 | None | | e7b5a37f-efaa-49fb-ac01-4b65cee6e54d | tcp | 10.128.0.0/14 | 2379:2379 | None | | efee2c5e-3b75-4bce-9dd1-87e048d425aa | udp | 10.196.0.0/16 | 4789:4789 | None | | f02880ff-458d-4ee8-bfd2-276e51831ba8 | tcp | 10.196.0.0/16 | 53:53 | None | | f88e69e0-a415-4aa7-857e-7af3e4f29c1d | tcp | 10.128.0.0/14 | 9000:9999 | None | | f938026c-0366-49fd-ae83-e578190ee5c8 | udp | 10.196.0.0/16 | 6081:6081 | None | | fdb964c7-9a21-4d38-a780-0d1fcfe8b868 | tcp | 10.196.0.0/16 | 10250:10250 | None | +--------------------------------------+-------------+---------------+-------------+-----------------------+ $ openstack security group rule list --ingress 744ed27d-5789-46b1-9c96-0838704190b3 +--------------------------------------+-------------+---------------+-------------+-----------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+---------------+-------------+-----------------------+ | 0bec190f-df98-4f72-94e5-8ea6ef8eb95d | udp | 10.128.0.0/14 | 9000:9999 | None | | 0c36070f-0121-4721-a9ab-accabf9d3658 | udp | 10.196.0.0/16 | 6081:6081 | None | | 264df5f6-9c2d-48e3-88e2-cad5e20b2738 | tcp | 10.128.0.0/14 | 9000:9999 | None | | 2cdc62c4-a77a-4f6e-a391-fe01cf40fb92 | udp | 10.196.0.0/16 | 9000:9999 | None | | 2d6f6918-b031-47bd-83b6-7cd4fe380d43 | tcp | 10.196.0.0/16 | 10250:10250 | None | | 2d766718-76fe-4807-bd93-1be613295de8 | tcp | 0.0.0.0/0 | 80:80 | None | | 353369ae-b999-4917-bff6-c907e386e285 | udp | 10.196.0.0/16 | 4789:4789 | None | | 3c7db47b-1de1-4f01-8eea-791f61869379 | tcp | 10.128.0.0/14 | 10250:10250 | None | | 61d227e8-905d-44c0-b83b-be315b2c92f2 | tcp | 10.196.0.0/16 | 9000:9999 | None | | 6726b0f0-fa5a-40df-b26e-fa89cc3f2b9e | tcp | 0.0.0.0/0 | 22:22 | None | | 69c87281-3f1d-4589-a705-02c51d46a80a | tcp | 0.0.0.0/0 | 443:443 | None | | 82ace9e8-3874-48c3-a715-a4ac26673c3d | udp | 10.196.0.0/16 | 30000:32767 | None | | 961b1410-7512-4dbe-8549-c929e23fb44f | udp | 10.196.0.0/16 | 5353:5353 | None | | a52e4374-e50b-456c-855e-d0a86682a357 | tcp | 10.196.0.0/16 | 30000:32767 | None | | bf2b81c4-0117-4b08-b41d-5f30bb8e950d | tcp | 10.196.0.0/16 | 1936:1936 | None | | cbd3af0c-dc16-4aef-8be8-fc4516aa5769 | tcp | 10.128.0.0/14 | 53:53 | None | | d36cff55-fb70-4aed-b086-63c17055e9ac | icmp | 0.0.0.0/0 | | None | | d708963a-a869-49cd-8db0-2a6529682ae1 | vrrp | 10.196.0.0/16 | | None | | de0fe90d-51c3-469e-83f3-9922408c6b67 | udp | 10.128.0.0/14 | 53:53 | None | | e939d765-e63e-42e1-871f-2c5404268fc5 | tcp | 10.128.0.0/14 | 1936:1936 | None | +--------------------------------------+-------------+---------------+-------------+-----------------------+
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409