Description of problem: Looking closely at the security groups created: * there are ports that are enabled that do not show up as part of the OpenShift documentation: https://docs.openshift.com/container-platform/3.10/install/prerequisites.html * there are ports that should be enabled/disabled depending on if the variables are set in the OSEv3.yml file openshift-ansible-openshift.example.com-node-secgrp +--------------------------------------+-------------+-----------------+-------------+--------------------------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-----------------+-------------+--------------------------------------+ | 323467c2-9c86-46aa-9a47-1e60fb2af046 | tcp | None | 10255:10255 | 1be9055c-e98f-4d02-a561-080814f43d9e | | 3711d299-17f6-4ee9-82d1-298a67b7f48b | tcp | 0.0.0.0/0 | 30000:32767 | None | | 4c3af587-40ed-457d-85b5-a5c87d9ed68f | udp | None | 4789:4789 | 1be9055c-e98f-4d02-a561-080814f43d9e | | 50d68827-8c0e-4cc6-a593-bae9f48ac0f5 | tcp | None | 53:53 | None | | 726665a4-67fe-4b27-8421-c61983fb3bad | None | None | | None | | 7abef9ff-e0e0-4ede-864e-f25c8824450b | tcp | None | 10250:10250 | 1be9055c-e98f-4d02-a561-080814f43d9e | | 96f2a4dc-54fb-4c8a-9657-47bdc6435ca9 | udp | None | 10250:10250 | 1be9055c-e98f-4d02-a561-080814f43d9e | | 96fe369b-29a2-4988-a9cb-76a64480c0b5 | udp | None | 10255:10255 | 1be9055c-e98f-4d02-a561-080814f43d9e | | 9d6ed588-f85e-4293-b91e-db3b50216506 | tcp | 192.168.99.0/24 | 30000:32767 | None | | a18f4674-2aad-4a86-9c02-6aeac9f3301d | udp | None | 53:53 | None | | ee958658-720c-4390-9517-aace41c0f2a2 | None | None | | None | +--------------------------------------+-------------+-----------------+-------------+--------------------------------------+ Recommendations: - Port 10255 udp/tcp is for metrics. It should only be enabled when openshift_metrics_install_metrics is set to true in OSEv3.yml - Ports 30000-32767 udp/tcp - is for Nodeports. Nodeports are used to expose applications without using router. This should be disabled by default. - Port 53 udp/tcp - Not required for node security group. This is used for DNS node which is not created using this group. - Port 10250/udp - The UDP of this port is not required only TCP. The TCP version is used for kubelet communication. - Port 9100 tcp (currently missing) - enable when openshift_hosted_prometheus_deploy=true within OSEv3.yml Questions: - Port 10255 - from what I found this is because kubelet exposes endpoints on this port. However, with manual deployments I have never enabled this port. Can someone confirm this is required? openshift-ansible-openshift.example.com-master-secgrp +--------------------------------------+-------------+----------+-------------+-----------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+----------+-------------+-----------------------+ | 0626d242-b408-4374-a1c9-9c111e71f709 | udp | None | 8053:8053 | None | | 066c3795-8ae2-44cd-b737-8acba02cfd59 | tcp | None | 24224:24224 | None | | 25479e04-0c54-402c-b193-95b5a8af94e7 | tcp | None | 8443:8443 | None | | 5f6b1087-1589-491e-828d-11529b12bc2b | tcp | None | 2224:2224 | None | | 9eee7ae2-7a6b-4369-8b7f-6c52187eb364 | udp | None | 24224:24224 | None | | b18dc55a-654b-4dae-9ec4-c38aa8e3226b | tcp | None | 9090:9090 | None | | bd0cd8da-5a2f-4dc4-bde1-dcb65b939331 | udp | None | 5404:5405 | None | | c05909f3-a376-4ec1-9051-22bafa9bb994 | tcp | None | 4001:4001 | None | | ecc0bf02-7ca4-4b00-9760-a297a0cd1aee | None | None | | None | | f53aeb99-ace4-4813-9667-ec7791816780 | None | None | | None | | fa1355f3-e387-4b98-bf03-76dcac36c975 | tcp | None | 8053:8053 | None | +--------------------------------------+-------------+----------+-------------+-----------------------+ Recommendations: - Port 4001 - Used for embedded etcd (non-clustered) to accept changes in state (according to old documentation, unsure if still used.) However, if it is still used it should be enabled only when "openshift_openstack_num_masters" is set to 1 in all.yml Following ports below not here: https://docs.openshift.com/container-platform/3.10/install/prerequisites.html - Port 2224 - unsure what this is used for - Port 9090 - unsure what this is used for - Port 5404-5405 - unsure what this is used for openshift-ansible-openshift.example.com-infra-secgrp +--------------------------------------+-------------+----------+------------+-----------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+----------+------------+-----------------------+ | 0eaabf8f-312d-4ccd-aced-78498d2036f6 | tcp | None | 443:443 | None | | 1c3e9e8e-f753-4e35-8905-25e3feaee81a | None | None | | None | | 41e4f07d-bdee-404f-a295-0d4619c42388 | tcp | None | 1936:1936 | None | | 4e0481bb-db90-466d-a47b-e6b5c3b29580 | tcp | None | 80:80 | None | | 7d07c773-ff9c-47aa-a052-9e7589bcbff2 | None | None | | None | +--------------------------------------+-------------+----------+------------+-----------------------+ Ports: https://docs.openshift.com/container-platform/3.10/install/prerequisites.html Recommendations: - Port 9300 tcp (missing) - enable when openshift_logging_install_logging=true in OSEv3.yml - Port 9200 tcp (missing) - enable when openshift_logging_install_logging=true in OSEv3.yml Question: - Port 1936 - (Optional) Required to be open when running the template router to access statistics. Can be open externally or internally to connections depending on if you want the statistics to be expressed publicly. Since it is optional do we still want it enabled for the installer? Via the prerequisites.html, you still need to enable firewall rules to make sure it works: Port 1936 can still be inaccessible due to your iptables rules. Use the following to configure iptables to open port 1936: # iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp \ --dport 1936 -j ACCEPT
Roger, This pertains to installation via the Openstack playbooks?
Scott, Yes. i.e. ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml
https://github.com/openshift/openshift-ansible/pull/10173
(In reply to egarcia from comment #3) > https://github.com/openshift/openshift-ansible/pull/10173 Which version should I give a try, since the target is 4.0 and no "Fixed in version", means I have to try 4.0 latest version?
Ah, sorry about that. I haven't looked at this code base in a while :). It should definitely be in the 3.11 release.
Change target release version to 3.11.z per #comment 6
Checked with openshift-ansible-3.11.88-1.git.0.42d1b9a.el7.noarch And seems like this patch https://github.com/openshift/openshift-ansible/pull/10173 is not backported into 3.11.z (openstack) security group rule list openshift-ansible-wjiang-ocp.shiftstack.com-node-secgrp +--------------------------------------+-------------+-----------------+-------------+--------------------------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-----------------+-------------+--------------------------------------+ | 0724d2db-bcc5-4dce-9f11-0f3cd3176be0 | tcp | None | 10255:10255 | 70be2138-cd63-4ac6-8591-854c514ff0a4 | | 0e0dc215-fab2-4ea3-b06d-5e2586030a8e | tcp | None | 53:53 | None | | 15f959fa-deea-4053-a78f-c2743e90622c | udp | None | 10250:10250 | 70be2138-cd63-4ac6-8591-854c514ff0a4 | | 1bf4a62e-fe89-4e70-b932-0bc5cdf86dbe | udp | None | 10255:10255 | 70be2138-cd63-4ac6-8591-854c514ff0a4 | | 3e3512e2-22e8-44f2-9959-97850840be12 | tcp | 0.0.0.0/0 | 30000:32767 | None | | 5a422404-33fd-427f-99cd-dd914d9e8570 | udp | None | 4789:4789 | 70be2138-cd63-4ac6-8591-854c514ff0a4 | | 8048fae9-a5ba-44f4-a5ff-a3379385e450 | tcp | None | 10250:10250 | 70be2138-cd63-4ac6-8591-854c514ff0a4 | | ae495707-64c8-4208-870b-9df77cce6b5c | None | None | | None | | be8b635f-d2ed-4e7c-a40a-b0df96f44810 | tcp | 192.168.99.0/24 | 30000:32767 | None | | f42c72e2-d51f-4e22-a8e2-a334089d1c62 | None | None | | None | | f4da60e2-ef81-4a17-a4c1-23671aba1e59 | udp | None | 53:53 | None | +--------------------------------------+-------------+-----------------+-------------+--------------------------------------+ So move back to modified.
Apologies, It has now been backported: https://github.com/openshift/openshift-ansible/pull/11476
Checked with openshift-ansible-3.11.285, and it's fixed. # openstack security group rule list openshift-ansible-wjiang-ocp.shiftstack.com-common-secgrp +--------------------------------------+-------------+-----------+------------+-----------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-----------+------------+-----------------------+ | 021038d3-120e-403b-83a1-bd508969ec3a | icmp | 0.0.0.0/0 | | None | | 67519f6f-3bf6-4341-a524-9fc3ad9fcd40 | None | None | | None | | 6cc8d134-43f9-4ae6-a8fe-758f62330ee4 | None | None | | None | | f97dee03-1734-4fc5-95e8-d86e5fa819e4 | tcp | 0.0.0.0/0 | 22:22 | None | +--------------------------------------+-------------+-----------+------------+-----------------------+ # openstack security group rule list openshift-ansible-wjiang-ocp.shiftstack.com-master-secgrp +--------------------------------------+-------------+----------+-------------+-----------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+----------+-------------+-----------------------+ | 1176d138-a30f-4b52-82e7-da50f6c499a4 | udp | None | 24224:24224 | None | | 324a9f37-35c2-4af8-9f73-13ffe48bf72c | tcp | None | 9090:9090 | None | | 64830a89-c3fc-486e-83ac-51ac918b3158 | udp | None | 8053:8053 | None | | a3107147-4a9d-4b06-839a-1b13bce50cd8 | tcp | None | 4001:4001 | None | | abb61ed0-2f64-4ca4-a2ce-cdba13499042 | tcp | None | 8053:8053 | None | | b0dbcaf2-1bb7-4e39-8b4e-d52978a29505 | None | None | | None | | c0f1f656-b59a-4f5b-a9e1-59c66215afed | None | None | | None | | d301d8f0-243e-4009-b49c-5ea9bfb300ee | tcp | None | 24224:24224 | None | | d3c6340e-751f-4740-9f4b-76f86186f287 | tcp | None | 8443:8443 | None | +--------------------------------------+-------------+----------+-------------+-----------------------+ # openstack security group rule list openshift-ansible-wjiang-ocp.shiftstack.com-node-secgrp +--------------------------------------+-------------+----------+-------------+--------------------------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+----------+-------------+--------------------------------------+ | 2d7a968e-8c68-46dc-91ee-0fdc3ac5388a | udp | None | 53:53 | None | | 2fdbf1df-9c72-4f95-a98e-e47f8a3513b7 | None | None | | None | | 4ff05bbd-4d5d-49f7-b1c6-1d7124c0a6ff | tcp | None | 10250:10250 | None | | 692e9670-916f-4d19-8461-75b93e49498e | None | None | | None | | b936dadf-4c4f-42ad-9382-96aca4d80b81 | tcp | None | 53:53 | None | | dd160b40-612d-4eac-b59c-f072ab570793 | udp | None | 4789:4789 | 21f03295-cb8b-4da7-9319-5993ccb63814 | +--------------------------------------+-------------+----------+-------------+--------------------------------------+
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.286 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3695