Bug 1613489

Summary:	OpenShift Ansible Installer enables and is missing ports in different security groups
Product:	OpenShift Container Platform	Reporter:	rlopez
Component:	Installer	Assignee:	aos-install
Installer sub component:	openshift-ansible	QA Contact:	Gaoyun Pei <gpei>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	low
Priority:	medium	CC:	aos-bugs, gpei, jokerman, mmccomas, rlopez, tzumainn, wsun
Version:	3.11.0
Target Milestone:	---
Target Release:	3.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-09-16 07:46:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description rlopez 2018-08-07 16:19:10 UTC

Description of problem:

Looking closely at the security groups created:

* there are ports that are enabled that do not show up as part of the OpenShift documentation: https://docs.openshift.com/container-platform/3.10/install/prerequisites.html

* there are ports that should be enabled/disabled depending on if the variables are set in the OSEv3.yml file

openshift-ansible-openshift.example.com-node-secgrp
+--------------------------------------+-------------+-----------------+-------------+--------------------------------------+
| ID                                   | IP Protocol | IP Range        | Port Range  | Remote Security Group                |
+--------------------------------------+-------------+-----------------+-------------+--------------------------------------+
| 323467c2-9c86-46aa-9a47-1e60fb2af046 | tcp         | None            | 10255:10255 | 1be9055c-e98f-4d02-a561-080814f43d9e |
| 3711d299-17f6-4ee9-82d1-298a67b7f48b | tcp         | 0.0.0.0/0       | 30000:32767 | None                                 |
| 4c3af587-40ed-457d-85b5-a5c87d9ed68f | udp         | None            | 4789:4789   | 1be9055c-e98f-4d02-a561-080814f43d9e |
| 50d68827-8c0e-4cc6-a593-bae9f48ac0f5 | tcp         | None            | 53:53       | None                                 |
| 726665a4-67fe-4b27-8421-c61983fb3bad | None        | None            |             | None                                 |
| 7abef9ff-e0e0-4ede-864e-f25c8824450b | tcp         | None            | 10250:10250 | 1be9055c-e98f-4d02-a561-080814f43d9e |
| 96f2a4dc-54fb-4c8a-9657-47bdc6435ca9 | udp         | None            | 10250:10250 | 1be9055c-e98f-4d02-a561-080814f43d9e |
| 96fe369b-29a2-4988-a9cb-76a64480c0b5 | udp         | None            | 10255:10255 | 1be9055c-e98f-4d02-a561-080814f43d9e |
| 9d6ed588-f85e-4293-b91e-db3b50216506 | tcp         | 192.168.99.0/24 | 30000:32767 | None                                 |
| a18f4674-2aad-4a86-9c02-6aeac9f3301d | udp         | None            | 53:53       | None                                 |
| ee958658-720c-4390-9517-aace41c0f2a2 | None        | None            |             | None                                 |
+--------------------------------------+-------------+-----------------+-------------+--------------------------------------+


Recommendations:

- Port 10255 udp/tcp is for metrics. It should only be enabled when openshift_metrics_install_metrics is set to true in OSEv3.yml

- Ports 30000-32767 udp/tcp - is for Nodeports. Nodeports are used to expose applications without using router. This should be disabled by default.

- Port 53 udp/tcp - Not required for node security group. This is used for DNS node which is not created using this group. 

- Port 10250/udp - The UDP of this port is not required only TCP. The TCP version is used for kubelet communication.

- Port 9100 tcp (currently missing) - enable when openshift_hosted_prometheus_deploy=true within OSEv3.yml

Questions:

- Port 10255 - from what I found this is because kubelet exposes endpoints on this port. However, with manual deployments I have never enabled this port. Can someone confirm this is required?

openshift-ansible-openshift.example.com-master-secgrp
+--------------------------------------+-------------+----------+-------------+-----------------------+
| ID                                   | IP Protocol | IP Range | Port Range  | Remote Security Group |
+--------------------------------------+-------------+----------+-------------+-----------------------+
| 0626d242-b408-4374-a1c9-9c111e71f709 | udp         | None     | 8053:8053   | None                  |
| 066c3795-8ae2-44cd-b737-8acba02cfd59 | tcp         | None     | 24224:24224 | None                  |
| 25479e04-0c54-402c-b193-95b5a8af94e7 | tcp         | None     | 8443:8443   | None                  |
| 5f6b1087-1589-491e-828d-11529b12bc2b | tcp         | None     | 2224:2224   | None                  |
| 9eee7ae2-7a6b-4369-8b7f-6c52187eb364 | udp         | None     | 24224:24224 | None                  |
| b18dc55a-654b-4dae-9ec4-c38aa8e3226b | tcp         | None     | 9090:9090   | None                  |
| bd0cd8da-5a2f-4dc4-bde1-dcb65b939331 | udp         | None     | 5404:5405   | None                  |
| c05909f3-a376-4ec1-9051-22bafa9bb994 | tcp         | None     | 4001:4001   | None                  |
| ecc0bf02-7ca4-4b00-9760-a297a0cd1aee | None        | None     |             | None                  |
| f53aeb99-ace4-4813-9667-ec7791816780 | None        | None     |             | None                  |
| fa1355f3-e387-4b98-bf03-76dcac36c975 | tcp         | None     | 8053:8053   | None                  |
+--------------------------------------+-------------+----------+-------------+-----------------------+

Recommendations:

- Port 4001 - Used for embedded etcd (non-clustered) to accept changes in state (according to old documentation, unsure if still used.) However, if it is still used it should be enabled only when "openshift_openstack_num_masters" is set to 1 in all.yml

Following ports below not here: https://docs.openshift.com/container-platform/3.10/install/prerequisites.html

- Port 2224 - unsure what this is used for
- Port 9090 - unsure what this is used for
- Port 5404-5405 - unsure what this is used for



openshift-ansible-openshift.example.com-infra-secgrp
+--------------------------------------+-------------+----------+------------+-----------------------+
| ID                                   | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+----------+------------+-----------------------+
| 0eaabf8f-312d-4ccd-aced-78498d2036f6 | tcp         | None     | 443:443    | None                  |
| 1c3e9e8e-f753-4e35-8905-25e3feaee81a | None        | None     |            | None                  |
| 41e4f07d-bdee-404f-a295-0d4619c42388 | tcp         | None     | 1936:1936  | None                  |
| 4e0481bb-db90-466d-a47b-e6b5c3b29580 | tcp         | None     | 80:80      | None                  |
| 7d07c773-ff9c-47aa-a052-9e7589bcbff2 | None        | None     |            | None                  |
+--------------------------------------+-------------+----------+------------+-----------------------+

Ports: https://docs.openshift.com/container-platform/3.10/install/prerequisites.html

Recommendations: 
- Port 9300 tcp (missing) - enable when openshift_logging_install_logging=true in OSEv3.yml
- Port 9200 tcp (missing) - enable when openshift_logging_install_logging=true in OSEv3.yml

Question:
- Port 1936 - (Optional) Required to be open when running the template router to access statistics. 
Can be open externally or internally to connections depending on if you want the statistics to be 
expressed publicly. Since it is optional do we still want it enabled for the installer? Via the prerequisites.html, you still need to enable firewall rules to make sure it works:

Port 1936 can still be inaccessible due to your iptables rules. Use the following to configure iptables to open port 1936:

# iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp \
    --dport 1936 -j ACCEPT

Comment 1 Scott Dodson 2018-08-07 18:35:15 UTC

Roger,

This pertains to installation via the Openstack playbooks?

Comment 2 rlopez 2018-08-07 18:39:44 UTC

Scott,

Yes.

i.e.

ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml

Comment 3 egarcia 2018-12-04 14:21:14 UTC

https://github.com/openshift/openshift-ansible/pull/10173

Comment 5 weiwei jiang 2019-02-19 02:33:43 UTC

(In reply to egarcia from comment #3)
> https://github.com/openshift/openshift-ansible/pull/10173

Which version should I give a try, since the target is 4.0 and no "Fixed in version", means I have to try 4.0 latest version?

Comment 6 egarcia 2019-02-19 15:13:59 UTC

Ah, sorry about that. I haven't looked at this code base in a while :). It should definitely be in the 3.11 release.

Comment 7 Wei Sun 2019-02-20 03:06:20 UTC

Change target release version to 3.11.z per #comment 6

Comment 8 weiwei jiang 2019-02-25 09:39:57 UTC

Checked with openshift-ansible-3.11.88-1.git.0.42d1b9a.el7.noarch
And seems like this patch https://github.com/openshift/openshift-ansible/pull/10173 is not backported into 3.11.z



(openstack) security group rule list openshift-ansible-wjiang-ocp.shiftstack.com-node-secgrp
+--------------------------------------+-------------+-----------------+-------------+--------------------------------------+
| ID                                   | IP Protocol | IP Range        | Port Range  | Remote Security Group                |
+--------------------------------------+-------------+-----------------+-------------+--------------------------------------+
| 0724d2db-bcc5-4dce-9f11-0f3cd3176be0 | tcp         | None            | 10255:10255 | 70be2138-cd63-4ac6-8591-854c514ff0a4 |
| 0e0dc215-fab2-4ea3-b06d-5e2586030a8e | tcp         | None            | 53:53       | None                                 |
| 15f959fa-deea-4053-a78f-c2743e90622c | udp         | None            | 10250:10250 | 70be2138-cd63-4ac6-8591-854c514ff0a4 |
| 1bf4a62e-fe89-4e70-b932-0bc5cdf86dbe | udp         | None            | 10255:10255 | 70be2138-cd63-4ac6-8591-854c514ff0a4 |
| 3e3512e2-22e8-44f2-9959-97850840be12 | tcp         | 0.0.0.0/0       | 30000:32767 | None                                 |
| 5a422404-33fd-427f-99cd-dd914d9e8570 | udp         | None            | 4789:4789   | 70be2138-cd63-4ac6-8591-854c514ff0a4 |
| 8048fae9-a5ba-44f4-a5ff-a3379385e450 | tcp         | None            | 10250:10250 | 70be2138-cd63-4ac6-8591-854c514ff0a4 |
| ae495707-64c8-4208-870b-9df77cce6b5c | None        | None            |             | None                                 |
| be8b635f-d2ed-4e7c-a40a-b0df96f44810 | tcp         | 192.168.99.0/24 | 30000:32767 | None                                 |
| f42c72e2-d51f-4e22-a8e2-a334089d1c62 | None        | None            |             | None                                 |
| f4da60e2-ef81-4a17-a4c1-23671aba1e59 | udp         | None            | 53:53       | None                                 |
+--------------------------------------+-------------+-----------------+-------------+--------------------------------------+

So move back to modified.

Comment 9 egarcia 2019-04-08 20:39:03 UTC

Apologies, It has now been backported: https://github.com/openshift/openshift-ansible/pull/11476

Comment 11 weiwei jiang 2020-09-10 11:25:07 UTC

Checked with openshift-ansible-3.11.285, and it's fixed.

# openstack security group rule list openshift-ansible-wjiang-ocp.shiftstack.com-common-secgrp 
+--------------------------------------+-------------+-----------+------------+-----------------------+
| ID                                   | IP Protocol | IP Range  | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+------------+-----------------------+
| 021038d3-120e-403b-83a1-bd508969ec3a | icmp        | 0.0.0.0/0 |            | None                  |
| 67519f6f-3bf6-4341-a524-9fc3ad9fcd40 | None        | None      |            | None                  |
| 6cc8d134-43f9-4ae6-a8fe-758f62330ee4 | None        | None      |            | None                  |
| f97dee03-1734-4fc5-95e8-d86e5fa819e4 | tcp         | 0.0.0.0/0 | 22:22      | None                  |
+--------------------------------------+-------------+-----------+------------+-----------------------+
# openstack security group rule list openshift-ansible-wjiang-ocp.shiftstack.com-master-secgrp                                                                                                                    
+--------------------------------------+-------------+----------+-------------+-----------------------+
| ID                                   | IP Protocol | IP Range | Port Range  | Remote Security Group |
+--------------------------------------+-------------+----------+-------------+-----------------------+
| 1176d138-a30f-4b52-82e7-da50f6c499a4 | udp         | None     | 24224:24224 | None                  |
| 324a9f37-35c2-4af8-9f73-13ffe48bf72c | tcp         | None     | 9090:9090   | None                  |
| 64830a89-c3fc-486e-83ac-51ac918b3158 | udp         | None     | 8053:8053   | None                  |
| a3107147-4a9d-4b06-839a-1b13bce50cd8 | tcp         | None     | 4001:4001   | None                  |
| abb61ed0-2f64-4ca4-a2ce-cdba13499042 | tcp         | None     | 8053:8053   | None                  |
| b0dbcaf2-1bb7-4e39-8b4e-d52978a29505 | None        | None     |             | None                  |
| c0f1f656-b59a-4f5b-a9e1-59c66215afed | None        | None     |             | None                  |
| d301d8f0-243e-4009-b49c-5ea9bfb300ee | tcp         | None     | 24224:24224 | None                  |
| d3c6340e-751f-4740-9f4b-76f86186f287 | tcp         | None     | 8443:8443   | None                  |
+--------------------------------------+-------------+----------+-------------+-----------------------+
# openstack security group rule list openshift-ansible-wjiang-ocp.shiftstack.com-node-secgrp                                                                                                                      
+--------------------------------------+-------------+----------+-------------+--------------------------------------+
| ID                                   | IP Protocol | IP Range | Port Range  | Remote Security Group                |
+--------------------------------------+-------------+----------+-------------+--------------------------------------+
| 2d7a968e-8c68-46dc-91ee-0fdc3ac5388a | udp         | None     | 53:53       | None                                 |
| 2fdbf1df-9c72-4f95-a98e-e47f8a3513b7 | None        | None     |             | None                                 |
| 4ff05bbd-4d5d-49f7-b1c6-1d7124c0a6ff | tcp         | None     | 10250:10250 | None                                 |
| 692e9670-916f-4d19-8461-75b93e49498e | None        | None     |             | None                                 |
| b936dadf-4c4f-42ad-9382-96aca4d80b81 | tcp         | None     | 53:53       | None                                 |
| dd160b40-612d-4eac-b59c-f072ab570793 | udp         | None     | 4789:4789   | 21f03295-cb8b-4da7-9319-5993ccb63814 |
+--------------------------------------+-------------+----------+-------------+--------------------------------------+

Comment 15 errata-xmlrpc 2020-09-16 07:46:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 3.11.286 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3695