Bug 1731748

Updating the QA Contact to a Hemant. Hemant will be rerouting them to the appropriate QE Associate. 

Regards,
Giri

Updating the QA Contact to a Hemant. Hemant will be rerouting them to the appropriate QE Associate. 

Regards,
Giri

Hi Vagner,

Sorry for my ignorance, can I know why mon needs 3300 port ?
Downstream QE has not faced any installation issues for not opening 3300 port.

This can be an RFE for ceph-ansible and also it would help us to set severity of this BZ.

Regards,
Vasishta
QE, Ceph

Hi Vasishta,

IIRC my deployment failed because port 3300 wasn't opened. According to upstream documentation (https://docs.ceph.com/docs/master/rados/configuration/common/#monitors), Ceph MON currently uses port 3300.

I have not updated my environment so far, so it's still running RHCS 4 beta1. This environment has 3 nodes, with the following services collocated on same node: MON, MGR and OSD.

IP addresses of the nodes (disregard the hostname, as I wasn't collocating services initially):

- ceph-osd01: 192.168.50.11
- ceph-osd02: 192.168.50.12
- ceph-osd03: 192.168.50.13

From the output below you can see that nodes 192.168.50.12 and 192.168.50.13 are connecting to port 3300 of node 192.168.50.11.

[root@ceph-osd01 ~]# podman ps
CONTAINER ID  IMAGE                                                COMMAND               CREATED         STATUS             PORTS  NAMES
d6191bd12433  ceph-ansible.example.com:5000/rhceph-4-rhel8:latest  /opt/ceph-contain...  19 minutes ago  Up 19 minutes ago         ceph-osd-0
78f44feb231f  ceph-ansible.example.com:5000/rhceph-4-rhel8:latest  /opt/ceph-contain...  19 minutes ago  Up 19 minutes ago         ceph-mgr-ceph-osd01
453ec132e7ba  ceph-ansible.example.com:5000/rhceph-4-rhel8:latest  /opt/ceph-contain...  19 minutes ago  Up 19 minutes ago         ceph-mon-ceph-osd01
[root@ceph-osd01 ~]# ss -latupn | grep 3300
tcp   LISTEN  0        128         192.168.50.11:3300            0.0.0.0:*       users:(("ceph-mon",pid=7054,fd=26))                                            
tcp   ESTAB   0        0           192.168.50.11:45356     192.168.50.11:3300    users:(("ceph-osd",pid=7144,fd=53))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.11:45356   users:(("ceph-mon",pid=7054,fd=36))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.13:45122   users:(("ceph-mon",pid=7054,fd=38))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.13:45086   users:(("ceph-mon",pid=7054,fd=33))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.11:45346   users:(("ceph-mon",pid=7054,fd=32))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.12:56384   users:(("ceph-mon",pid=7054,fd=40))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.13:45144   users:(("ceph-mon",pid=7054,fd=39))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.12:56310   users:(("ceph-mon",pid=7054,fd=34))                                            
tcp   ESTAB   0        0           192.168.50.11:45346     192.168.50.11:3300    users:(("ceph-mgr",pid=7040,fd=19))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.12:56346   users:(("ceph-mon",pid=7054,fd=37))

Looking further into this, I think the documentation provided with Beta1[1] was wrong - I haven't reviewed documentation for more recent beta, though.

Documentation states (page 13, section 2.9):

~~~
The Monitor daemons use port 6789 for communication within the Ceph storage cluster.
~~~

And later on the same page:

~~~
2. On all monitor nodes, open port 6789 on the public network:
~~~

Instead, it should tell user to enable the "ceph-mon" service, as it automatically opens both 3300 and 6789 ports. 

~~~
[root@ceph-ansible ~]# firewall-cmd --permanent --service=ceph-mon --get-ports
3300/tcp 6789/tcp
~~~

Same recommendation applies to MGR and OSD nodes. Instead of telling to open specific ports, we should tell to enable "ceph" service, as it already opens ports 6800-7300.


[1] ftp://partners.redhat.com/d8556772a349f93d26ffa995bbc9008e/Red_Hat_Ceph_Storage-4-DRAFT_-_Installation_Guide_-_DRAFT-en-US062519.pdf

Hi Vagner,

Thanks a lot for the detailed inputs, it was helpful.

I think ceph-ansible has this implemented already -
https://github.com/ceph/ceph-ansible/blob/stable-4.0/roles/ceph-infra/tasks/configure_firewall.yml

Regards,
Vasishta Shastry
QE, Ceph

Hi Vasishta,

IIUC the firewall configuration will be applied by the playbook only if configure_firewall variable is set to True. It's also my understanding that the documentation I mentioned in comment #5 applies to users that would like to manually configure the firewall, instead of letting ceph-ansible doing so (when configure_firewall is set to False). Aforementioned document needs to be fixed or the deployment will fail.