Bug 1731748 - RHCS installation guide - MON should have port 3300 opened
Summary: RHCS installation guide - MON should have port 3300 opened
Keywords:
Status: NEW
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: Documentation
Version: 4.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: 4.1
Assignee: ceph-docs@redhat.com
QA Contact: Tejas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-21 17:23 UTC by Vagner Farias
Modified: 2020-02-18 02:24 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Vagner Farias 2019-07-21 17:23:48 UTC
Description of problem:
MON nodes also require port 3300/tcp to be opened, but only port 6789 is documented. Without port 3300 being opened, deployment never completes.

A section such as the following should be added.

On all monitor nodes, open port 3300 on the public network:

[root@monitor ~]# firewall-cmd --zone=public --add-port=3300/tcp
[root@monitor ~]# firewall-cmd --zone=public --add-port=3300/tcp--permanent

To limit access based on the source address:

firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
port="3300" accept"

firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
port="3300" accept" --permanent

Plus the example for above rules.

Comment 1 Giridhar Ramaraju 2019-08-05 13:07:09 UTC
Updating the QA Contact to a Hemant. Hemant will be rerouting them to the appropriate QE Associate. 

Regards,
Giri

Comment 2 Giridhar Ramaraju 2019-08-05 13:09:42 UTC
Updating the QA Contact to a Hemant. Hemant will be rerouting them to the appropriate QE Associate. 

Regards,
Giri

Comment 3 Vasishta 2019-11-15 10:59:10 UTC
Hi Vagner,

Sorry for my ignorance, can I know why mon needs 3300 port ?
Downstream QE has not faced any installation issues for not opening 3300 port.

This can be an RFE for ceph-ansible and also it would help us to set severity of this BZ.

Regards,
Vasishta
QE, Ceph

Comment 4 Vagner Farias 2019-11-15 14:52:44 UTC
Hi Vasishta,

IIRC my deployment failed because port 3300 wasn't opened. According to upstream documentation (https://docs.ceph.com/docs/master/rados/configuration/common/#monitors), Ceph MON currently uses port 3300.

I have not updated my environment so far, so it's still running RHCS 4 beta1. This environment has 3 nodes, with the following services collocated on same node: MON, MGR and OSD.

IP addresses of the nodes (disregard the hostname, as I wasn't collocating services initially):

- ceph-osd01: 192.168.50.11
- ceph-osd02: 192.168.50.12
- ceph-osd03: 192.168.50.13

From the output below you can see that nodes 192.168.50.12 and 192.168.50.13 are connecting to port 3300 of node 192.168.50.11.

[root@ceph-osd01 ~]# podman ps
CONTAINER ID  IMAGE                                                COMMAND               CREATED         STATUS             PORTS  NAMES
d6191bd12433  ceph-ansible.example.com:5000/rhceph-4-rhel8:latest  /opt/ceph-contain...  19 minutes ago  Up 19 minutes ago         ceph-osd-0
78f44feb231f  ceph-ansible.example.com:5000/rhceph-4-rhel8:latest  /opt/ceph-contain...  19 minutes ago  Up 19 minutes ago         ceph-mgr-ceph-osd01
453ec132e7ba  ceph-ansible.example.com:5000/rhceph-4-rhel8:latest  /opt/ceph-contain...  19 minutes ago  Up 19 minutes ago         ceph-mon-ceph-osd01
[root@ceph-osd01 ~]# ss -latupn | grep 3300
tcp   LISTEN  0        128         192.168.50.11:3300            0.0.0.0:*       users:(("ceph-mon",pid=7054,fd=26))                                            
tcp   ESTAB   0        0           192.168.50.11:45356     192.168.50.11:3300    users:(("ceph-osd",pid=7144,fd=53))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.11:45356   users:(("ceph-mon",pid=7054,fd=36))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.13:45122   users:(("ceph-mon",pid=7054,fd=38))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.13:45086   users:(("ceph-mon",pid=7054,fd=33))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.11:45346   users:(("ceph-mon",pid=7054,fd=32))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.12:56384   users:(("ceph-mon",pid=7054,fd=40))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.13:45144   users:(("ceph-mon",pid=7054,fd=39))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.12:56310   users:(("ceph-mon",pid=7054,fd=34))                                            
tcp   ESTAB   0        0           192.168.50.11:45346     192.168.50.11:3300    users:(("ceph-mgr",pid=7040,fd=19))                                            
tcp   ESTAB   0        0           192.168.50.11:3300      192.168.50.12:56346   users:(("ceph-mon",pid=7054,fd=37))

Comment 5 Vagner Farias 2019-11-15 15:43:37 UTC
Looking further into this, I think the documentation provided with Beta1[1] was wrong - I haven't reviewed documentation for more recent beta, though.

Documentation states (page 13, section 2.9):

~~~
The Monitor daemons use port 6789 for communication within the Ceph storage cluster.
~~~

And later on the same page:

~~~
2. On all monitor nodes, open port 6789 on the public network:
~~~

Instead, it should tell user to enable the "ceph-mon" service, as it automatically opens both 3300 and 6789 ports. 

~~~
[root@ceph-ansible ~]# firewall-cmd --permanent --service=ceph-mon --get-ports
3300/tcp 6789/tcp
~~~

Same recommendation applies to MGR and OSD nodes. Instead of telling to open specific ports, we should tell to enable "ceph" service, as it already opens ports 6800-7300.


[1] ftp://partners.redhat.com/d8556772a349f93d26ffa995bbc9008e/Red_Hat_Ceph_Storage-4-DRAFT_-_Installation_Guide_-_DRAFT-en-US062519.pdf

Comment 6 Vasishta 2019-11-18 02:40:21 UTC
Hi Vagner,

Thanks a lot for the detailed inputs, it was helpful.

I think ceph-ansible has this implemented already -
https://github.com/ceph/ceph-ansible/blob/stable-4.0/roles/ceph-infra/tasks/configure_firewall.yml

Regards,
Vasishta Shastry
QE, Ceph

Comment 7 Vagner Farias 2019-11-24 03:32:15 UTC
Hi Vasishta,

IIUC the firewall configuration will be applied by the playbook only if configure_firewall variable is set to True. It's also my understanding that the documentation I mentioned in comment #5 applies to users that would like to manually configure the firewall, instead of letting ceph-ansible doing so (when configure_firewall is set to False). Aforementioned document needs to be fixed or the deployment will fail.


Note You need to log in before you can comment on or make changes to this bug.