Bug 1551637
Summary: | [RGW Container] options added to the ceph.conf are not being accepted to the running config of the RGW process upon restart | ||
---|---|---|---|
Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | jquinn <jquinn> |
Component: | Container | Assignee: | Sébastien Han <shan> |
Status: | CLOSED ERRATA | QA Contact: | Vasishta <vashastr> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 3.0 | CC: | dang, gabrioux, hchen, jim.curtis, jtudelag, kdreyer, mhackett, pprakash, rperiyas, shan, tpetr, tserlin, vashastr, vumrao |
Target Milestone: | z4 | ||
Target Release: | 3.0 | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | RHEL: ceph-ansible-3.0.36-1.el7cp Ubuntu: ceph-ansible_3.0.36-2redhat1 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-07-11 18:34:04 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1553254, 1572368 |
Description
jquinn
2018-03-05 15:32:57 UTC
In further testing the below configuration works for the config being pushed to the process. I tried using the FQDN section configured in the ceph_conf_overrides during the install but it would fail. After the install, I am able to just update the ceph.conf to reflect the fqdn and upon restarts of the service it will pull in the config from the ceph.conf. ** current ceph.conf, updated the grace period value in the new section created using the FQDN ** [root@vm250-102 /]# ll /var/log/ceph total 0 [root@vm250-102 /]# cat /etc/ceph/ceph.conf #[client.rgw.vm250-102] #debug_rgw = 20 #host = vm250-102 #keyring = /var/lib/ceph/radosgw/ceph-rgw.vm250-102/keyring #log file = /var/log/ceph/ceph-rgw-vm250-102.log #rgw frontends = civetweb port=10.74.250.102:8080 num_threads=100 #osd_heartbeat_grace = 40 [global] cluster network = 10.74.250.0/21 fsid = c8a35a99-331d-4ac6-b010-ee292b3f6816 mon host = 10.74.250.108 public network = 10.74.250.0/21 rgws_use_fqdn = true [client.rgw.vm250-102.gsslab.pnq2.redhat.com] debug_rgw = 20 osd_heartbeat_grace = 60 host = vm250-102 keyring = /var/lib/ceph/radosgw/ceph-rgw.vm250-102/keyring log file = /var/log/ceph/ceph-rgw-vm250-102.log rgw frontends = civetweb port=10.74.250.102:8080 num_threads=100 [root@vm250-102 /]# ** restarted docker container and now the value is updated ** [root@vm250-102 /]# ceph daemon /var/run/ceph/ceph-client.rgw.vm250-102.gsslab.pnq2.redhat.com.asok config show |grep -i heartbeat_grace "mon_osd_adjust_heartbeat_grace": "true", "osd_heartbeat_grace": "60", [root@vm250-102 /] Thanks, Joe When trying to add multi-site configuration to the RGW running config, the rgw_zone does not get picked up as part of the running config for the RGW process within the container. It appears that the rgw_zone parameter is not allowed to be passed in, and has to be passed in as part of the container startup. See the steps below to workaround this issue. **** Configuration of multi-site steps all complete successfully I have added the "rgw_zone = joezone" to the ceph.conf per docs*** [root@vm250-102 /]# ceph daemon /var/run/ceph/ceph-client.rgw.vm250-102.gsslab.pnq2.redhat.com.asok config show |grep -i rgw_zone "rgw_zone": "", "rgw_zone_root_pool": ".rgw.root", "rgw_zonegroup": "", "rgw_zonegroup_root_pool": ".rgw.root", [root@vm250-102 /]# ** After a restart the rgw_zone is not populated *** [root@vm250-102 /]# ceph daemon /var/run/ceph/ceph-client.rgw.vm250-102.gsslab.pnq2.redhat.com.asok config show |grep -i rgw_zone "rgw_zone": "", "rgw_zone_root_pool": ".rgw.root", "rgw_zonegroup": "", "rgw_zonegroup_root_pool": ".rgw.root", [root@vm250-102 /]# *** Upon docker startup the option for rgw_zone is not included, maybe part of a private class within the code? *** [root@vm250-102 ~]# docker inspect 2ce0c97944a2 |grep -i rgw "Name": "/ceph-rgw-vm250-102", "RGW_CIVETWEB_IP=10.74.250.102", "CEPH_DAEMON=RGW", "RGW_CIVETWEB_PORT=8080", [root@vm250-102 ~]# ** I added the below to rgws.yml, ran the playbook again, updated the ceph.conf to reflect the FQDN for the client per the original issue in the BZ and the rgw_zone ** ceph_rgw_docker_extra_env: "-e RGW_ZONE=joezone" ** The rgw_zone is now passed in as part of initial container configuration *** [root@vm250-102 ~]# docker inspect 7a9e4a9272fb |grep -i rgw "Name": "/ceph-rgw-vm250-102", "RGW_CIVETWEB_IP=10.74.250.102", "CEPH_DAEMON=RGW", "RGW_CIVETWEB_PORT=8080", "RGW_ZONE=joezone", [root@vm250-102 ~]# ** And now the running process has the rgw_zone configuration ** [root@vm250-102 /]# ceph daemon /var/run/ceph/ceph-client.rgw.vm250-102.gsslab.pnq2.redhat.com.asok config show |grep rgw_zone "rgw_zone": "joezone", "rgw_zone_root_pool": ".rgw.root", "rgw_zonegroup": "", "rgw_zonegroup_root_pool": ".rgw.root", [root@vm250-102 /]# Thanks, Joe First, we should try to understand why the ceph.conf is ignored on containerized deployments. I don't think it's ignored, it's just being overwritten by the container CLI. It's just a matter of variable precedence. Let me work on a patch now. Hi Seb, I am fighting with setting performance tuning for RGW in containers, specifically the in all.yml radosgw_civetweb_num_threads: 512 #default 100 in non-container env it is part of rgw_frontends variable "rgw_frontends": "civetweb port=10.10.92.91:8080 num_threads=512", rgw_frontends is overridden by container start of RGW, as there is no num_threads option ----------------------------- # find / -name "start_rgw.sh" /var/lib/docker/overlay2/439572f5c98fe1a2b438171a4cc834e13745676116f914f1ae888375f673570b/diff/start_rgw.sh /var/lib/docker/overlay2/263d778874297c8fee362cdd06cad80924c2b8bb312e06ab3251d1df2555776b/merged/start_rgw.sh [root@rgws-1 ~]# grep num_threads /var/lib/docker/overlay2/439572f5c98fe1a2b438171a4cc834e13745676116f914f1ae888375f673570b/diff/start_rgw.sh: local rgw_frontends="civetweb port=$RGW_CIVETWEB_IP:$RGW_CIVETWEB_PORT" /var/lib/docker/overlay2/263d778874297c8fee362cdd06cad80924c2b8bb312e06ab3251d1df2555776b/merged/start_rgw.sh: local rgw_frontends="civetweb port=$RGW_CIVETWEB_IP:$RGW_CIVETWEB_PORT" -------------------- [root@rgws-1 ~]# docker exec ceph-rgw-rgws-1 ceph --admin-daemon /var/run/ceph/ceph-client.rgw.rgws-1.asok config diff { "diff": { "current": { "admin_socket": "/var/run/ceph/ceph-client.rgw.rgws-1.asok", "cluster_network": "192.168.1.0/28", "err_to_stderr": "true", "fsid": "d5a1e891-afca-4e6f-b2dc-e7c08db07dbf", "internal_safe_to_start_threads": "true", "keyring": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1/keyring", "log_max_recent": "10000", "mds_data": "/var/lib/ceph/mds/ceph-rgw.rgws-1", "mgr_data": "/var/lib/ceph/mgr/ceph-rgw.rgws-1", "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", "mon_data": "/var/lib/ceph/mon/ceph-rgw.rgws-1", "mon_debug_dump_location": "/var/log/ceph/ceph-client.rgw.rgws-1.tdump", "mon_host": "10.10.93.0,10.10.92.94,10.10.93.107", "osd_data": "/var/lib/ceph/osd/ceph-rgw.rgws-1", "osd_journal": "/var/lib/ceph/osd/ceph-rgw.rgws-1/journal", "public_network": "10.10.92.0/22", "rgw_data": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1", "rgw_frontends": "civetweb port=10.10.92.91:8080", <----- "rgw_thread_pool_size": "512", "setgroup": "ceph", "setuser": "ceph" cat /etc/ceph/ceph.conf [client.rgw.rgws-1] host = rgws-1 keyring = /var/lib/ceph/radosgw/ceph-rgw.rgws-1/keyring log file = /var/log/ceph/ceph-rgw-rgws-1.log rgw frontends = civetweb port=10.10.92.91:8080 num_threads=512 rgw_thread_pool_size = 512 --------------------------- editing these files is the only way how to set this parameter in container environment: local rgw_frontends="civetweb port=$RGW_CIVETWEB_IP:$RGW_CIVETWEB_PORT num_threads=512" [root@rgws-1 ~]# docker exec ceph-rgw-rgws-1 ceph --admin-daemon /var/run/ceph/ceph-client.rgw.rgws-1.asok config diff { "diff": { "current": { "admin_socket": "/var/run/ceph/ceph-client.rgw.rgws-1.asok", "cluster_network": "192.168.1.0/28", "err_to_stderr": "true", "fsid": "d5a1e891-afca-4e6f-b2dc-e7c08db07dbf", "internal_safe_to_start_threads": "true", "keyring": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1/keyring", "log_max_recent": "10000", "mds_data": "/var/lib/ceph/mds/ceph-rgw.rgws-1", "mgr_data": "/var/lib/ceph/mgr/ceph-rgw.rgws-1", "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", "mon_data": "/var/lib/ceph/mon/ceph-rgw.rgws-1", "mon_debug_dump_location": "/var/log/ceph/ceph-client.rgw.rgws-1.tdump", "mon_host": "10.10.93.0,10.10.92.94,10.10.93.107", "osd_data": "/var/lib/ceph/osd/ceph-rgw.rgws-1", "osd_journal": "/var/lib/ceph/osd/ceph-rgw.rgws-1/journal", "public_network": "10.10.92.0/22", "rgw_data": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1", "rgw_frontends": "civetweb port=10.10.92.91:8080 num_threads=512", "rgw_thread_pool_size": "512", "setgroup": "ceph", "setuser": "ceph" Tomas, can you please open another BZ for this? I'll fix this. (In reply to leseb from comment #11) > Tomas, can you please open another BZ for this? > I'll fix this. Hi Seb, new BZ#1582411, thanks Thanks Tomas! Hi, I tried different rgw configs and some random values, working fine for me. A conf file snippet - debug rgw = 10 rgw_enable_usage_log = true rgw_bucket_default_quota_max_size = 1024000 rgw bucket default quota max objects = 10000 rgw user max buckets = 333 Please let me know if there are concerns, suggestions. Regards, Vasishta shastry AQE, Ceph Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2178 |