Description of problem: I am fighting with setting performance tuning for RGW in containers, in all.yml you can set the tuning parameter "radosgw_civetweb_num_threads": radosgw_civetweb_num_threads: 512 #default 100 in non-container env it is part of rgw_frontends variable, that is correct: "rgw_frontends": "civetweb port=10.10.92.91:8080 num_threads=512", In container environment rgw_frontends configuration from ceph.conf is overridden (or not used, I am not familiar with the process flow) by container start of RGW, as there is no num_threads option ----------------------------- # find / -name "start_rgw.sh" /var/lib/docker/overlay2/439572f5c98fe1a2b438171a4cc834e13745676116f914f1ae888375f673570b/diff/start_rgw.sh /var/lib/docker/overlay2/263d778874297c8fee362cdd06cad80924c2b8bb312e06ab3251d1df2555776b/merged/start_rgw.sh [root@rgws-1 ~]# grep num_threads /var/lib/docker/overlay2/439572f5c98fe1a2b438171a4cc834e13745676116f914f1ae888375f673570b/diff/start_rgw.sh: local rgw_frontends="civetweb port=$RGW_CIVETWEB_IP:$RGW_CIVETWEB_PORT" /var/lib/docker/overlay2/263d778874297c8fee362cdd06cad80924c2b8bb312e06ab3251d1df2555776b/merged/start_rgw.sh: local rgw_frontends="civetweb port=$RGW_CIVETWEB_IP:$RGW_CIVETWEB_PORT" -------------------- [root@rgws-1 ~]# docker exec ceph-rgw-rgws-1 ceph --admin-daemon /var/run/ceph/ceph-client.rgw.rgws-1.asok config diff { "diff": { "current": { "admin_socket": "/var/run/ceph/ceph-client.rgw.rgws-1.asok", "cluster_network": "192.168.1.0/28", "err_to_stderr": "true", "fsid": "d5a1e891-afca-4e6f-b2dc-e7c08db07dbf", "internal_safe_to_start_threads": "true", "keyring": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1/keyring", "log_max_recent": "10000", "mds_data": "/var/lib/ceph/mds/ceph-rgw.rgws-1", "mgr_data": "/var/lib/ceph/mgr/ceph-rgw.rgws-1", "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", "mon_data": "/var/lib/ceph/mon/ceph-rgw.rgws-1", "mon_debug_dump_location": "/var/log/ceph/ceph-client.rgw.rgws-1.tdump", "mon_host": "10.10.93.0,10.10.92.94,10.10.93.107", "osd_data": "/var/lib/ceph/osd/ceph-rgw.rgws-1", "osd_journal": "/var/lib/ceph/osd/ceph-rgw.rgws-1/journal", "public_network": "10.10.92.0/22", "rgw_data": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1", "rgw_frontends": "civetweb port=10.10.92.91:8080", <----- "rgw_thread_pool_size": "512", "setgroup": "ceph", "setuser": "ceph" cat /etc/ceph/ceph.conf [client.rgw.rgws-1] host = rgws-1 keyring = /var/lib/ceph/radosgw/ceph-rgw.rgws-1/keyring log file = /var/log/ceph/ceph-rgw-rgws-1.log rgw frontends = civetweb port=10.10.92.91:8080 num_threads=512 <----- rgw_thread_pool_size = 512 --------------------------- editing these files is the only way how to set this parameter in container environment: local rgw_frontends="civetweb port=$RGW_CIVETWEB_IP:$RGW_CIVETWEB_PORT num_threads=512" [root@rgws-1 ~]# docker exec ceph-rgw-rgws-1 ceph --admin-daemon /var/run/ceph/ceph-client.rgw.rgws-1.asok config diff { "diff": { "current": { "admin_socket": "/var/run/ceph/ceph-client.rgw.rgws-1.asok", "cluster_network": "192.168.1.0/28", "err_to_stderr": "true", "fsid": "d5a1e891-afca-4e6f-b2dc-e7c08db07dbf", "internal_safe_to_start_threads": "true", "keyring": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1/keyring", "log_max_recent": "10000", "mds_data": "/var/lib/ceph/mds/ceph-rgw.rgws-1", "mgr_data": "/var/lib/ceph/mgr/ceph-rgw.rgws-1", "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", "mon_data": "/var/lib/ceph/mon/ceph-rgw.rgws-1", "mon_debug_dump_location": "/var/log/ceph/ceph-client.rgw.rgws-1.tdump", "mon_host": "10.10.93.0,10.10.92.94,10.10.93.107", "osd_data": "/var/lib/ceph/osd/ceph-rgw.rgws-1", "osd_journal": "/var/lib/ceph/osd/ceph-rgw.rgws-1/journal", "public_network": "10.10.92.0/22", "rgw_data": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1", "rgw_frontends": "civetweb port=10.10.92.91:8080 num_threads=512", "rgw_thread_pool_size": "512", "setgroup": "ceph", "setuser": "ceph" Version-Release number of selected component (if applicable): ceph-radosgw-12.2.4-6.el7cp.x86_64 ceph-ansible-3.0.33-1.el7cp.noarch docker-1.13.1-53.git774336d.el7.x86_64 How reproducible: Always Steps to Reproduce: 1. deploy Ceph with site-docker.yml 2. in all.yml you set the tuning parameter "radosgw_civetweb_num_threads" 3. Actual results: Expected results: Additional info:
[root@rgws-1 ~]# docker inspect ceph-rgw-rgws-1 [ { "Id": "0c47b95650de6f68773d16d1f7c86e25a094acc7960d2123f8993876c707d37e", "Created": "2018-05-22T09:29:03.671062759Z", "Path": "/entrypoint.sh", "Args": [], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 11080, "ExitCode": 0, "Error": "", "StartedAt": "2018-05-22T09:29:03.937237779Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:9f92f1dc96eccd12eda1e809a3539e58f83faad6289a21beb1a6ebac05b91f42", "ResolvConfPath": "/var/lib/docker/containers/0c47b95650de6f68773d16d1f7c86e25a094acc7960d2123f8993876c707d37e/resolv.conf", "HostnamePath": "/var/lib/docker/containers/0c47b95650de6f68773d16d1f7c86e25a094acc7960d2123f8993876c707d37e/hostname", "HostsPath": "/var/lib/docker/containers/0c47b95650de6f68773d16d1f7c86e25a094acc7960d2123f8993876c707d37e/hosts", "LogPath": "", "Name": "/ceph-rgw-rgws-1", "RestartCount": 0, "Driver": "overlay2", "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c110,c264", "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c110,c264", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "/var/lib/ceph:/var/lib/ceph", "/etc/ceph:/etc/ceph", "/etc/localtime:/etc/localtime:ro" ], "ContainerIDFile": "", "LogConfig": { "Type": "journald", "Config": {} }, "NetworkMode": "host", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": true, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "docker-runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 2147483648, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 200000, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DiskQuota": 0, "KernelMemory": 0, "MemoryReservation": 0, "MemorySwap": 4294967296, "MemorySwappiness": -1, "OomKillDisable": false, "PidsLimit": 0, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0 }, "GraphDriver": { "Name": "overlay2", "Data": { "LowerDir": "/var/lib/docker/overlay2/263d778874297c8fee362cdd06cad80924c2b8bb312e06ab3251d1df2555776b-init/diff:/var/lib/docker/overlay2/439572f5c98fe1a2b438171a4cc834e13745676116f914f1ae888375f673570b/diff:/var/lib/docker/overlay2/7c2723bc210e2197ee5434cb1c40f3f0d937a48a3df282d4b37c7833aeb59592/diff:/var/lib/docker/overlay2/833f5e514da7ef110d2eddf9a33ec26284fcd0a3f8d129abfe1a50f3a14cfcbf/diff", "MergedDir": "/var/lib/docker/overlay2/263d778874297c8fee362cdd06cad80924c2b8bb312e06ab3251d1df2555776b/merged", "UpperDir": "/var/lib/docker/overlay2/263d778874297c8fee362cdd06cad80924c2b8bb312e06ab3251d1df2555776b/diff", "WorkDir": "/var/lib/docker/overlay2/263d778874297c8fee362cdd06cad80924c2b8bb312e06ab3251d1df2555776b/work" } }, "Mounts": [ { "Type": "bind", "Source": "/var/lib/ceph", "Destination": "/var/lib/ceph", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/ceph", "Destination": "/etc/ceph", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/localtime", "Destination": "/etc/localtime", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "volume", "Name": "4493aa87c8cd704288bbcd01665d900156aa25ba99b95e0d5844faaa10b5e328", "Source": "/var/lib/docker/volumes/4493aa87c8cd704288bbcd01665d900156aa25ba99b95e0d5844faaa10b5e328/_data", "Destination": "/etc/ganesha", "Driver": "local", "Mode": "", "RW": true, "Propagation": "" } ], "Config": { "Hostname": "rgws-1.container.quicklab.rdu2.cee.redhat.com", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "ExposedPorts": { "5000/tcp": {}, "6789/tcp": {}, "6800/tcp": {}, "6801/tcp": {}, "6802/tcp": {}, "6803/tcp": {}, "6804/tcp": {}, "6805/tcp": {}, "80/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "RGW_CIVETWEB_IP=10.10.92.91", "CEPH_DAEMON=RGW", "CLUSTER=ceph", "RGW_CIVETWEB_PORT=8080", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "container=oci", "CEPH_VERSION=luminous", "CEPH_POINT_RELEASE=" ], "Cmd": null, "ArgsEscaped": true, "Image": "registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest", "Volumes": { "/etc/ceph": {}, "/etc/ganesha": {}, "/var/lib/ceph": {} }, "WorkingDir": "/", "Entrypoint": [ "/entrypoint.sh" ], "OnBuild": null, "Labels": { "CEPH_POINT_RELEASE": "", "GIT_BRANCH": "master", "GIT_CLEAN": "True", "GIT_COMMIT": "99f689cd2c12f8332924db6a0cc0463bb26631b0", "GIT_REPO": "git:ceph/ceph-container.git", "RELEASE": "master", "architecture": "x86_64", "authoritative-source-url": "registry.access.redhat.com", "build-date": "2018-04-18T13:01:58.678631", "com.redhat.build-host": "ip-10-29-120-145.ec2.internal", "com.redhat.component": "rhceph-rhel7-docker", "description": "Red Hat Ceph Storage 3", "distribution-scope": "public", "install": "/usr/bin/docker run --rm --privileged -v /:/host -e MON_IP=${MON_IP} -e CEPH_PUBLIC_NETWORK=${CEPH_PUBLIC_NETWORK} -e CEPH_DAEMON=${CEPH_DAEMON} -e MON_NAME=${MON_NAME} -e OSD_DEVICE=${OSD_DEVICE} -e HOST=/host -e IMAGE=${IMAGE} --entrypoint=/install.sh ${IMAGE}", "io.k8s.description": "Red Hat Ceph Storage 3", "io.k8s.display-name": "Red Hat Ceph Storage 3 on RHEL 7", "io.openshift.tags": "rhceph ceph", "name": "rhceph", "release": "6", "run": "/usr/bin/docker run -d --net=host --pid=host -e MON_NAME=${MON_NAME} -e MON_IP=${MON_IP} -e CEPH_PUBLIC_NETWORK=${CEPH_PUBLIC_NETWORK} -e CEPH_DAEMON=${CEPH_DAEMON} -v /etc/ceph:/etc/ceph -v /var/lib/ceph:/var/lib/ceph ${IMAGE}", "summary": "Provides the latest Red Hat Ceph Storage 3 on RHEL 7 in a fully featured and supported base image.", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/3-6", "vcs-ref": "9fe91bb07dc2b866b3bd024bbaf43f09d4eb05e9", "vcs-type": "git", "vendor": "Red Hat, Inc.", "version": "3" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "dc6d0017cc78f7ddfa04bc35545e85f332c01211d26ae09a30be17d9d096ed63", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/default", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "host": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "6c222e50bc44578677a1cce4b250769a8b57bf3218590f831330117d56e8df49", "EndpointID": "6d2d85251ba0509b150d7713bdfdf3f135a058024bf56670ad9a5f034c8c097c", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "" } } } } ]
Is this not same as - https://bugzilla.redhat.com/show_bug.cgi?id=1551637? Or I am missing something.
Vikhyat, it's different, it's another option.
(In reply to leseb from comment #5) > Vikhyat, it's different, it's another option. Ahh right. It is with one particular configuration thanks for the feedback, leseb.
remote: *** Checking commit 6c1f7c9c2fcc5a65e63c0d4e4cb38c1a99780b9d remote: *** Resolves: remote: *** Approved: remote: *** rhbz#1582411 (qa_ack?, ceph-3.0?, ceph-3.y?, pm_ack+, devel_ack?) remote: *** Commit 6c1f7c9c2fcc5a65e63c0d4e4cb38c1a99780b9d allowed remote: * Publishing information for 1 commits To ssh://pkgs.devel.redhat.com/containers/rhceph-rhel7
In https://github.com/ceph/ceph-ansible/releases/tag/v3.1.0rc16
Hi tomas, you should see it in the ceph.conf, can you share it so I can have a look? Thanks
(In reply to leseb from comment #21) > Hi tomas, you should see it in the ceph.conf, can you share it so I can have > a look? > Thanks Hi Seb, yes it is in ceph.conf, no issue with that part: [root@rgws-1 ~]# cat /etc/ceph/ceph.conf [client.rgw.rgws-0] host = rgws-0 keyring = /var/lib/ceph/radosgw/ceph-rgw.rgws-0/keyring log file = /var/log/ceph/ceph-rgw-rgws-0.log rgw frontends = civetweb port=10.10.95.187:8080 num_threads=512 [client.rgw.rgws-1] host = rgws-1 keyring = /var/lib/ceph/radosgw/ceph-rgw.rgws-1/keyring log file = /var/log/ceph/ceph-rgw-rgws-1.log rgw frontends = civetweb port=10.10.95.186:8080 num_threads=512 <-------- rgw_user_max_buckets = 666 rgw_zone = tpetrzone [global] cluster network = 192.168.1.0/28 debug_rgw = 0/1 fsid = 9f76a45a-acc3-4a08-b88b-5faaa0a067ab log file = /dev/null mon host = 10.10.95.184,10.10.95.17,10.10.95.185 public network = 10.10.92.0/22 but not in the runtime config: [root@rgws-1 ~]# docker exec ceph-rgw-rgws-1 ceph daemon /var/run/ceph/ceph-client.rgw.rgws-1.asok config diff { "diff": { "current": { "admin_socket": "/var/run/ceph/ceph-client.rgw.rgws-1.asok", "cluster_network": "192.168.1.0/28", "err_to_stderr": "true", "fsid": "9f76a45a-acc3-4a08-b88b-5faaa0a067ab", "internal_safe_to_start_threads": "true", "keyring": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1/keyring", "log_max_recent": "10000", "mds_data": "/var/lib/ceph/mds/ceph-rgw.rgws-1", "mgr_data": "/var/lib/ceph/mgr/ceph-rgw.rgws-1", "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", "mon_data": "/var/lib/ceph/mon/ceph-rgw.rgws-1", "mon_debug_dump_location": "/var/log/ceph/ceph-client.rgw.rgws-1.tdump", "mon_host": "10.10.95.184,10.10.95.17,10.10.95.185", "osd_data": "/var/lib/ceph/osd/ceph-rgw.rgws-1", "osd_journal": "/var/lib/ceph/osd/ceph-rgw.rgws-1/journal", "public_network": "10.10.92.0/22", "rgw_data": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1", "rgw_frontends": "civetweb port=10.10.95.186:8080", <-------- "rgw_user_max_buckets": "666", "setgroup": "ceph", "setuser": "ceph"
Weird, do we need double quotes?
(In reply to leseb from comment #23) > Weird, do we need double quotes? This is how the civetweb num_threads is defined in all.yml: ./roles/ceph-defaults/defaults/main.yml:376:radosgw_civetweb_port: 8080 ./roles/ceph-defaults/defaults/main.yml:377:radosgw_civetweb_num_threads: 100 ./roles/ceph-defaults/defaults/main.yml:378:radosgw_civetweb_options: "num_threads={{ radosgw_civetweb_num_threads }}" to put it in ceph.conf works, and from there is should be read by rgw process (in non container environment it is and works perfectly): ./roles/ceph-config/templates/ceph.conf.j2:185:rgw frontends = {{ radosgw_frontend_type }} {{ 'port' if radosgw_frontend_type == 'civetweb' else 'endpoint' }}=[{{ hostvars[host]['ansible_all_' + ip_version + '_addresses'] | ipaddr(radosgw_address_block) | first }}]:{{ radosgw_frontend_port }} {{ radosgw_frontend_options }} ./roles/ceph-config/templates/ceph.conf.j2:189:rgw frontends = {{ radosgw_frontend_type }} {{ 'port' if radosgw_frontend_type == 'civetweb' else 'endpoint' }}={{ radosgw_address_block | ipaddr(radosgw_address_block) | first }}:{{ radosgw_frontend_port }} {{ radosgw_frontend_options }} so checking the same in non-container env would return: "rgw_frontends": "civetweb port=10.10.92.91:8080 num_threads=512", <-------- But how to force it into container env, I do not know.
The container also reads the ceph.conf now and is not forcing any CLI option, if you look at the rgw process in the container you will see we are not passing --rgw-frontend anymore. Hence my question, can you try putting quotes around maybe? Thanks.
(In reply to leseb from comment #25) > The container also reads the ceph.conf now and is not forcing any CLI > option, if you look at the rgw process in the container you will see we are > not passing --rgw-frontend anymore. > Hence my question, can you try putting quotes around maybe? > > Thanks. Hi Seb, it did not helped: [root@rgws-1 ~]# docker exec ceph-rgw-rgws-1 cat /etc/ceph/ceph.conf |grep rgw [client.rgw.rgws-0] host = rgws-0 keyring = /var/lib/ceph/radosgw/ceph-rgw.rgws-0/keyring log file = /var/log/ceph/ceph-rgw-rgws-0.log rgw frontends = civetweb port=10.10.95.239:8080 num_threads=100 [client.rgw.rgws-1] host = rgws-1 keyring = /var/lib/ceph/radosgw/ceph-rgw.rgws-1/keyring log file = /var/log/ceph/ceph-rgw-rgws-1.log rgw frontends = "civetweb port=10.10.95.238:8080 num_threads=100" rgw_user_max_buckets = 666 rgw_zone = tpetrzone debug_rgw = 0/1 [root@rgws-1 ~]# docker exec ceph-rgw-rgws-1 ceph daemon /var/run/ceph/ceph-client.rgw.rgws-1.asok config diff | grep rgw "admin_socket": "/var/run/ceph/ceph-client.rgw.rgws-1.asok", "keyring": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1/keyring", "mds_data": "/var/lib/ceph/mds/ceph-rgw.rgws-1", "mgr_data": "/var/lib/ceph/mgr/ceph-rgw.rgws-1", "mon_data": "/var/lib/ceph/mon/ceph-rgw.rgws-1", "mon_debug_dump_location": "/var/log/ceph/ceph-client.rgw.rgws-1.tdump", "osd_data": "/var/lib/ceph/osd/ceph-rgw.rgws-1", "osd_journal": "/var/lib/ceph/osd/ceph-rgw.rgws-1/journal", "rgw_data": "/var/lib/ceph/radosgw/ceph-rgw.rgws-1", "rgw_frontends": "civetweb port=10.10.95.238:8080", "rgw_user_max_buckets": "666",
*** Bug 1631086 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2820