Description of problem: FFU: post upgrade ceph cluster reports HEALTH_WARN with too many PGs per OSD (224 > max 200): configuration 3 ceph OSD nodes with 1 OSD disk each: parameter_defaults: CephPoolDefaultPgNum: 32 CephAnsibleDisksConfig: devices: - '/dev/vdb' osd_scenario: collocated Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Deploy OSP10 with 3 controllers + 2 computes + 3 ceph OSD nodes(one OSD disk for each node). Use the following puppet-ceph config: ExtraConfig: ceph::profile::params::osd_pool_default_pg_num: 32 ceph::profile::params::osd_pool_default_pgp_num: 32 ceph::profile::params::osds: '/dev/vdb': {} 2. Upgrade to OSP13 via fast forward procedure 3. Check ceph cluster status Actual results: [root@controller-0 heat-admin]# ceph -s cluster: id: 12246696-4c14-11e8-8efc-5254004cbd7c health: HEALTH_WARN too many PGs per OSD (224 > max 200) services: mon: 3 daemons, quorum controller-1,controller-0,controller-2 mgr: controller-1(active), standbys: controller-0, controller-2 osd: 3 osds: 3 up, 3 in data: pools: 6 pools, 224 pgs objects: 0 objects, 0 bytes usage: 330 MB used, 104 GB / 104 GB avail pgs: 224 active+clean Expected results: Before upgrade the cluster was HEALTH_OK so I'd expect it to be the same. Additional info: If there was a change in the default requirements we should at least document so that operators are aware of the changes.
o/ gfidente discussing on DFG:Upgrades bz triage call can you please look at this or re-assign to someone else in ceph team thanks
This is due to a change in Ceph >= 12.2.1 which lowered the maximum number of PGs allowed for each OSD, before a _WARN message is issued [1] The release notes read as follows: "The maximum number of PGs per OSD before the monitor issues a warning has been reduced from 300 to 200 PGs. 200 is still twice the generally recommended target of 100 PGs per OSD. This limit can be adjusted via the mon_max_pg_per_osd option on the monitors. The older mon_pg_warn_max_per_osd option has been removed." Basically previously consuming 224 PGs was not an issue, not it becomes an issue, but in production environments with much more than 3 OSDs (disks), we should still be operating below the limit ... so this is unlikely to happen in real environments. Yet, the amount of PGs consumed by a pool *can not* be decreased, so for pre-existing deployments, should they be hitting the limit, the best fix is probably to raise the limit to what it was using an environment file like the following: parameter_defaults: CephAnsibleConfigOverrides: mon_max_pg_per_osd: 300 Marius, do you think we should make a release note out of this BZ? 1. http://docs.ceph.com/docs/master/releases/luminous/
(In reply to Giulio Fidente from comment #2) > parameter_defaults: > CephAnsibleConfigOverrides: > mon_max_pg_per_osd: 300 > > Marius, do you think we should make a release note out of this BZ? > Yeah, I'm ok with making it a release note but I tried setting mon_max_pg_per_osd during upgrade and it looks that the override value didn't get applied. This is the environment file containing the ceph-ansible parameters used during upgrade: parameter_defaults: CephPoolDefaultPgNum: 32 CephAnsibleDisksConfig: devices: - '/dev/vdb' osd_scenario: collocated CephAnsibleConfigOverrides: mon_max_pg_per_osd: 300 After upgrade the mon_max_pg_per_osd value is still set to 200: [root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash [root@controller-0 /]# ceph --admin-daemon /var/run/ceph/ceph-mon.controller-0.asok config get mon_max_pg_per_osd { "mon_max_pg_per_osd": "200" }
(In reply to Marius Cornea from comment #3) > (In reply to Giulio Fidente from comment #2) > > parameter_defaults: > > CephAnsibleConfigOverrides: > > mon_max_pg_per_osd: 300 > > > > Marius, do you think we should make a release note out of this BZ? > > > > Yeah, I'm ok with making it a release note but I tried setting > mon_max_pg_per_osd during upgrade and it looks that the override value > didn't get applied. > > > This is the environment file containing the ceph-ansible parameters used > during upgrade: > > parameter_defaults: > CephPoolDefaultPgNum: 32 > CephAnsibleDisksConfig: > devices: > - '/dev/vdb' > osd_scenario: collocated > CephAnsibleConfigOverrides: > mon_max_pg_per_osd: 300 > > After upgrade the mon_max_pg_per_osd value is still set to 200: > > [root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash > [root@controller-0 /]# ceph --admin-daemon > /var/run/ceph/ceph-mon.controller-0.asok config get mon_max_pg_per_osd > { > "mon_max_pg_per_osd": "200" > } Thanks Marius! Can you confirm you see the new key into ceph.conf? If so we might need special cmdline to inject the new setting: # ceph tell osd.* injectargs '--mon-max-pg-per-osd 300'
(In reply to Giulio Fidente from comment #4) > (In reply to Marius Cornea from comment #3) > > (In reply to Giulio Fidente from comment #2) > > > parameter_defaults: > > > CephAnsibleConfigOverrides: > > > mon_max_pg_per_osd: 300 > > > > > > Marius, do you think we should make a release note out of this BZ? > > > > > > > Yeah, I'm ok with making it a release note but I tried setting > > mon_max_pg_per_osd during upgrade and it looks that the override value > > didn't get applied. > > > > > > This is the environment file containing the ceph-ansible parameters used > > during upgrade: > > > > parameter_defaults: > > CephPoolDefaultPgNum: 32 > > CephAnsibleDisksConfig: > > devices: > > - '/dev/vdb' > > osd_scenario: collocated > > CephAnsibleConfigOverrides: > > mon_max_pg_per_osd: 300 > > > > After upgrade the mon_max_pg_per_osd value is still set to 200: > > > > [root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash > > [root@controller-0 /]# ceph --admin-daemon > > /var/run/ceph/ceph-mon.controller-0.asok config get mon_max_pg_per_osd > > { > > "mon_max_pg_per_osd": "200" > > } > > Thanks Marius! > > Can you confirm you see the new key into ceph.conf? > > If so we might need special cmdline to inject the new setting: > > # ceph tell osd.* injectargs '--mon-max-pg-per-osd 300' So it looks like the option didn't make it into ceph.conf, I'm wondering if I'm missing something here on the Director configuration side: [root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash [root@controller-0 /]# grep mon_max_pg_per_osd /etc/ceph/ceph.conf [root@controller-0 /]# [root@controller-0 /]#
(In reply to Marius Cornea from comment #5) > (In reply to Giulio Fidente from comment #4) > > (In reply to Marius Cornea from comment #3) > > > (In reply to Giulio Fidente from comment #2) > > > > parameter_defaults: > > > > CephAnsibleConfigOverrides: > > > > mon_max_pg_per_osd: 300 > > > > > > > > Marius, do you think we should make a release note out of this BZ? > > > > > > > > > > Yeah, I'm ok with making it a release note but I tried setting > > > mon_max_pg_per_osd during upgrade and it looks that the override value > > > didn't get applied. > > > > > > > > > This is the environment file containing the ceph-ansible parameters used > > > during upgrade: > > > > > > parameter_defaults: > > > CephPoolDefaultPgNum: 32 > > > CephAnsibleDisksConfig: > > > devices: > > > - '/dev/vdb' > > > osd_scenario: collocated > > > CephAnsibleConfigOverrides: > > > mon_max_pg_per_osd: 300 > > > > > > After upgrade the mon_max_pg_per_osd value is still set to 200: > > > > > > [root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash > > > [root@controller-0 /]# ceph --admin-daemon > > > /var/run/ceph/ceph-mon.controller-0.asok config get mon_max_pg_per_osd > > > { > > > "mon_max_pg_per_osd": "200" > > > } > > > > Thanks Marius! > > > > Can you confirm you see the new key into ceph.conf? > > > > If so we might need special cmdline to inject the new setting: > > > > # ceph tell osd.* injectargs '--mon-max-pg-per-osd 300' > > So it looks like the option didn't make it into ceph.conf, I'm wondering if > I'm missing something here on the Director configuration side: > > [root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash > [root@controller-0 /]# grep mon_max_pg_per_osd /etc/ceph/ceph.conf > [root@controller-0 /]# > [root@controller-0 /]# OK, so it seems the correct parameter is CephConfigOverrides, the below environment file made it get the cluster in a HEALTH_OK state: parameter_defaults: CephPoolDefaultPgNum: 32 CephAnsibleDisksConfig: devices: - '/dev/vdb' osd_scenario: collocated CephConfigOverrides: mon_max_pg_per_osd: 300