Bug 1575118 - FFU: post upgrade ceph cluster reports HEALTH_WARN with too many PGs per OSD (224 > max 200)
Summary: FFU: post upgrade ceph cluster reports HEALTH_WARN with too many PGs per OSD ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: ceph
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 13.0 (Queens)
Assignee: Giulio Fidente
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-04 19:38 UTC by Marius Cornea
Modified: 2018-06-28 15:17 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Ceph Release 12.2.1 lowers the maximum number of PGs allowed for each OSD. The lower limit may cause the monitor to prematurely issue a HEALTH_WARN message. The monitor warning threshold has been reduced from 300 to 200 PGs per OSD. 200 is still twice the generally recommended target of 100 PGs per OSD. This limit can be adjusted via the mon_max_pg_per_osd option on the monitors. The older mon_pg_warn_max_per_osd option has been removed. The amount of PGs consumed by a pool can not be decreased. If the upgrade causes a pre-existing deployment to reach the maximum limit, you can raise the limit to its pre-upgrade value during the ceph-upgrade step. In an environment file, add a parameter setting like this: parameter_defaults: CephConfigOverrides: mon_max_pg_per_osd: 300 The setting is applied into ceph.conf and the cluster stays in HEALTH_OK state.
Clone Of:
Environment:
Last Closed: 2018-06-28 15:17:06 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Marius Cornea 2018-05-04 19:38:57 UTC
Description of problem:

FFU: post upgrade ceph cluster reports HEALTH_WARN with too many PGs per OSD (224 > max 200):

configuration 3 ceph OSD nodes with 1 OSD disk each:

parameter_defaults:
    CephPoolDefaultPgNum:  32
    CephAnsibleDisksConfig:
        devices:
            - '/dev/vdb'
        osd_scenario: collocated


Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP10 with 3 controllers + 2 computes + 3 ceph OSD nodes(one OSD disk for each node). Use the following puppet-ceph config:

    ExtraConfig:
      ceph::profile::params::osd_pool_default_pg_num: 32
      ceph::profile::params::osd_pool_default_pgp_num: 32
      ceph::profile::params::osds:
       '/dev/vdb': {}

2. Upgrade to OSP13 via fast forward procedure

3. Check ceph cluster status

Actual results:
[root@controller-0 heat-admin]# ceph -s
  cluster:
    id:     12246696-4c14-11e8-8efc-5254004cbd7c
    health: HEALTH_WARN
            too many PGs per OSD (224 > max 200)
 
  services:
    mon: 3 daemons, quorum controller-1,controller-0,controller-2
    mgr: controller-1(active), standbys: controller-0, controller-2
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   6 pools, 224 pgs
    objects: 0 objects, 0 bytes
    usage:   330 MB used, 104 GB / 104 GB avail
    pgs:     224 active+clean


Expected results:
Before upgrade the cluster was HEALTH_OK so I'd expect it to be the same.

Additional info:

If there was a change in the default requirements we should at least document so that operators are aware of the changes.

Comment 1 Marios Andreou 2018-05-07 12:35:57 UTC
o/ gfidente discussing on DFG:Upgrades bz triage call can you please look at this or re-assign to someone else in ceph team thanks

Comment 2 Giulio Fidente 2018-05-07 15:18:02 UTC
This is due to a change in Ceph >= 12.2.1 which lowered the maximum number of PGs allowed for each OSD, before a _WARN message is issued [1]

The release notes read as follows: "The maximum number of PGs per OSD before the monitor issues a warning has been reduced from 300 to 200 PGs. 200 is still twice the generally recommended target of 100 PGs per OSD. This limit can be adjusted via the mon_max_pg_per_osd option on the monitors. The older mon_pg_warn_max_per_osd option has been removed."

Basically previously consuming 224 PGs was not an issue, not it becomes an issue, but in production environments with much more than 3 OSDs (disks), we should still be operating below the limit ... so this is unlikely to happen in real environments.

Yet, the amount of PGs consumed by a pool *can not* be decreased, so for pre-existing deployments, should they be hitting the limit, the best fix is probably to raise the limit to what it was using an environment file like the following:

parameter_defaults:
  CephAnsibleConfigOverrides:
    mon_max_pg_per_osd: 300

Marius, do you think we should make a release note out of this BZ?

1. http://docs.ceph.com/docs/master/releases/luminous/

Comment 3 Marius Cornea 2018-05-07 19:28:06 UTC
(In reply to Giulio Fidente from comment #2)
> parameter_defaults:
>   CephAnsibleConfigOverrides:
>     mon_max_pg_per_osd: 300
> 
> Marius, do you think we should make a release note out of this BZ?
> 

Yeah, I'm ok with making it a release note but I tried setting mon_max_pg_per_osd during upgrade and it looks that the override value didn't get applied.


This is the environment file containing the ceph-ansible parameters used during upgrade:

parameter_defaults:
    CephPoolDefaultPgNum:  32
    CephAnsibleDisksConfig:
        devices:
            - '/dev/vdb'
        osd_scenario: collocated
    CephAnsibleConfigOverrides:
      mon_max_pg_per_osd: 300

After upgrade the mon_max_pg_per_osd value is still set to 200:

[root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash
[root@controller-0 /]# ceph --admin-daemon /var/run/ceph/ceph-mon.controller-0.asok config get mon_max_pg_per_osd
{
    "mon_max_pg_per_osd": "200"
}

Comment 4 Giulio Fidente 2018-05-07 19:57:59 UTC
(In reply to Marius Cornea from comment #3)
> (In reply to Giulio Fidente from comment #2)
> > parameter_defaults:
> >   CephAnsibleConfigOverrides:
> >     mon_max_pg_per_osd: 300
> > 
> > Marius, do you think we should make a release note out of this BZ?
> > 
> 
> Yeah, I'm ok with making it a release note but I tried setting
> mon_max_pg_per_osd during upgrade and it looks that the override value
> didn't get applied.
> 
> 
> This is the environment file containing the ceph-ansible parameters used
> during upgrade:
> 
> parameter_defaults:
>     CephPoolDefaultPgNum:  32
>     CephAnsibleDisksConfig:
>         devices:
>             - '/dev/vdb'
>         osd_scenario: collocated
>     CephAnsibleConfigOverrides:
>       mon_max_pg_per_osd: 300
> 
> After upgrade the mon_max_pg_per_osd value is still set to 200:
> 
> [root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash
> [root@controller-0 /]# ceph --admin-daemon
> /var/run/ceph/ceph-mon.controller-0.asok config get mon_max_pg_per_osd
> {
>     "mon_max_pg_per_osd": "200"
> }

Thanks Marius!

Can you confirm you see the new key into ceph.conf?

If so we might need special cmdline to inject the new setting:

  # ceph tell osd.* injectargs '--mon-max-pg-per-osd 300'

Comment 5 Marius Cornea 2018-05-07 20:01:09 UTC
(In reply to Giulio Fidente from comment #4)
> (In reply to Marius Cornea from comment #3)
> > (In reply to Giulio Fidente from comment #2)
> > > parameter_defaults:
> > >   CephAnsibleConfigOverrides:
> > >     mon_max_pg_per_osd: 300
> > > 
> > > Marius, do you think we should make a release note out of this BZ?
> > > 
> > 
> > Yeah, I'm ok with making it a release note but I tried setting
> > mon_max_pg_per_osd during upgrade and it looks that the override value
> > didn't get applied.
> > 
> > 
> > This is the environment file containing the ceph-ansible parameters used
> > during upgrade:
> > 
> > parameter_defaults:
> >     CephPoolDefaultPgNum:  32
> >     CephAnsibleDisksConfig:
> >         devices:
> >             - '/dev/vdb'
> >         osd_scenario: collocated
> >     CephAnsibleConfigOverrides:
> >       mon_max_pg_per_osd: 300
> > 
> > After upgrade the mon_max_pg_per_osd value is still set to 200:
> > 
> > [root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash
> > [root@controller-0 /]# ceph --admin-daemon
> > /var/run/ceph/ceph-mon.controller-0.asok config get mon_max_pg_per_osd
> > {
> >     "mon_max_pg_per_osd": "200"
> > }
> 
> Thanks Marius!
> 
> Can you confirm you see the new key into ceph.conf?
> 
> If so we might need special cmdline to inject the new setting:
> 
>   # ceph tell osd.* injectargs '--mon-max-pg-per-osd 300'

So it looks like the option didn't make it into ceph.conf, I'm wondering if I'm missing something here on the Director configuration side:

[root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash
[root@controller-0 /]# grep mon_max_pg_per_osd  /etc/ceph/ceph.conf 
[root@controller-0 /]# 
[root@controller-0 /]#

Comment 6 Marius Cornea 2018-05-07 20:34:15 UTC
(In reply to Marius Cornea from comment #5)
> (In reply to Giulio Fidente from comment #4)
> > (In reply to Marius Cornea from comment #3)
> > > (In reply to Giulio Fidente from comment #2)
> > > > parameter_defaults:
> > > >   CephAnsibleConfigOverrides:
> > > >     mon_max_pg_per_osd: 300
> > > > 
> > > > Marius, do you think we should make a release note out of this BZ?
> > > > 
> > > 
> > > Yeah, I'm ok with making it a release note but I tried setting
> > > mon_max_pg_per_osd during upgrade and it looks that the override value
> > > didn't get applied.
> > > 
> > > 
> > > This is the environment file containing the ceph-ansible parameters used
> > > during upgrade:
> > > 
> > > parameter_defaults:
> > >     CephPoolDefaultPgNum:  32
> > >     CephAnsibleDisksConfig:
> > >         devices:
> > >             - '/dev/vdb'
> > >         osd_scenario: collocated
> > >     CephAnsibleConfigOverrides:
> > >       mon_max_pg_per_osd: 300
> > > 
> > > After upgrade the mon_max_pg_per_osd value is still set to 200:
> > > 
> > > [root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash
> > > [root@controller-0 /]# ceph --admin-daemon
> > > /var/run/ceph/ceph-mon.controller-0.asok config get mon_max_pg_per_osd
> > > {
> > >     "mon_max_pg_per_osd": "200"
> > > }
> > 
> > Thanks Marius!
> > 
> > Can you confirm you see the new key into ceph.conf?
> > 
> > If so we might need special cmdline to inject the new setting:
> > 
> >   # ceph tell osd.* injectargs '--mon-max-pg-per-osd 300'
> 
> So it looks like the option didn't make it into ceph.conf, I'm wondering if
> I'm missing something here on the Director configuration side:
> 
> [root@controller-0 heat-admin]# docker exec -it ceph-mon-controller-0 bash
> [root@controller-0 /]# grep mon_max_pg_per_osd  /etc/ceph/ceph.conf 
> [root@controller-0 /]# 
> [root@controller-0 /]#

OK, so it seems the correct parameter is CephConfigOverrides, the below environment file made it get the cluster in a HEALTH_OK state:

parameter_defaults:
    CephPoolDefaultPgNum:  32
    CephAnsibleDisksConfig:
        devices:
            - '/dev/vdb'
        osd_scenario: collocated
    CephConfigOverrides:
      mon_max_pg_per_osd: 300


Note You need to log in before you can comment on or make changes to this bug.