Bug 1283721 - [RFE] Allow for different values of pg_num, pgp_num and size for each Ceph pool
[RFE] Allow for different values of pg_num, pgp_num and size for each Ceph pool
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
urgent Severity high
: rc
: 10.0 (Newton)
Assigned To: Giulio Fidente
Yogev Rabl
: FutureFeature, Reopened, Triaged
: 1292981 (view as bug list)
Depends On: 1252546 1330065
Blocks: 1335593 1356451 1413723 1319336 1387433
  Show dependency treegraph
 
Reported: 2015-11-19 11:38 EST by Felipe Alfaro Solana
Modified: 2017-02-28 21:04 EST (History)
37 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-5.0.0-0.20160907212643.90c852e.2.el7ost
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 1418038 (view as bug list)
Environment:
Last Closed: 2016-12-14 10:18:12 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1517969 None None None Never
OpenStack gerrit 346794 None None None 2016-08-09 08:12 EDT
OpenStack gerrit 346796 None None None 2016-07-26 12:01 EDT

  None (edit)
Description Felipe Alfaro Solana 2015-11-19 11:38:32 EST
Description of problem:
Puppet code used to deploy Ceph completely ignores any customisation done in puppet/hieradata/ceph.yaml, in the sense that all Ceph pools are created with a default size of 64 placement groups.

I am proposing a code change that not only fixes this, but also allows configuring each pool with a different set of parameters: pg_num, pgp_num and size. This makes sense, as usually the Ceph pool used by Glance will be typically smaller than the Ceph pool used to store Cinder volumes or Nova ephermeral disks.

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-0.8.6-71.el7ost.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy an Overcloud
2. All Ceph pools will have pg_num == pgp_num == 64 no matter what is specified in the file puppet/hieradata/ceph.yaml

Actual results:
All Ceph pools will have pg_num == pgp_num == 64 no matter what is specified in the file puppet/hieradata/ceph.yaml

Expected results:
Ceph pools should be configured independently.

Additional info:
I have created a Gerrit change to implement the fix: https://review.openstack.org/#/c/247669
Comment 2 Felipe Alfaro Solana 2015-11-19 11:41:39 EST
Can we please target the fix for 7.2? This is a critical bug that is affecting Telefónica's ability to deploy production OpenStack clusters.
Comment 4 Giulio Fidente 2015-11-20 05:44:55 EST

*** This bug has been marked as a duplicate of bug 1252546 ***
Comment 5 Jaromir Coufal 2015-11-20 07:35:04 EST
(In reply to Felipe Alfaro Solana from comment #2)
> Can we please target the fix for 7.2? This is a critical bug that is
> affecting Telefónica's ability to deploy production OpenStack clusters.

Hi Felipe, thanks a lot for your proposed patch. Unfortunately we cannot add more patches to 7.2 since we did not pass just dev freeze, it is already post blocker freeze. This will have to go to 8.0.
Comment 6 Felipe Alfaro Solana 2015-11-20 10:27:17 EST
Will there be a 7.3 release before 8.0? We aren't going to upgrade to 8.0 until February next year and can't wait that long for this patch to be officially supported.
Comment 7 Jaromir Coufal 2015-11-23 07:18:15 EST
Felipe, there is no plan for 7.3 release before 8, since we are targeting ~2-month release cadence. Can you please send me an e-mail and provide better context? Based on that we will see what we can do to support you. Thanks, Jarda
Comment 9 Jaromir Coufal 2015-12-03 08:03:17 EST
Guilio, is this bug duplicate and therefor fixed in another bugzilla from comment #4?
Comment 10 Giulio Fidente 2015-12-03 11:20:04 EST
hi, no this is not a duplicate, it is a valid RFE and it depends on 1252546

this RFE is about making it possible to use *different* values of pg_num (etc) for different pools; the patch for 1252546 is now merged upstream; we need to review and merge the patch for this BZ instead
Comment 26 Giulio Fidente 2016-09-16 09:30:45 EDT
Sample usage to override the defaults of the 'volumes' pool:

parameter_defaults:
  ExtraConfig:
    CephPools:
      volumes:
        size: 5,
        pg_num: 128
        pgp_num: 128
Comment 29 Yogev Rabl 2016-10-30 11:14:03 EDT
verification failed on openstack-tripleo-heat-templates-5.0.0-0.8.0rc3.el7ost.noarch

I ran the following deployment command

openstack overcloud deploy \ 
--templates /usr/share/openstack-tripleo-heat-templates/ \
-e usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
-e usr/share/openstack-tripleo-heat-templates/environments/net-two-nic-with-vlans.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \ 
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-radosgw.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \
--control-scale 3 \
--compute-scale 2 \
--ceph-storage-scale 3 \
--control-flavor control \
--compute-flavor compute \
--ceph-storage-flavor ceph-storage \
--libvirt-type qemu \
--ntp-server 0.rhel.pool.ntp.org

Storage-environment.yaml content is:

resource_registry:
  OS::TripleO::Services::CephMon: ../puppet/services/ceph-mon.yaml
  OS::TripleO::Services::CephOSD: ../puppet/services/ceph-osd.yaml
  OS::TripleO::Services::CephClient: ../puppet/services/ceph-client.yaml

parameter_defaults:

  #### BACKEND SELECTION ####

  ## Whether to enable iscsi backend for Cinder.
  CinderEnableIscsiBackend: false
  ## Whether to enable rbd (Ceph) backend for Cinder.
  CinderEnableRbdBackend: true
  ## Cinder Backup backend can be either 'ceph' or 'swift'.
  CinderBackupBackend: ceph
  ## Whether to enable NFS backend for Cinder.
  # CinderEnableNfsBackend: false
  ## Whether to enable rbd (Ceph) backend for Nova ephemeral storage.
  NovaEnableRbdBackend: true
  ## Glance backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GlanceBackend: rbd
  ## Gnocchi backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GnocchiBackend: rbd
  ExtraConfig:
    ceph::profile::params::osds:
      '/dev/vda': {}
      '/dev/vdb': {}
      '/dev/vdc': {}
      '/dev/vdd': {}
      '/dev/vde': {}
    ceph::profile::params::osd_crush_update_on_start: false
    CephPools:
      volumes:
        size: 3
        pg_num: 128
        pgp_num: 128
      vms:
        size: 1
        pg_num: 128
        pgp_num: 128
      images:
        size: 5
        pg_num: 128
        pgp_num: 128

and the number of pg_num of the pools is 32
Comment 30 Giulio Fidente 2016-10-31 07:58:32 EDT
hi Yogev, I am *verry* sorry but my syntax in comment #26 was wrong.

CephPools is a Heat parameter, like GlanceBackend and goes at the same level of it, not folded within ExtraConfig (which instead, pushed hieradata on the nodes)

Is there any chance we could try this again and update the BZ?
Comment 31 Yogev Rabl 2016-10-31 08:20:03 EDT
Sure, I'll rerun it with the proper configuration.
Comment 32 Yogev Rabl 2016-10-31 08:20:12 EDT
Sure, I'll rerun it with the proper configuration.
Comment 33 Yogev Rabl 2016-10-31 09:24:18 EDT
Verified with the configuration

resource_registry:
  OS::TripleO::Services::CephMon: ../puppet/services/ceph-mon.yaml
  OS::TripleO::Services::CephOSD: ../puppet/services/ceph-osd.yaml
  OS::TripleO::Services::CephClient: ../puppet/services/ceph-client.yaml

parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: true
  CinderBackupBackend: ceph
  NovaEnableRbdBackend: true
  GlanceBackend: rbd
  GnocchiBackend: rbd
  CephPools:
    volumes:
      size: 3
      pg_num: 128
      pgp_num: 128
    vms:
      size: 1
      pg_num: 128
      pgp_num: 128
    images:
      size: 5
      pg_num: 128
      pgp_num: 128
  ExtraConfig:
    ceph::profile::params::osds:
      '/dev/vda': {}
      '/dev/vdb': {}
      '/dev/vdc': {}
      '/dev/vdd': {}
      '/dev/vde': {}
Comment 36 errata-xmlrpc 2016-12-14 10:18:12 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html
Comment 38 John Fulton 2017-01-11 11:10:49 EST
*** Bug 1292981 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.