Bug 1283721

Summary: [RFE] Allow for different values of pg_num, pgp_num and size for each Ceph pool
Product: Red Hat OpenStack Reporter: Felipe Alfaro Solana <felipe.alfaro>
Component: openstack-tripleo-heat-templatesAssignee: Giulio Fidente <gfidente>
Status: CLOSED ERRATA QA Contact: Yogev Rabl <yrabl>
Severity: high Docs Contact:
Priority: urgent    
Version: 7.0 (Kilo)CC: alan_bishop, arkady_kanevsky, cdevine, christopher_dearborn, ddomingo, dmacpher, dyocum, felipe.alfaro, gael_rehault, gfidente, jcoufal, jjoyce, joherr, John_walsh, jraju, jschluet, kasmith, kbader, kschinck, kurt_hey, mburns, morazi, nalmond, nlevine, nlevinki, ochalups, randy_perryman, rcernin, rhel-osp-director-maint, rsussman, skinjo, smerrow, sreichar, tvvcox, vumrao, wayne_allen, yrabl
Target Milestone: rcKeywords: FutureFeature, Reopened, Triaged
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-5.0.0-0.20160907212643.90c852e.2.el7ost Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 1418038 (view as bug list) Environment:
Last Closed: 2016-12-14 15:18:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1252546, 1330065    
Bug Blocks: 1319336, 1335593, 1356451, 1387433, 1413723    

Description Felipe Alfaro Solana 2015-11-19 16:38:32 UTC
Description of problem:
Puppet code used to deploy Ceph completely ignores any customisation done in puppet/hieradata/ceph.yaml, in the sense that all Ceph pools are created with a default size of 64 placement groups.

I am proposing a code change that not only fixes this, but also allows configuring each pool with a different set of parameters: pg_num, pgp_num and size. This makes sense, as usually the Ceph pool used by Glance will be typically smaller than the Ceph pool used to store Cinder volumes or Nova ephermeral disks.

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-0.8.6-71.el7ost.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy an Overcloud
2. All Ceph pools will have pg_num == pgp_num == 64 no matter what is specified in the file puppet/hieradata/ceph.yaml

Actual results:
All Ceph pools will have pg_num == pgp_num == 64 no matter what is specified in the file puppet/hieradata/ceph.yaml

Expected results:
Ceph pools should be configured independently.

Additional info:
I have created a Gerrit change to implement the fix: https://review.openstack.org/#/c/247669

Comment 2 Felipe Alfaro Solana 2015-11-19 16:41:39 UTC
Can we please target the fix for 7.2? This is a critical bug that is affecting Telefónica's ability to deploy production OpenStack clusters.

Comment 4 Giulio Fidente 2015-11-20 10:44:55 UTC

*** This bug has been marked as a duplicate of bug 1252546 ***

Comment 5 Jaromir Coufal 2015-11-20 12:35:04 UTC
(In reply to Felipe Alfaro Solana from comment #2)
> Can we please target the fix for 7.2? This is a critical bug that is
> affecting Telefónica's ability to deploy production OpenStack clusters.

Hi Felipe, thanks a lot for your proposed patch. Unfortunately we cannot add more patches to 7.2 since we did not pass just dev freeze, it is already post blocker freeze. This will have to go to 8.0.

Comment 6 Felipe Alfaro Solana 2015-11-20 15:27:17 UTC
Will there be a 7.3 release before 8.0? We aren't going to upgrade to 8.0 until February next year and can't wait that long for this patch to be officially supported.

Comment 7 Jaromir Coufal 2015-11-23 12:18:15 UTC
Felipe, there is no plan for 7.3 release before 8, since we are targeting ~2-month release cadence. Can you please send me an e-mail and provide better context? Based on that we will see what we can do to support you. Thanks, Jarda

Comment 9 Jaromir Coufal 2015-12-03 13:03:17 UTC
Guilio, is this bug duplicate and therefor fixed in another bugzilla from comment #4?

Comment 10 Giulio Fidente 2015-12-03 16:20:04 UTC
hi, no this is not a duplicate, it is a valid RFE and it depends on 1252546

this RFE is about making it possible to use *different* values of pg_num (etc) for different pools; the patch for 1252546 is now merged upstream; we need to review and merge the patch for this BZ instead

Comment 26 Giulio Fidente 2016-09-16 13:30:45 UTC
Sample usage to override the defaults of the 'volumes' pool:

parameter_defaults:
  ExtraConfig:
    CephPools:
      volumes:
        size: 5,
        pg_num: 128
        pgp_num: 128

Comment 29 Yogev Rabl 2016-10-30 15:14:03 UTC
verification failed on openstack-tripleo-heat-templates-5.0.0-0.8.0rc3.el7ost.noarch

I ran the following deployment command

openstack overcloud deploy \ 
--templates /usr/share/openstack-tripleo-heat-templates/ \
-e usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
-e usr/share/openstack-tripleo-heat-templates/environments/net-two-nic-with-vlans.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \ 
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-radosgw.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \
--control-scale 3 \
--compute-scale 2 \
--ceph-storage-scale 3 \
--control-flavor control \
--compute-flavor compute \
--ceph-storage-flavor ceph-storage \
--libvirt-type qemu \
--ntp-server 0.rhel.pool.ntp.org

Storage-environment.yaml content is:

resource_registry:
  OS::TripleO::Services::CephMon: ../puppet/services/ceph-mon.yaml
  OS::TripleO::Services::CephOSD: ../puppet/services/ceph-osd.yaml
  OS::TripleO::Services::CephClient: ../puppet/services/ceph-client.yaml

parameter_defaults:

  #### BACKEND SELECTION ####

  ## Whether to enable iscsi backend for Cinder.
  CinderEnableIscsiBackend: false
  ## Whether to enable rbd (Ceph) backend for Cinder.
  CinderEnableRbdBackend: true
  ## Cinder Backup backend can be either 'ceph' or 'swift'.
  CinderBackupBackend: ceph
  ## Whether to enable NFS backend for Cinder.
  # CinderEnableNfsBackend: false
  ## Whether to enable rbd (Ceph) backend for Nova ephemeral storage.
  NovaEnableRbdBackend: true
  ## Glance backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GlanceBackend: rbd
  ## Gnocchi backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GnocchiBackend: rbd
  ExtraConfig:
    ceph::profile::params::osds:
      '/dev/vda': {}
      '/dev/vdb': {}
      '/dev/vdc': {}
      '/dev/vdd': {}
      '/dev/vde': {}
    ceph::profile::params::osd_crush_update_on_start: false
    CephPools:
      volumes:
        size: 3
        pg_num: 128
        pgp_num: 128
      vms:
        size: 1
        pg_num: 128
        pgp_num: 128
      images:
        size: 5
        pg_num: 128
        pgp_num: 128

and the number of pg_num of the pools is 32

Comment 30 Giulio Fidente 2016-10-31 11:58:32 UTC
hi Yogev, I am *verry* sorry but my syntax in comment #26 was wrong.

CephPools is a Heat parameter, like GlanceBackend and goes at the same level of it, not folded within ExtraConfig (which instead, pushed hieradata on the nodes)

Is there any chance we could try this again and update the BZ?

Comment 31 Yogev Rabl 2016-10-31 12:20:03 UTC
Sure, I'll rerun it with the proper configuration.

Comment 32 Yogev Rabl 2016-10-31 12:20:12 UTC
Sure, I'll rerun it with the proper configuration.

Comment 33 Yogev Rabl 2016-10-31 13:24:18 UTC
Verified with the configuration

resource_registry:
  OS::TripleO::Services::CephMon: ../puppet/services/ceph-mon.yaml
  OS::TripleO::Services::CephOSD: ../puppet/services/ceph-osd.yaml
  OS::TripleO::Services::CephClient: ../puppet/services/ceph-client.yaml

parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: true
  CinderBackupBackend: ceph
  NovaEnableRbdBackend: true
  GlanceBackend: rbd
  GnocchiBackend: rbd
  CephPools:
    volumes:
      size: 3
      pg_num: 128
      pgp_num: 128
    vms:
      size: 1
      pg_num: 128
      pgp_num: 128
    images:
      size: 5
      pg_num: 128
      pgp_num: 128
  ExtraConfig:
    ceph::profile::params::osds:
      '/dev/vda': {}
      '/dev/vdb': {}
      '/dev/vdc': {}
      '/dev/vdd': {}
      '/dev/vde': {}

Comment 36 errata-xmlrpc 2016-12-14 15:18:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html

Comment 38 John Fulton 2017-01-11 16:10:49 UTC
*** Bug 1292981 has been marked as a duplicate of this bug. ***