Red Hat Bugzilla – Bug 1243306
If you enable ceph configuration for nova (compute nodes), this also puts ephemeral disks on ceph
Last modified: 2016-09-23 06:13:45 EDT
Hi, Looking through the openstack tripleo heat templates, we can see that there is a variable NovaEnableRbdBackend Which if set to true will configure a ceph client configuration on the compute nodes (good) and also set ephemeral storage to go onto ceph as well (if I'm reading this correctly) the puppet classs class nova::compute::rbd ( $libvirt_rbd_user, $libvirt_rbd_secret_uuid = false, $libvirt_rbd_secret_key = undef, $libvirt_images_rbd_pool = 'rbd', $libvirt_images_rbd_ceph_conf = '/etc/ceph/ceph.conf', $rbd_keyring = 'client.nova', $ephemeral_storage = true, ) defaults ephemeral_storage to true, and I don't believe we override it. Without NovaEnableRbdBackend instances cannot use cinder backed onto ceph. So to get the ideal situation of nova compute configured so it can use ceph, but ephemeral disks on local storage, is not done "out of the box". We should manually override ephemeral_storage to false Regards, Graeme
It might be a fix, Gilles can you have a look? https://review.openstack.org/202206 Thanks
Hi, I am more than happy to provide documentation on how to change this if you would like. Emilien, I had a look at the review you mentioned and it looks ok to me, except I noted on the review that I think you had some of the parameters around the wrong way. Let me know if you require documentation, or if we are going for a code fix. Regards, Graeme
Graeme, This seems like a case of adding the following to puppet/hieradata/compute.yaml: nova::compute::rbd::ephemeral_storage: false Can you confirm this? Or is it something different? - Dan
Hi Dan, I ran into the same problem Graeme encountered, and your solution of adding this parameter did the trick. I'm now able to have ephemeral storage in compute node and attach Ceph volume to the instance. mThank you very much! Cheerz, Nico
(In reply to Dan Macpherson from comment #7) > Graeme, > > This seems like a case of adding the following to > puppet/hieradata/compute.yaml: > > nova::compute::rbd::ephemeral_storage: false > > Can you confirm this? Or is it something different? > > - Dan Hi Dan, Yep that should do the trick as a work around. This is being addressed properly at https://review.openstack.org/#/c/202206/ Regards, Graeme
Hi, This needs to be pushed up in priority as now the workaround is broken, with this commit https://github.com/openstack/tripleo-heat-templates/commit/5965f7ab6cc60105e614d32ef2bb29205d81237d Which changes the if condition to not be conditional on NovaEnableRbdBackend, but rather on the nova::compute::rbd::ephemeral_storage hiera param itself. This now means that if you set nova::compute::rbd::ephemeral_storage to false as per the workaround, the if condition never fires and you never get the ceph client installed. So we are back to the original problem, either have ephemeral images on ceph and have a working ceph client, or don't and nova is never configured with ceph, meaning that cinder volumes on ceph break. The workaround NOW is to manually modify the puppet manifest to include ceph client as needed. Regards, Graeme
BTW, the comment associated with the parameter is not clear. ## Whether to enable rbd (Ceph) backend for Nova ephemeral storage. NovaEnableRbdBackend: true It should also tell that it disables all Ceph configuration. This could mislead people a lot.
The proper solution is to use the following patch to puppet/hieradata/compute.yaml: @@ -11,6 +11,8 @@ nova::compute::libvirt::vncserver_listen: '0.0.0.0' nova::compute::libvirt::migration_support: true nova::compute::rbd::libvirt_rbd_secret_uuid: "%{hiera('ceph::profile::params::fsid')}" +# Do not use Ceph for ephemeral disks. +nova::compute::rbd::ephemeral_storage: false nova::config::nova_config: cinder/catalog_info:
For the record, the patch mentioned before has been tested by me and I can confirm that ephemeral disks are using hypervisor local disks and that Cinder over Ceph also works fine.
This bug did not make the OSP 8.0 release. It is being deferred to OSP 10.
I think purpose of the params could have been misunderstood, from OSPd8 the expecations are that: NovaEnableRbdBackend: true will make Nova guests' ephemeral disks to go on RBD CinderEnableRbdBackend: true will add an RBD backend in Cinder (and configure the required settings in Nova so that guests can attach Cinder volumes stored on RBD) To use Ceph for Cinder, but not for the Nova ephemeral disks, the params should be set as: CinderEnableRbdBackend: true NovaEnableRbdBackend: false could you check if the above is working as intended?
Up, if Giulio comment resolves the issue we need to close this.
can someone confirm that Giulio's comment solve this issue or not?
(In reply to Giulio Fidente from comment #17) > I think purpose of the params could have been misunderstood, from OSPd8 the > expecations are that: > > NovaEnableRbdBackend: true > > will make Nova guests' ephemeral disks to go on RBD > > CinderEnableRbdBackend: true > > will add an RBD backend in Cinder (and configure the required settings in > Nova so that guests can attach Cinder volumes stored on RBD) > > > To use Ceph for Cinder, but not for the Nova ephemeral disks, the params > should be set as: > > CinderEnableRbdBackend: true > NovaEnableRbdBackend: false > > could you check if the above is working as intended? This is now correct, so this is issue is resolved (to me at least) Regards, Graeme
Thanks Graeme! I'm closing this then.