Bug 1385368 - Glance NFS share not mounted (pacemaker resource not created)
Summary: Glance NFS share not mounted (pacemaker resource not created)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 10.0 (Newton)
Assignee: Jiri Stransky
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-16 13:50 UTC by Ian Pilcher
Modified: 2017-01-10 03:29 UTC (History)
14 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.0.0-1.3.el7ost puppet-tripleo-5.3.0-5.el7ost
Doc Type: Release Note
Doc Text:
To accommodate composable services, NFS mounts used as an Image Service (glance) back end are no longer managed by Pacemaker. As a result, the glance NFS back end parameter interface has changed: The new method is to use an environment file to enable the glance NFS backend. For example: ---- parameter_defaults: GlanceBackend: file GlanceNfsEnabled: true GlanceNfsShare: IP:/some/exported/path ---- Note: the GlanceNfsShare setting will vary depending on your deployment. In addition, mount options can be customized using the `GlanceNfsOptions` parameter. If the Glance NFS backend was previously used in Red Hat OpenStack Platform 9, the environment file contents must be updated to match the Red Hat OpenStack Platform 10 format.
Clone Of:
Environment:
Last Closed: 2016-12-14 16:20:23 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 392128 None None None 2016-11-01 13:18:59 UTC
OpenStack gerrit 392131 None None None 2016-11-01 13:20:43 UTC
Red Hat Bugzilla 1387723 None CLOSED [Docs][Director] Glance NFS backend instructions update 2019-03-19 18:46:21 UTC
Launchpad 1635606 None None None 2016-10-21 15:45:32 UTC

Internal Links: 1387723

Description Ian Pilcher 2016-10-16 13:50:38 UTC
It appears that the pacemaker resource for the Glance NFS share is not being created.

Selected bits from storage-environment.yaml:

  ## Glance backend can be either 'rbd' (Ceph), 'swift' or 'file'.                                                
  GlanceBackend: 'file'                                                                                           
  #### GLANCE FILE BACKEND PACEMAKER SETTINGS (used for mounting NFS) ####

  ## Whether to make Glance 'file' backend a mount managed by Pacemaker
  GlanceFilePcmkManage: true
  ## File system type of the mount
  GlanceFilePcmkFstype: 'nfs'
  ## Pacemaker mount point, e.g. '192.168.122.1:/export/glance' for NFS
  ##  (If using IPv6, use both double- and single-quotes,
  ##  e.g. "'[fdd0::1]:/export/glance'")
  GlanceFilePcmkDevice: '172.31.248.1:/srv/nfs/osp_glance'
  ## Options for the mount managed by Pacemaker
  GlanceFilePcmkOptions: 'rw,sync,context=system_u:object_r:glance_var_lib_t:s0'

After deployment, the Glance NFS share is not mounted on the controllers. "pcs status" shows that no resource has been created:

Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-1 (version 1.1.15-10.el7-e174ec8) - partition with quorum
Last updated: Sun Oct 16 13:50:23 2016          Last change: Sun Oct 16 13:17:28 2016 by root via cibadmin on overcloud-controller-0

3 nodes and 19 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-192.168.20.18       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-192.168.17.5        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.168.18.18       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 ip-172.31.249.10       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-192.168.18.12       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-2 ]
     Slaves: [ overcloud-controller-0 overcloud-controller-1 ]
 ip-172.31.248.13       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Comment 1 Michele Baldessari 2016-10-17 06:59:41 UTC
So the reason this is not created is because in Newton/OSP10, the glance resources are actually defaulting to systemd values. So what we need to do here is probably the following:
1) See if we can workaround this in the meantime by mounting the nfs share via some other hooks
2) Check (and eventually add if they do not exist) the proper parameters to the base/systemd glance profile and add filesystem mounting step there. We also need to fix the storage-environment.yaml file, obviously.

Comment 3 Jiri Stransky 2016-10-18 16:50:53 UTC
Yes we'll probably have to replace the pacemaker NFS mounting with some workaround like a Puppet exec. Storage-environment.yaml will be a challenge w/r/t backwards compatibilty, we may have to provide some manual instructions how to convert to the new format.

Also i wonder how this affects upgrading. The migration step to NG HA probably doesn't explicitly delete the NFS mounts, so they'd probably persist in the cluster even though they're not defined in Puppet. This might be fine as long as the services don't move from their original machines. (CCing Marios for visibility)

In general doing this feels a bit hacky, i think it would be nicer if Glance had real NFS backend support like e.g. Cinder does, so that we didn't have to prepare the storage means on the side of config management.

Paul/Erno, which services do we have to collocate the NFS mounts with? Are they only needed by glance-api, or also glance-registry?

Comment 6 Marios Andreou 2016-10-19 08:17:58 UTC
o/ all thanks jistr for heads up... so I guess we are already hitting this and not noticing (or even not hitting it cos we don't have storage in the env, and in this particular case nfs backend) ... whatever the fixup is it will have to be factored into the upgrade somehow too (perhaps as part of the ng ha migration) ... will continue to monitor for now (+adding note for today's scrum thanks)

Comment 7 Erno Kuvaja 2016-10-19 14:05:43 UTC
@jistr The back end storage in needed only on the API nodes.

Comment 8 Ian Pilcher 2016-10-20 14:16:32 UTC
Slightly aside, we need to make sure that we have a test for this in QE.  Note that it isn't sufficient to simply check that Glance "works," as it will happily write the image to the local disk, and things will appear fine as long as you're hitting that node.

Comment 9 Jiri Stransky 2016-10-21 15:56:06 UTC
Patches posted upstream to t-h-t and puppet-tripleo. Tested with:

parameter_defaults:
  GlanceBackend: file
  GlanceNfsEnabled: true
  GlanceNfsShare: 172.16.10.251:/export/glance

and used this script to create the shares: https://paste.fedoraproject.org/456688/70649201/raw/

Glance image storage directory is a NFS mount, and pingtest succeeds fine, the images are getting created inside NFS.

[root@overcloud-controller-0 ~]# mount | grep glance
172.16.10.251:/export/glance on /var/lib/glance/images type nfs4 (rw,relatime,context=system_u:object_r:glance_var_lib_t:s0,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.16.10.11,local_lock=none,addr=172.16.10.251)

[root@overcloud-controller-0 ~]# grep glance /etc/fstab
172.16.10.251:/export/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0

The NFS mount also gets restored after machine restart thanks to a fstab entry. This is probably the closest we can mimick the original pacemaker behavior. There will be changes in parameters though, as the semantics don't fully match the old way. (Even though we only ever tested and documented NFS, pacemaker was in theory capable of mounting other things, and the old param names went with the pacemaker semantics on that.)

Comment 10 Jiri Stransky 2016-10-21 16:27:31 UTC
Filed docs bug 1387723.

Comment 11 Jiri Stransky 2016-11-01 13:20:43 UTC
Merged to master, backports to stable/newton submitted. I replaced the external tracker links to point to the backport patches.

Comment 13 Marius Cornea 2016-11-10 11:40:54 UTC
parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderEnableNfsBackend: true
  NovaEnableRbdBackend: false

  CinderNfsMountOptions: 'rw,sync'
  CinderNfsServers: '10.0.0.254:/srv/nfs/cinder'

  GlanceBackend: 'file'
  GlanceNfsEnabled: true
  GlanceNfsShare: '10.0.0.254:/srv/nfs/glance'

resulted in:

======================
overcloud-serviceapi-2.localdomain
mounts:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.154,local_lock=none,addr=10.0.0.254 0 0
fstab:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0

======================
overcloud-serviceapi-0.localdomain
mounts:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.149,local_lock=none,addr=10.0.0.254 0 0
fstab:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0

======================
overcloud-serviceapi-1.localdomain
mounts:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.150,local_lock=none,addr=10.0.0.254 0 0
fstab:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0

======================
overcloud-controller-0.localdomain
mounts:10.0.0.254:/srv/nfs/cinder /var/lib/cinder/mnt/93dfa45819ccd57c0cb9b93cd07c9128 nfs4 rw,sync,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.138,local_lock=none,addr=10.0.0.254 0 0
fstab:

======================
overcloud-controller-2.localdomain
mounts:
fstab:
======================
overcloud-controller-1.localdomain
mounts:
fstab:
======================

I'm using a custom role that runs the Glance services hence the Glance mounts show on different nodes than controllers.

Comment 14 Ian Pilcher 2016-11-30 13:07:08 UTC
Let's make sure that we document (and test) using IPv6 addresses for GlanceNfsShare.

Comment 15 Marius Cornea 2016-11-30 15:35:55 UTC
(In reply to Ian Pilcher from comment #14)
> Let's make sure that we document (and test) using IPv6 addresses for
> GlanceNfsShare.

Also tested on IPv6:

parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderEnableNfsBackend: true
  NovaEnableRbdBackend: false

  CinderNfsMountOptions: 'rw,sync'
  CinderNfsServers: "[fd00:fd00:fd00:3000::1]:/srv/nfs/cinder"

  GlanceBackend: 'file'
  GlanceNfsEnabled: true
  GlanceNfsShare: "'[fd00:fd00:fd00:3000::1]:/srv/nfs/glance'"

======================
overcloud-serviceapi-0.localdomain
mounts:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=fd00:fd00:fd00:3000::1d,local_lock=none,addr=fd00:fd00:fd00:3000::1 0 0
fstab:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0
======================
overcloud-serviceapi-2.localdomain
mounts:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=fd00:fd00:fd00:3000::1c,local_lock=none,addr=fd00:fd00:fd00:3000::1 0 0
fstab:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0
======================
overcloud-serviceapi-1.localdomain
mounts:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=fd00:fd00:fd00:3000::16,local_lock=none,addr=fd00:fd00:fd00:3000::1 0 0
fstab:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0
======================

Comment 17 errata-xmlrpc 2016-12-14 16:20:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.