It appears that the pacemaker resource for the Glance NFS share is not being created. Selected bits from storage-environment.yaml: ## Glance backend can be either 'rbd' (Ceph), 'swift' or 'file'. GlanceBackend: 'file' #### GLANCE FILE BACKEND PACEMAKER SETTINGS (used for mounting NFS) #### ## Whether to make Glance 'file' backend a mount managed by Pacemaker GlanceFilePcmkManage: true ## File system type of the mount GlanceFilePcmkFstype: 'nfs' ## Pacemaker mount point, e.g. '192.168.122.1:/export/glance' for NFS ## (If using IPv6, use both double- and single-quotes, ## e.g. "'[fdd0::1]:/export/glance'") GlanceFilePcmkDevice: '172.31.248.1:/srv/nfs/osp_glance' ## Options for the mount managed by Pacemaker GlanceFilePcmkOptions: 'rw,sync,context=system_u:object_r:glance_var_lib_t:s0' After deployment, the Glance NFS share is not mounted on the controllers. "pcs status" shows that no resource has been created: Cluster name: tripleo_cluster Stack: corosync Current DC: overcloud-controller-1 (version 1.1.15-10.el7-e174ec8) - partition with quorum Last updated: Sun Oct 16 13:50:23 2016 Last change: Sun Oct 16 13:17:28 2016 by root via cibadmin on overcloud-controller-0 3 nodes and 19 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Full list of resources: ip-192.168.20.18 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-192.168.17.5 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 Clone Set: haproxy-clone [haproxy] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Master/Slave Set: galera-master [galera] Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] ip-192.168.18.18 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 ip-172.31.249.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-192.168.18.12 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 Clone Set: rabbitmq-clone [rabbitmq] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Master/Slave Set: redis-master [redis] Masters: [ overcloud-controller-2 ] Slaves: [ overcloud-controller-0 overcloud-controller-1 ] ip-172.31.248.13 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
So the reason this is not created is because in Newton/OSP10, the glance resources are actually defaulting to systemd values. So what we need to do here is probably the following: 1) See if we can workaround this in the meantime by mounting the nfs share via some other hooks 2) Check (and eventually add if they do not exist) the proper parameters to the base/systemd glance profile and add filesystem mounting step there. We also need to fix the storage-environment.yaml file, obviously.
Yes we'll probably have to replace the pacemaker NFS mounting with some workaround like a Puppet exec. Storage-environment.yaml will be a challenge w/r/t backwards compatibilty, we may have to provide some manual instructions how to convert to the new format. Also i wonder how this affects upgrading. The migration step to NG HA probably doesn't explicitly delete the NFS mounts, so they'd probably persist in the cluster even though they're not defined in Puppet. This might be fine as long as the services don't move from their original machines. (CCing Marios for visibility) In general doing this feels a bit hacky, i think it would be nicer if Glance had real NFS backend support like e.g. Cinder does, so that we didn't have to prepare the storage means on the side of config management. Paul/Erno, which services do we have to collocate the NFS mounts with? Are they only needed by glance-api, or also glance-registry?
o/ all thanks jistr for heads up... so I guess we are already hitting this and not noticing (or even not hitting it cos we don't have storage in the env, and in this particular case nfs backend) ... whatever the fixup is it will have to be factored into the upgrade somehow too (perhaps as part of the ng ha migration) ... will continue to monitor for now (+adding note for today's scrum thanks)
@jistr The back end storage in needed only on the API nodes.
Slightly aside, we need to make sure that we have a test for this in QE. Note that it isn't sufficient to simply check that Glance "works," as it will happily write the image to the local disk, and things will appear fine as long as you're hitting that node.
Patches posted upstream to t-h-t and puppet-tripleo. Tested with: parameter_defaults: GlanceBackend: file GlanceNfsEnabled: true GlanceNfsShare: 172.16.10.251:/export/glance and used this script to create the shares: https://paste.fedoraproject.org/456688/70649201/raw/ Glance image storage directory is a NFS mount, and pingtest succeeds fine, the images are getting created inside NFS. [root@overcloud-controller-0 ~]# mount | grep glance 172.16.10.251:/export/glance on /var/lib/glance/images type nfs4 (rw,relatime,context=system_u:object_r:glance_var_lib_t:s0,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.16.10.11,local_lock=none,addr=172.16.10.251) [root@overcloud-controller-0 ~]# grep glance /etc/fstab 172.16.10.251:/export/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 The NFS mount also gets restored after machine restart thanks to a fstab entry. This is probably the closest we can mimick the original pacemaker behavior. There will be changes in parameters though, as the semantics don't fully match the old way. (Even though we only ever tested and documented NFS, pacemaker was in theory capable of mounting other things, and the old param names went with the pacemaker semantics on that.)
Filed docs bug 1387723.
Merged to master, backports to stable/newton submitted. I replaced the external tracker links to point to the backport patches.
parameter_defaults: CinderEnableIscsiBackend: false CinderEnableRbdBackend: false CinderEnableNfsBackend: true NovaEnableRbdBackend: false CinderNfsMountOptions: 'rw,sync' CinderNfsServers: '10.0.0.254:/srv/nfs/cinder' GlanceBackend: 'file' GlanceNfsEnabled: true GlanceNfsShare: '10.0.0.254:/srv/nfs/glance' resulted in: ====================== overcloud-serviceapi-2.localdomain mounts:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.154,local_lock=none,addr=10.0.0.254 0 0 fstab:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 ====================== overcloud-serviceapi-0.localdomain mounts:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.149,local_lock=none,addr=10.0.0.254 0 0 fstab:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 ====================== overcloud-serviceapi-1.localdomain mounts:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.150,local_lock=none,addr=10.0.0.254 0 0 fstab:10.0.0.254:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 ====================== overcloud-controller-0.localdomain mounts:10.0.0.254:/srv/nfs/cinder /var/lib/cinder/mnt/93dfa45819ccd57c0cb9b93cd07c9128 nfs4 rw,sync,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.138,local_lock=none,addr=10.0.0.254 0 0 fstab: ====================== overcloud-controller-2.localdomain mounts: fstab: ====================== overcloud-controller-1.localdomain mounts: fstab: ====================== I'm using a custom role that runs the Glance services hence the Glance mounts show on different nodes than controllers.
Let's make sure that we document (and test) using IPv6 addresses for GlanceNfsShare.
(In reply to Ian Pilcher from comment #14) > Let's make sure that we document (and test) using IPv6 addresses for > GlanceNfsShare. Also tested on IPv6: parameter_defaults: CinderEnableIscsiBackend: false CinderEnableRbdBackend: false CinderEnableNfsBackend: true NovaEnableRbdBackend: false CinderNfsMountOptions: 'rw,sync' CinderNfsServers: "[fd00:fd00:fd00:3000::1]:/srv/nfs/cinder" GlanceBackend: 'file' GlanceNfsEnabled: true GlanceNfsShare: "'[fd00:fd00:fd00:3000::1]:/srv/nfs/glance'" ====================== overcloud-serviceapi-0.localdomain mounts:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=fd00:fd00:fd00:3000::1d,local_lock=none,addr=fd00:fd00:fd00:3000::1 0 0 fstab:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 ====================== overcloud-serviceapi-2.localdomain mounts:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=fd00:fd00:fd00:3000::1c,local_lock=none,addr=fd00:fd00:fd00:3000::1 0 0 fstab:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 ====================== overcloud-serviceapi-1.localdomain mounts:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 rw,context=system_u:object_r:glance_var_lib_t:s0,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=fd00:fd00:fd00:3000::16,local_lock=none,addr=fd00:fd00:fd00:3000::1 0 0 fstab:[fd00:fd00:fd00:3000::1]:/srv/nfs/glance /var/lib/glance/images nfs4 bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 ======================
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html