Bug 1284988

Getting the same thing in my deployments using latest puddle of OSP8 beta.  Any potential fixes here?

Seems like the following are not defined anywhere:

GlanceFilePcmkManage
GlanceFilePcmkFstype
GlanceFilePcmkDevice
GlanceFilePcmkOptions

We (Fujitsu) found the same problem in our environment using the following template:

[stack@osp8bdr2 ~(UC)]$ cat ~/templates/storage-environment.yaml
parameters:
  CinderLVMLoopDeviceSize: 200000
  CinderEnableIscsiBackend: true
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: 'retry=1'
  CinderNfsServers: '10.111.128.9:/NFS/cinder'
  GlanceBackend: file
  GlanceFilePcmkDevice: '10.111.128.9:/NFS/glance'
  GlanceFilePcmkManage: true
  GlanceFilePcmkOptions: context=system_u:object_r:glance_var_lib_t:s0

I have used OSP 8 Beta 3.

It works without the "Glance*" lines.

Jirka, do you recall if we definitely wanted to support NFS-backed Glance? If we do, I guess we need to fix this.

Hi,

from my point of view, as a Partner (Fujitsu), it should be possible to use NFS Glance backend. 

Regards,

Ralf

This functionality used to work in RHEL-OSP6, where Pacemaker was used to monitor the NFS mount backed by an NFS backend.  We really need this in RHEL-OSP8.

Hello, sorry for the delayed response, i'm out of office at the moment.

There is a change in OSP8 storage-environment.yaml file when compared to OSP7, because the NFS support patch that eventually landed upstream and went into Liberty is subtly different than what went into Kilo.

Please try using `parameter_defaults:` instead of `parameters:` in the storage-environment.yaml file pasted in comment #2. This should fix the "parameter was not defined" errors.

This information should be in final OSP8 docs, there's bug 1301427 reported to track this change, so i'm going to close this bug and only keep the docs bug open. If the problem persists by any chance, please re-open this bug.

*** This bug has been marked as a duplicate of bug 1301427 ***

Hi Jiri,

I have tested with the following storage-environment.yaml file:

--- 
parameter_defaults:
  CinderLVMLoopDeviceSize: 200000
  CinderEnableIscsiBackend: true
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: 'retry=1'
  CinderNfsServers: '10.111.128.21:/mnt/nas/nv3/data/cinder'
  GlanceBackend: file
  GlanceFilePcmkManage: true
  GlanceFilePcmkFstype: nfs
  GlanceFilePcmkDevice: '10.111.128.21:/mnt/nas/nv3/data/glance'
  GlanceFilePcmkOptions: 'context=system_u:object_r:glance_var_lib_t:s0'
---

The deployment works without an error:

--- 
[stack@osp8dr1 ~(UC)]$ openstack overcloud deploy --stack osp8r1 --templates ~/templates/my-overcloud -e ~/templates/my-overcloud/environments/network-isolation.yaml -e ~/templates/network-environment.yaml -e ~/templates/my-overcloud/environments/puppet-pacemaker.yaml -e ~/templates/my-overcloud/rhel-registration/environment-rhel-registration.yaml -e ~/templates/my-overcloud/rhel-registration/rhel-registration-resource-registry.yaml -e ~/templates/storage-environment.yaml  -e  ~/templates/pre-deployment.yaml -e ~/templates/post-deployment.yaml --control-scale 3 --compute-scale 3 --control-flavor control --compute-flavor compute --ntp-server time.pdb.fsc.net --neutron-network-type vxlan --neutron-tunnel-types vxlan
Deploying templates in the directory /home/stack/templates/my-overcloud
/home/stack/.ssh/known_hosts updated.
Original contents retained as /home/stack/.ssh/known_hosts.old
PKI initialization in init-keystone is deprecated and will be removed.
Warning: Permanently added '192.168.126.20' (ECDSA) to the list of known hosts.
No handlers could be found for logger "oslo_config.cfg"
2016-02-12 09:26:23.657 15868 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Connection to 192.168.126.20 closed.
Overcloud Endpoint: http://192.168.126.20:5000/v2.0/
Overcloud Deployed
----

BUT, there seems to be something missing/broken in regard to Cinder NFS and Glance PCMK:

Cinder NFS
==========

1. The openstack-cinder-volume service for nfs failed to start

[stack@osp8dr1 ~(OC-admin)]$ cinder service-list | grep nfs
|  cinder-volume   |  osp8r1-controller.localdomain@tripleo_nfs  | nova | enabled |  down |             -              |        -        |

Reason: The config file for the [tripleo_nfs] backend "/etc/cinder/shares-nfs.conf" is empty:

[root@osp8r1-controller-0 cinder]# cat /etc/cinder/shares-nfs.conf
[root@osp8r1-controller-0 cinder]#

The value of "CinderNfsServers" didn't find the way into the file. If I add "10.111.128.21:/mnt/nas/nv3/data/cinder" to "/etc/cinder/shares-nfs.conf" the openstack-cinder-volume service for tripleo_nfs get's started.

--- 
[root@osp8r1-controller-0 cinder]# cat /etc/cinder/shares-nfs.conf
10.111.128.21:/mnt/nas/nv3/data/cinder

[stack@osp8dr1 templates(OC-admin)]$ cinder service-list | grep nfs
|  cinder-volume   |  osp8r1-controller.localdomain@tripleo_nfs  | nova | enabled |   up  | 2016-02-12T08:52:18.000000 |        -        |
--- 

Glance NFS
==========

1. Default Store for Glance is set to "file" --> OK

[root@osp8r1-controller-0 ~]# grep "^default_store" /etc/glance/glance-api.conf
default_store=file


2. No PCMK resource created for FS /var/lib/glance/images __> NOTOK

[root@osp8r1-controller-0 ~]# pcs status | grep -i glance
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]

[root@osp8r1-controller-0 ~]# pcs status | grep -i fs
[root@osp8r1-controller-0 ~]#

3. Images canh be created, but will be stored locally on the active Glance Controller node --> NOTOK

[stack@osp8dr1 ~(OC-admin)]$ glance image-create --name RHEL72 --disk-format qcow2 --copy-from http://192.168.126.10:8120/IMAGES/RHEL72.qcow2 --container-format bare --is-public True

[stack@osp8dr1 ~(OC-admin)]$ glance image-list
+--------------------------------------+--------+-------------+------------------+-----------+--------+
| ID                                   | Name   | Disk Format | Container Format | Size      | Status |
+--------------------------------------+--------+-------------+------------------+-----------+--------+
| 35849d1e-c4f4-4058-8860-0d174fcd0932 | RHEL72 | qcow2       | bare             | 474909696 | active |
+--------------------------------------+--------+-------------+------------------+-----------+--------+

[root@osp8r1-controller-0 images]# pwd
/var/lib/glance/images
[root@osp8r1-controller-0 images]# ls -al
total 463780
drwxr-x---. 2 glance glance        49 Feb 12 10:05 .
drwxr-xr-x. 3 glance nobody        19 Feb 12 09:19 ..
-rw-r-----. 1 glance glance 474909696 Feb 12 10:05 35849d1e-c4f4-4058-8860-0d174fcd0932
[root@osp8r1-controller-0 images]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       558G   32G  527G   6% /


Can you please have a look what is going wrong here? Thanks!

Regards,

Ralf

+1 To Ralf's comments.  This is still broken.

Glance NFS exhibits the same behavior in my environment.  There is no pacemaker fileshare resource created for the Glance mount on NFS and so images are stored on the currently active controller node at the time.  Not at all desired.

I suspect the generic NFS Cinder problem has to do with something in Puppet.  I am using the Cinder NetApp specific TripleO/Heat templates and my /etc/cinder/shares.conf is populated correctly.  I still think the generic NFS needs to be fixed though, as Ralf indicates and shows output of the bugged behavior above.

Looks like this is working for me in the OSP8-Beta7 (both mainline and Director) code base.

df -h output
==========
192.168.67.1:/glance         250G  258M  250G   1% /var/lib/glance/images

mount output
==========
192.168.67.1:/glance on /var/lib/glance/images type nfs4 (rw,relatime,context=system_u:object_r:glance_var_lib_t:s0,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.67.105,local_lock=none,addr=192.168.67.1)

pacemaker output
==========
 Clone: fs-varlibglanceimages-clone
  Resource: fs-varlibglanceimages (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=192.168.67.1:/glance directory=/var/lib/glance/images fstype=nfs options=context=system_u:object_r:glance_var_lib_t:s0 
   Operations: start interval=0s timeout=60 (fs-varlibglanceimages-start-interval-0s)
               stop interval=0s timeout=60 (fs-varlibglanceimages-stop-interval-0s)
               monitor interval=20 timeout=40 (fs-varlibglanceimages-monitor-interval-20)

Thanks for the info, i'm glad that it's working.

Hi,

works also for me with OSP 8 Beta 7. Thanks!

Regards,

Ralf

This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

If i understand the comments correctly, things worked as expected at the time of release and we just didn't close out the BZ, so i'll close it. If there's still an issue, please feel free to reopen.