Bug 1284988 - rhel-osp-director: 8.0 - Attempt to deploy overcloud with glance over NFS: ERROR: The Parameter (GlanceFilePcmkManage) was not defined in template.
rhel-osp-director: 8.0 - Attempt to deploy overcloud with glance over NFS: ER...
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
urgent Severity urgent
: ---
: 10.0 (Newton)
Assigned To: Jiri Stransky
Shai Revivo
: Reopened, Triaged
Depends On:
Blocks: 1290377
  Show dependency treegraph
 
Reported: 2015-11-24 10:51 EST by Alexander Chuzhoy
Modified: 2016-09-08 12:42 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-09-08 12:42:01 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alexander Chuzhoy 2015-11-24 10:51:00 EST
rhel-osp-director: 8.0 - Attempt to deploy overcloud with glance over NFS: ERROR: The Parameter (GlanceFilePcmkManage) was not defined in template.   


Environment:
openstack-heat-api-cfn-5.0.0-1.el7ost.noarch
openstack-nova-compute-12.0.0-2.el7ost.noarch
openstack-neutron-ml2-7.0.0-4.el7ost.noarch
openstack-neutron-openvswitch-7.0.0-4.el7ost.noarch
openstack-nova-api-12.0.0-2.el7ost.noarch
openstack-glance-11.0.0-1.el7ost.noarch
openstack-heat-api-cloudwatch-5.0.0-1.el7ost.noarch
openstack-ceilometer-api-5.0.0-1.el7ost.noarch
openstack-ironic-inspector-2.2.2-1.el7ost.noarch
openstack-swift-object-2.5.0-1.el7ost.noarch
openstack-tripleo-common-0.0.2-3.el7ost.noarch
openstack-ceilometer-notification-5.0.0-1.el7ost.noarch
openstack-puppet-modules-7.0.1-2.el7ost.noarch
openstack-nova-conductor-12.0.0-2.el7ost.noarch
openstack-tripleo-image-elements-0.9.7-1.el7ost.noarch
openstack-keystone-8.0.0-1.el7ost.noarch
openstack-ceilometer-common-5.0.0-1.el7ost.noarch
openstack-ironic-conductor-4.2.0-2.1.el7ost.noarch
openstack-heat-templates-0-0.1.20151019.el7ost.noarch
openstack-tripleo-heat-templates-0.8.7-2.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-nova-common-12.0.0-2.el7ost.noarch
openstack-neutron-common-7.0.0-4.el7ost.noarch
openstack-neutron-7.0.0-4.el7ost.noarch
openstack-ceilometer-central-5.0.0-1.el7ost.noarch
openstack-swift-account-2.5.0-1.el7ost.noarch
openstack-swift-proxy-2.5.0-1.el7ost.noarch
openstack-ironic-common-4.2.0-2.1.el7ost.noarch
openstack-swift-2.5.0-1.el7ost.noarch
openstack-swift-container-2.5.0-1.el7ost.noarch
openstack-tempest-liberty-20151020.2.el7ost.noarch
openstack-heat-api-5.0.0-1.el7ost.noarch
openstack-ceilometer-polling-5.0.0-1.el7ost.noarch
openstack-selinux-0.6.45-1.el7ost.noarch
openstack-ceilometer-alarm-5.0.0-1.el7ost.noarch
openstack-nova-scheduler-12.0.0-2.el7ost.noarch
openstack-nova-cert-12.0.0-2.el7ost.noarch
python-openstackclient-1.7.1-1.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.2-1.el7ost.noarch
openstack-heat-engine-5.0.0-1.el7ost.noarch
openstack-ceilometer-collector-5.0.0-1.el7ost.noarch
openstack-tripleo-0.0.7-1.el7ost.noarch
openstack-ironic-api-4.2.0-2.1.el7ost.noarch
openstack-swift-plugin-swift3-1.7-6.20150601git69f94393.el7ost.noarch
instack-undercloud-2.1.3-1.el7ost.noarch
openstack-heat-common-5.0.0-1.el7ost.noarch


Steps to reproduce:
Attempt to deploy overcloud with glance over NFS:
Added the following under parameters section in yaml file:
  GlanceBackend: file
  GlanceFilePcmkDevice: [IP:/share]
  GlanceFilePcmkManage: true
  GlanceFilePcmkOptions: context=system_u:object_r:glance_var_lib_t:s0

Run the deployment:

Result:
The deployment fails right away with:  ERROR: The Parameter (GlanceFilePcmkManage) was not defined in template.   

Expected result:
The deployment should proceed.
Comment 1 Dave Cain 2016-01-12 22:12:31 EST
Getting the same thing in my deployments using latest puddle of OSP8 beta.  Any potential fixes here?

Seems like the following are not defined anywhere:

GlanceFilePcmkManage
GlanceFilePcmkFstype
GlanceFilePcmkDevice
GlanceFilePcmkOptions
Comment 2 Ralf Boernemeier 2016-01-14 11:53:18 EST
We (Fujitsu) found the same problem in our environment using the following template:

[stack@osp8bdr2 ~(UC)]$ cat ~/templates/storage-environment.yaml
parameters:
  CinderLVMLoopDeviceSize: 200000
  CinderEnableIscsiBackend: true
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: 'retry=1'
  CinderNfsServers: '10.111.128.9:/NFS/cinder'
  GlanceBackend: file
  GlanceFilePcmkDevice: '10.111.128.9:/NFS/glance'
  GlanceFilePcmkManage: true
  GlanceFilePcmkOptions: context=system_u:object_r:glance_var_lib_t:s0

I have used OSP 8 Beta 3.

It works without the "Glance*" lines.
Comment 8 Hugh Brock 2016-02-05 06:37:16 EST
Jirka, do you recall if we definitely wanted to support NFS-backed Glance? If we do, I guess we need to fix this.
Comment 9 Ralf Boernemeier 2016-02-05 06:45:43 EST
Hi,

from my point of view, as a Partner (Fujitsu), it should be possible to use NFS Glance backend. 

Regards,

Ralf
Comment 10 Dave Cain 2016-02-05 09:15:10 EST
This functionality used to work in RHEL-OSP6, where Pacemaker was used to monitor the NFS mount backed by an NFS backend.  We really need this in RHEL-OSP8.
Comment 11 Jiri Stransky 2016-02-11 11:50:10 EST
Hello, sorry for the delayed response, i'm out of office at the moment.

There is a change in OSP8 storage-environment.yaml file when compared to OSP7, because the NFS support patch that eventually landed upstream and went into Liberty is subtly different than what went into Kilo.

Please try using `parameter_defaults:` instead of `parameters:` in the storage-environment.yaml file pasted in comment #2. This should fix the "parameter was not defined" errors.

This information should be in final OSP8 docs, there's bug 1301427 reported to track this change, so i'm going to close this bug and only keep the docs bug open. If the problem persists by any chance, please re-open this bug.

*** This bug has been marked as a duplicate of bug 1301427 ***
Comment 12 Ralf Boernemeier 2016-02-12 04:09:50 EST
Hi Jiri,

I have tested with the following storage-environment.yaml file:

--- 
parameter_defaults:
  CinderLVMLoopDeviceSize: 200000
  CinderEnableIscsiBackend: true
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: 'retry=1'
  CinderNfsServers: '10.111.128.21:/mnt/nas/nv3/data/cinder'
  GlanceBackend: file
  GlanceFilePcmkManage: true
  GlanceFilePcmkFstype: nfs
  GlanceFilePcmkDevice: '10.111.128.21:/mnt/nas/nv3/data/glance'
  GlanceFilePcmkOptions: 'context=system_u:object_r:glance_var_lib_t:s0'
---

The deployment works without an error:

--- 
[stack@osp8dr1 ~(UC)]$ openstack overcloud deploy --stack osp8r1 --templates ~/templates/my-overcloud -e ~/templates/my-overcloud/environments/network-isolation.yaml -e ~/templates/network-environment.yaml -e ~/templates/my-overcloud/environments/puppet-pacemaker.yaml -e ~/templates/my-overcloud/rhel-registration/environment-rhel-registration.yaml -e ~/templates/my-overcloud/rhel-registration/rhel-registration-resource-registry.yaml -e ~/templates/storage-environment.yaml  -e  ~/templates/pre-deployment.yaml -e ~/templates/post-deployment.yaml --control-scale 3 --compute-scale 3 --control-flavor control --compute-flavor compute --ntp-server time.pdb.fsc.net --neutron-network-type vxlan --neutron-tunnel-types vxlan
Deploying templates in the directory /home/stack/templates/my-overcloud
/home/stack/.ssh/known_hosts updated.
Original contents retained as /home/stack/.ssh/known_hosts.old
PKI initialization in init-keystone is deprecated and will be removed.
Warning: Permanently added '192.168.126.20' (ECDSA) to the list of known hosts.
No handlers could be found for logger "oslo_config.cfg"
2016-02-12 09:26:23.657 15868 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Connection to 192.168.126.20 closed.
Overcloud Endpoint: http://192.168.126.20:5000/v2.0/
Overcloud Deployed
----

BUT, there seems to be something missing/broken in regard to Cinder NFS and Glance PCMK:

Cinder NFS
==========

1. The openstack-cinder-volume service for nfs failed to start

[stack@osp8dr1 ~(OC-admin)]$ cinder service-list | grep nfs
|  cinder-volume   |  osp8r1-controller.localdomain@tripleo_nfs  | nova | enabled |  down |             -              |        -        |

Reason: The config file for the [tripleo_nfs] backend "/etc/cinder/shares-nfs.conf" is empty:

[root@osp8r1-controller-0 cinder]# cat /etc/cinder/shares-nfs.conf
[root@osp8r1-controller-0 cinder]#

The value of "CinderNfsServers" didn't find the way into the file. If I add "10.111.128.21:/mnt/nas/nv3/data/cinder" to "/etc/cinder/shares-nfs.conf" the openstack-cinder-volume service for tripleo_nfs get's started.

--- 
[root@osp8r1-controller-0 cinder]# cat /etc/cinder/shares-nfs.conf
10.111.128.21:/mnt/nas/nv3/data/cinder

[stack@osp8dr1 templates(OC-admin)]$ cinder service-list | grep nfs
|  cinder-volume   |  osp8r1-controller.localdomain@tripleo_nfs  | nova | enabled |   up  | 2016-02-12T08:52:18.000000 |        -        |
--- 

Glance NFS
==========

1. Default Store for Glance is set to "file" --> OK

[root@osp8r1-controller-0 ~]# grep "^default_store" /etc/glance/glance-api.conf
default_store=file


2. No PCMK resource created for FS /var/lib/glance/images __> NOTOK

[root@osp8r1-controller-0 ~]# pcs status | grep -i glance
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]

[root@osp8r1-controller-0 ~]# pcs status | grep -i fs
[root@osp8r1-controller-0 ~]#

3. Images canh be created, but will be stored locally on the active Glance Controller node --> NOTOK

[stack@osp8dr1 ~(OC-admin)]$ glance image-create --name RHEL72 --disk-format qcow2 --copy-from http://192.168.126.10:8120/IMAGES/RHEL72.qcow2 --container-format bare --is-public True

[stack@osp8dr1 ~(OC-admin)]$ glance image-list
+--------------------------------------+--------+-------------+------------------+-----------+--------+
| ID                                   | Name   | Disk Format | Container Format | Size      | Status |
+--------------------------------------+--------+-------------+------------------+-----------+--------+
| 35849d1e-c4f4-4058-8860-0d174fcd0932 | RHEL72 | qcow2       | bare             | 474909696 | active |
+--------------------------------------+--------+-------------+------------------+-----------+--------+

[root@osp8r1-controller-0 images]# pwd
/var/lib/glance/images
[root@osp8r1-controller-0 images]# ls -al
total 463780
drwxr-x---. 2 glance glance        49 Feb 12 10:05 .
drwxr-xr-x. 3 glance nobody        19 Feb 12 09:19 ..
-rw-r-----. 1 glance glance 474909696 Feb 12 10:05 35849d1e-c4f4-4058-8860-0d174fcd0932
[root@osp8r1-controller-0 images]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       558G   32G  527G   6% /


Can you please have a look what is going wrong here? Thanks!

Regards,

Ralf
Comment 13 Dave Cain 2016-02-12 22:33:53 EST
+1 To Ralf's comments.  This is still broken.

Glance NFS exhibits the same behavior in my environment.  There is no pacemaker fileshare resource created for the Glance mount on NFS and so images are stored on the currently active controller node at the time.  Not at all desired.

I suspect the generic NFS Cinder problem has to do with something in Puppet.  I am using the Cinder NetApp specific TripleO/Heat templates and my /etc/cinder/shares.conf is populated correctly.  I still think the generic NFS needs to be fixed though, as Ralf indicates and shows output of the bugged behavior above.
Comment 14 Dave Cain 2016-02-29 13:32:56 EST
Looks like this is working for me in the OSP8-Beta7 (both mainline and Director) code base.

df -h output
==========
192.168.67.1:/glance         250G  258M  250G   1% /var/lib/glance/images

mount output
==========
192.168.67.1:/glance on /var/lib/glance/images type nfs4 (rw,relatime,context=system_u:object_r:glance_var_lib_t:s0,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.67.105,local_lock=none,addr=192.168.67.1)

pacemaker output
==========
 Clone: fs-varlibglanceimages-clone
  Resource: fs-varlibglanceimages (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=192.168.67.1:/glance directory=/var/lib/glance/images fstype=nfs options=context=system_u:object_r:glance_var_lib_t:s0 
   Operations: start interval=0s timeout=60 (fs-varlibglanceimages-start-interval-0s)
               stop interval=0s timeout=60 (fs-varlibglanceimages-stop-interval-0s)
               monitor interval=20 timeout=40 (fs-varlibglanceimages-monitor-interval-20)
Comment 15 Jiri Stransky 2016-03-01 04:17:05 EST
Thanks for the info, i'm glad that it's working.
Comment 16 Ralf Boernemeier 2016-03-08 08:01:18 EST
Hi,

works also for me with OSP 8 Beta 7. Thanks!

Regards,

Ralf
Comment 17 Mike Burns 2016-04-07 16:57:01 EDT
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 19 Jiri Stransky 2016-09-08 12:42:01 EDT
If i understand the comments correctly, things worked as expected at the time of release and we just didn't close out the BZ, so i'll close it. If there's still an issue, please feel free to reopen.

Note You need to log in before you can comment on or make changes to this bug.