Bug 1416356 - Overcloud deployment using external NFS storage for Cinder fails
Summary: Overcloud deployment using external NFS storage for Cinder fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 11.0 (Ocata)
Assignee: Angus Thomas
QA Contact: lkuchlan
URL:
Whiteboard:
Depends On:
Blocks: 1394019
TreeView+ depends on / blocked
 
Reported: 2017-01-25 10:56 UTC by Marius Cornea
Modified: 2017-05-17 19:43 UTC (History)
17 users (show)

Fixed In Version: openstack-tripleo-heat-templates-6.0.0-0.10.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-17 19:43:11 UTC


Attachments (Terms of Use)
Example storage yaml file used. (2.40 KB, text/plain)
2017-04-06 13:54 UTC, Tzach Shefi
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1245 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC
OpenStack gerrit 450085 None None None 2017-03-28 11:02:09 UTC

Description Marius Cornea 2017-01-25 10:56:10 UTC
Description of problem:
Overcloud deployment using external NFS storage for Cinder fails:

overcloud.AllNodesDeploySteps.ServiceApiDeployment_Step4:
  resource_type: OS::Heat::StructuredDeploymentGroup
  physical_resource_id: 5ec3a152-ca96-4b10-abee-5958c4698872
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
overcloud.AllNodesDeploySteps.ControllerDeployment_Step4.1:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: f2c85ab9-b077-4ebf-b3b1-6b3d71f23b0d
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
  deploy_stderr: |
    ...
    Warning: Unknown variable: '::snmp_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:220:21
    Warning: Unknown variable: '::snmp_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:230:24
    Warning: Unknown variable: '::snmp_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:240:25
    Warning: Unknown variable: '::snmp_trap_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:250:26
    Warning: Unknown variable: '::snmp_trap_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:260:29
    Warning: Unknown variable: '::snmp_trap_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:270:30
    Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: Scope(Oslo::Messaging::Rabbit[cinder_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, join(): Requires array to work with at /etc/puppet/modules/cinder/manifests/backend/nfs.pp:82:16  at /etc/puppet/modules/tripleo/manifests/profile/base/cinder/volume/nfs.pp:54 on node overcloud-controller-1.localdomain
    (truncated, view all with --long)
overcloud.AllNodesDeploySteps.ControllerDeployment_Step4.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 7c2f1c60-2a4e-43c8-92ef-df270adc471f
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
  deploy_stderr: |
    ...
    Warning: Unknown variable: '::snmp_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:220:21
    Warning: Unknown variable: '::snmp_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:230:24
    Warning: Unknown variable: '::snmp_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:240:25
    Warning: Unknown variable: '::snmp_trap_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:250:26
    Warning: Unknown variable: '::snmp_trap_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:260:29
    Warning: Unknown variable: '::snmp_trap_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:270:30
    Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: Scope(Oslo::Messaging::Rabbit[cinder_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, join(): Requires array to work with at /etc/puppet/modules/cinder/manifests/backend/nfs.pp:82:16  at /etc/puppet/modules/tripleo/manifests/profile/base/cinder/volume/nfs.pp:54 on node overcloud-controller-0.localdomain
    (truncated, view all with --long)
overcloud.AllNodesDeploySteps.ControllerDeployment_Step4.2:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 7ef67b16-54da-49dc-8fda-4e6fc3442115
  status: CREATE_FAILED
  status_reason: |
    Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
  deploy_stderr: |
    ...
    Warning: Unknown variable: '::snmp_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:220:21
    Warning: Unknown variable: '::snmp_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:230:24
    Warning: Unknown variable: '::snmp_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:240:25
    Warning: Unknown variable: '::snmp_trap_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:250:26
    Warning: Unknown variable: '::snmp_trap_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:260:29
    Warning: Unknown variable: '::snmp_trap_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:270:30
    Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: Scope(Oslo::Messaging::Rabbit[cinder_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, join(): Requires array to work with at /etc/puppet/modules/cinder/manifests/backend/nfs.pp:82:16  at /etc/puppet/modules/tripleo/manifests/profile/base/cinder/volume/nfs.pp:54 on node overcloud-controller-2.localdomain
    (truncated, view all with --long)
overcloud.AllNodesDeploySteps.ObjectStorageDeployment_Step4:
  resource_type: OS::Heat::StructuredDeploymentGroup
  physical_resource_id: fca4e64c-6baf-464d-bd0b-a575d0d4b612
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
overcloud.AllNodesDeploySteps.ComputeDeployment_Step4:
  resource_type: OS::Heat::StructuredDeploymentGroup
  physical_resource_id: fc19527d-2089-4c63-ba75-68dd89a0395e
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-6.0.0-0.20170120220933.36d493c.el7ost.noarch
puppet-cinder-10.1.0-0.20170118172105.1860b1c.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with custom roles and external NFS storage for Cinder:


Actual results:
Deployment fails

Expected results:
Deployment passes.

Additional info:
Environment files:
http://paste.openstack.org/show/596415/

Comment 4 Elise Gafford 2017-03-06 16:52:13 UTC
Hi all,

This appears to be a simple type error: the Cinder module requires CinderNfsShares to be a list, such that it can be joined. The environment file provided a string; wrapping this string in [] should solve this immediate problem. I have also posted a review for tripleo-heat-templates to change the defaults for this parameter to an empty list (from an empty string.)

Thanks,
Elise

Comment 5 Elise Gafford 2017-03-06 17:09:30 UTC
Note: I requested the blocker? flag be removed from this bug, as the fix is configuration, and any use of the NFS backend will require setting CinderNfsServers. Still, it would be best if we were to accept the default change patch prior to the next release.

Cheers,
Elise

Comment 6 Marius Cornea 2017-03-06 17:18:00 UTC
(In reply to Elise Gafford from comment #5)
> Note: I requested the blocker? flag be removed from this bug, as the fix is
> configuration, and any use of the NFS backend will require setting
> CinderNfsServers. Still, it would be best if we were to accept the default
> change patch prior to the next release.
> 
> Cheers,
> Elise

Hi Elise,

The following CinderNfsServers values used to work in OSP10:

CinderNfsServers: '10.0.0.254:/srv/nfs/cinder'
CinderNfsServers: "[fd00:fd00:fd00:3000::1]:/srv/nfs/cinder"

Could you please let me know what are the correct formats for OSP11 so I can test them?

Thanks!

Comment 7 Marius Cornea 2017-03-14 06:54:31 UTC
We also need this info to determine what we need to do in case of OSP10->OSP11 upgrade, if we need to change this parameters during upgrade or not.

Comment 9 Christian Schwede (cschwede) 2017-03-21 17:37:27 UTC
Proposed upstream fix: https://review.openstack.org/448208

@Marius, can you test that fix if possible? Like this?

On the undercloud:

1. cp -r /usr/share/openstack-tripleo-heat-templates .

2. Modify puppet/services/cinder-volume.yaml like this: https://review.openstack.org/#/c/448208/1/puppet/services/cinder-volume.yaml

3. Deploy with
overcloud deploy ... --templates openstack-tripleo-heat-templates ...

The options you have been using should continue to work then.

Comment 10 Marius Cornea 2017-03-22 11:36:09 UTC
(In reply to Christian Schwede (cschwede) from comment #9)
> Proposed upstream fix: https://review.openstack.org/448208
> 
> @Marius, can you test that fix if possible? Like this?
> 
> On the undercloud:
> 
> 1. cp -r /usr/share/openstack-tripleo-heat-templates .
> 
> 2. Modify puppet/services/cinder-volume.yaml like this:
> https://review.openstack.org/#/c/448208/1/puppet/services/cinder-volume.yaml
> 
> 3. Deploy with
> overcloud deploy ... --templates openstack-tripleo-heat-templates ...
> 
> The options you have been using should continue to work then.

Yes, I can confirm it works fine with the existing parameters after applying the patch. Thanks! 

Should I mark this bug as a blocker to make sure we keep track of the upstream patch landing in downstream in time for the release?

Comment 11 Christian Schwede (cschwede) 2017-03-22 11:52:23 UTC
@Marius: yes, adding the blocker flag helps - this looks like a clear regression. 
Thanks for verifying the patch!

Comment 12 Christian Schwede (cschwede) 2017-03-28 11:02:10 UTC
Backport merged upstream on stable/ocata.

Comment 14 Tzach Shefi 2017-04-06 13:54:53 UTC
Created attachment 1269401 [details]
Example storage yaml file used.

Verified on version:
openstack-tripleo-heat-templates-6.0.0-3.el7ost.noarch

Attaching yaml file for refute reference. 
Cinder create volumes works. 
Volume is created on NFS backend. 

[stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+-----------+------+-------------+----------+-------------+
| ID                                   | Status    | Name      | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------+------+-------------+----------+-------------+
| 531df6a7-2864-4818-b215-6b5e8057685a | available | nfsvolume | 1    | -           | false    |             |
+--------------------------------------+-----------+-----------+------+-------------+----------+-------------+

On nfs server
# ll 
total 0
-rw-r--r--. 1 root   root            0 פבר 16 14:23 ins_cinder_export
-rw-rw----. 1 cinder cinder 1073741824 אפר  6 16:45 volume-531df6a7-2864-4818-b215-6b5e8057685a


On overcloud_deploy.sh (overcloud deploy command) add refrance to your storage yaml file.

Comment 15 Tzach Shefi 2017-04-06 13:55:33 UTC
See comment 14 ^

Comment 17 errata-xmlrpc 2017-05-17 19:43:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.