Red Hat Bugzilla – Bug 1416356
Overcloud deployment using external NFS storage for Cinder fails
Last modified: 2017-05-17 15:43:11 EDT
Description of problem: Overcloud deployment using external NFS storage for Cinder fails: overcloud.AllNodesDeploySteps.ServiceApiDeployment_Step4: resource_type: OS::Heat::StructuredDeploymentGroup physical_resource_id: 5ec3a152-ca96-4b10-abee-5958c4698872 status: CREATE_FAILED status_reason: | CREATE aborted overcloud.AllNodesDeploySteps.ControllerDeployment_Step4.1: resource_type: OS::Heat::StructuredDeployment physical_resource_id: f2c85ab9-b077-4ebf-b3b1-6b3d71f23b0d status: CREATE_FAILED status_reason: | Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 deploy_stdout: | Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked. deploy_stderr: | ... Warning: Unknown variable: '::snmp_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:220:21 Warning: Unknown variable: '::snmp_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:230:24 Warning: Unknown variable: '::snmp_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:240:25 Warning: Unknown variable: '::snmp_trap_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:250:26 Warning: Unknown variable: '::snmp_trap_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:260:29 Warning: Unknown variable: '::snmp_trap_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:270:30 Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: Scope(Oslo::Messaging::Rabbit[cinder_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, join(): Requires array to work with at /etc/puppet/modules/cinder/manifests/backend/nfs.pp:82:16 at /etc/puppet/modules/tripleo/manifests/profile/base/cinder/volume/nfs.pp:54 on node overcloud-controller-1.localdomain (truncated, view all with --long) overcloud.AllNodesDeploySteps.ControllerDeployment_Step4.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 7c2f1c60-2a4e-43c8-92ef-df270adc471f status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 deploy_stdout: | Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked. deploy_stderr: | ... Warning: Unknown variable: '::snmp_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:220:21 Warning: Unknown variable: '::snmp_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:230:24 Warning: Unknown variable: '::snmp_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:240:25 Warning: Unknown variable: '::snmp_trap_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:250:26 Warning: Unknown variable: '::snmp_trap_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:260:29 Warning: Unknown variable: '::snmp_trap_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:270:30 Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: Scope(Oslo::Messaging::Rabbit[cinder_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, join(): Requires array to work with at /etc/puppet/modules/cinder/manifests/backend/nfs.pp:82:16 at /etc/puppet/modules/tripleo/manifests/profile/base/cinder/volume/nfs.pp:54 on node overcloud-controller-0.localdomain (truncated, view all with --long) overcloud.AllNodesDeploySteps.ControllerDeployment_Step4.2: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 7ef67b16-54da-49dc-8fda-4e6fc3442115 status: CREATE_FAILED status_reason: | Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 deploy_stdout: | Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked. deploy_stderr: | ... Warning: Unknown variable: '::snmp_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:220:21 Warning: Unknown variable: '::snmp_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:230:24 Warning: Unknown variable: '::snmp_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:240:25 Warning: Unknown variable: '::snmp_trap_service_enable'. at /etc/puppet/modules/snmp/manifests/params.pp:250:26 Warning: Unknown variable: '::snmp_trap_service_hasstatus'. at /etc/puppet/modules/snmp/manifests/params.pp:260:29 Warning: Unknown variable: '::snmp_trap_service_hasrestart'. at /etc/puppet/modules/snmp/manifests/params.pp:270:30 Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: Scope(Oslo::Messaging::Rabbit[cinder_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, join(): Requires array to work with at /etc/puppet/modules/cinder/manifests/backend/nfs.pp:82:16 at /etc/puppet/modules/tripleo/manifests/profile/base/cinder/volume/nfs.pp:54 on node overcloud-controller-2.localdomain (truncated, view all with --long) overcloud.AllNodesDeploySteps.ObjectStorageDeployment_Step4: resource_type: OS::Heat::StructuredDeploymentGroup physical_resource_id: fca4e64c-6baf-464d-bd0b-a575d0d4b612 status: CREATE_FAILED status_reason: | CREATE aborted overcloud.AllNodesDeploySteps.ComputeDeployment_Step4: resource_type: OS::Heat::StructuredDeploymentGroup physical_resource_id: fc19527d-2089-4c63-ba75-68dd89a0395e status: CREATE_FAILED status_reason: | CREATE aborted Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-6.0.0-0.20170120220933.36d493c.el7ost.noarch puppet-cinder-10.1.0-0.20170118172105.1860b1c.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud with custom roles and external NFS storage for Cinder: Actual results: Deployment fails Expected results: Deployment passes. Additional info: Environment files: http://paste.openstack.org/show/596415/
Hi all, This appears to be a simple type error: the Cinder module requires CinderNfsShares to be a list, such that it can be joined. The environment file provided a string; wrapping this string in [] should solve this immediate problem. I have also posted a review for tripleo-heat-templates to change the defaults for this parameter to an empty list (from an empty string.) Thanks, Elise
Note: I requested the blocker? flag be removed from this bug, as the fix is configuration, and any use of the NFS backend will require setting CinderNfsServers. Still, it would be best if we were to accept the default change patch prior to the next release. Cheers, Elise
(In reply to Elise Gafford from comment #5) > Note: I requested the blocker? flag be removed from this bug, as the fix is > configuration, and any use of the NFS backend will require setting > CinderNfsServers. Still, it would be best if we were to accept the default > change patch prior to the next release. > > Cheers, > Elise Hi Elise, The following CinderNfsServers values used to work in OSP10: CinderNfsServers: '10.0.0.254:/srv/nfs/cinder' CinderNfsServers: "[fd00:fd00:fd00:3000::1]:/srv/nfs/cinder" Could you please let me know what are the correct formats for OSP11 so I can test them? Thanks!
We also need this info to determine what we need to do in case of OSP10->OSP11 upgrade, if we need to change this parameters during upgrade or not.
Proposed upstream fix: https://review.openstack.org/448208 @Marius, can you test that fix if possible? Like this? On the undercloud: 1. cp -r /usr/share/openstack-tripleo-heat-templates . 2. Modify puppet/services/cinder-volume.yaml like this: https://review.openstack.org/#/c/448208/1/puppet/services/cinder-volume.yaml 3. Deploy with overcloud deploy ... --templates openstack-tripleo-heat-templates ... The options you have been using should continue to work then.
(In reply to Christian Schwede (cschwede) from comment #9) > Proposed upstream fix: https://review.openstack.org/448208 > > @Marius, can you test that fix if possible? Like this? > > On the undercloud: > > 1. cp -r /usr/share/openstack-tripleo-heat-templates . > > 2. Modify puppet/services/cinder-volume.yaml like this: > https://review.openstack.org/#/c/448208/1/puppet/services/cinder-volume.yaml > > 3. Deploy with > overcloud deploy ... --templates openstack-tripleo-heat-templates ... > > The options you have been using should continue to work then. Yes, I can confirm it works fine with the existing parameters after applying the patch. Thanks! Should I mark this bug as a blocker to make sure we keep track of the upstream patch landing in downstream in time for the release?
@Marius: yes, adding the blocker flag helps - this looks like a clear regression. Thanks for verifying the patch!
Backport merged upstream on stable/ocata.
Created attachment 1269401 [details] Example storage yaml file used. Verified on version: openstack-tripleo-heat-templates-6.0.0-3.el7ost.noarch Attaching yaml file for refute reference. Cinder create volumes works. Volume is created on NFS backend. [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-----------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-----------+------+-------------+----------+-------------+ | 531df6a7-2864-4818-b215-6b5e8057685a | available | nfsvolume | 1 | - | false | | +--------------------------------------+-----------+-----------+------+-------------+----------+-------------+ On nfs server # ll total 0 -rw-r--r--. 1 root root 0 פבר 16 14:23 ins_cinder_export -rw-rw----. 1 cinder cinder 1073741824 אפר 6 16:45 volume-531df6a7-2864-4818-b215-6b5e8057685a On overcloud_deploy.sh (overcloud deploy command) add refrance to your storage yaml file.
See comment 14 ^
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1245