Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1574787 - manila configuration fails when using manila-backend-vnx.yaml template
Summary: manila configuration fails when using manila-backend-vnx.yaml template
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: z2
: 13.0 (Queens)
Assignee: Goutham Pacha Ravi
QA Contact: Dustin Schoenbrun
URL:
Whiteboard:
Depends On:
Blocks: 1394885 1516324 1588258
TreeView+ depends on / blocked
 
Reported: 2018-05-04 03:40 UTC by Cody Swanson
Modified: 2019-12-16 19:34 UTC (History)
17 users (show)

Fixed In Version: openstack-tripleo-heat-templates-8.0.3-2.el7ost puppet-tripleo-8.3.4-2.el7ost
Doc Type: Bug Fix
Doc Text:
Manila configuration manifests for Dell-EMC storage systems (VNX, Unity, and VMAX) had incorrect configuration options. As a result, the overcloud deployment of manila-share service with Dell Storage systems failed. The Manila configuration manifests for Dell-EMC storage systems (VNX, Unity, and VMAX) have now been fixed. The overcloud deployment of manila-share service with Dell storage systems completes successfully.
Clone Of:
: 1588258 (view as bug list)
Environment:
Last Closed: 2018-08-29 16:35:54 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1771179 0 None None None 2018-05-17 17:26:26 UTC
Launchpad 1771656 0 None None None 2018-05-17 13:37:42 UTC
OpenStack gerrit 568348 0 None MERGED Fix for the manila backend configuration errors 2020-04-16 12:53:44 UTC
OpenStack gerrit 568945 0 None MERGED Remove share_backend_name from Dell-EMC manila backends 2020-04-16 12:53:44 UTC
OpenStack gerrit 570958 0 None MERGED Fix for the manila backend configuration errors 2020-04-16 12:53:44 UTC
Red Hat Product Errata RHBA-2018:2574 0 None None None 2018-08-29 16:36:50 UTC

Description Cody Swanson 2018-05-04 03:40:29 UTC
Description of problem:

manila configuration fails when using manila-backend-vnx.yaml template. does not create manila user, database in mariadb. also the template is missing manila::backend::dellemc_vnx::share_backend_name parameter and sets manila::backend::dellemc_vnx::emc_share_backend incorrectly causing additional errors.

May 01 13:53:47 wcmsc5-l-rh-ocld-0 puppet-user[396288]: (/Stage[main]/Manila::Db::Sync/Exec[manila-manage db_sync]) Failed to call refresh: manila-manage db sync returned 1 instead of one of [0]
May 01 13:53:47 wcmsc5-l-rh-ocld-0 puppet-user[396288]: (/Stage[main]/Manila::Db::Sync/Exec[manila-manage db_sync]) manila-manage db sync returned 1 instead of one of [0]


Version-Release number of selected component (if applicable):

openstack-tripleo-heat-templates-7.0.3-22.el7ost.noarch


How reproducible:

Every time it is deployed

Steps to Reproduce:
1. Deploy overcloud using supplied templates. 
2.
3.

Actual results:
Manila fails to deploy as described above.

Expected results:
Manila deploys properly. 

Additional info:

I've attached the customers manila deployment configuration. I've just reproduced this in my lab environment and Mark Jones was also able to reproduce this issue by attempting to deploy manila in his lab environment.

Comment 2 Rajini Karthik 2018-05-10 21:28:13 UTC
Can you attach the manila.conf file from the deployment pl

Comment 3 Rajini Karthik 2018-05-10 21:30:40 UTC
And possibly the environment file used for deployment pl

Comment 4 Rajini Karthik 2018-05-11 19:06:00 UTC
 I don't see the attachment. May be it is marked private

Comment 5 Cody Swanson 2018-05-11 19:08:40 UTC
(In reply to Rajini Karthik from comment #4)
>  I don't see the attachment. May be it is marked private

Hi Rajini,

The attachment is marked private due to containing customer data. I'm working on sanitizing the files you've requested, I will attach them this afternoon.

Comment 6 Alan Bishop 2018-05-12 19:45:11 UTC
I confirmed the two issues described in the first paragraph of the description, and discussed them with Rajini. She will fix malformed emc_shared_backend hiera value, and I will propose a change to puppet-tripleo that will eliminate the need to set the shared_backend_name. Both of those issues are easy to work around, which I see you have done using a custom template. The hiera values can also be set using ExtraConfig, like this:

parameter_defaults:
  ExtraConfig:
    manila::backend::dellemc_vnx::emc_share_backend: vnx
    manila::backend::dellemc_vnx::share_backend_name: tripleo_manila_vnx                                                                             

But that doesn't explain the deployment problem, which I was not able to reproduce. I would start by focusing on the "manila-manage db_sync" error. But here are a couple of other points to consider.

openstack-tripleo-heat-templates-7.0.3-22.el7ost.noarch shipped in 12z1, and a newer version is available in 12z2. I don't know if that's a factor, but it would be good to test using the latest version(s).

I also notice the manila-environment.yaml file in the BZ attachment is setting ManilaVNXNasServer with an IPv6 address, which is not supported in OSP-12 (but is planned for OSP-13). You might try deploying with an IPv4 address, even a bogus one. With a bogus address, the Manila VNX backend service will fail to come up, but the deployment should finish. Here is an example from my own test:

(overcloud) [stack@rhos-undercloud ~]$ manila service-list
+----+------------------+------------------------------+------+---------+-------+----------------------------+
| Id | Binary           | Host                         | Zone | Status  | State | Updated_at                 |
+----+------------------+------------------------------+------+---------+-------+----------------------------+
| 1  | manila-scheduler | hostgroup                    | nova | enabled | up    | 2018-05-12T19:42:07.000000 |
| 2  | manila-share     | hostgroup@tripleo_manila_vnx | nova | enabled | down  | None                       |
+----+------------------+------------------------------+------+---------+-------+----------------------------+

Comment 8 Rajini Karthik 2018-05-14 20:55:32 UTC
Fix proposed for 
https://review.openstack.org/#/c/568348/
manila::backend::dellemc_vnx::emc_share_backend

Comment 9 Goutham Pacha Ravi 2018-05-17 17:31:07 UTC
Targeting for RH OSP 13 because there's a workaround for OSP 12 as Alan mentioned prior.

Comment 11 Rajini Karthik 2018-05-29 16:10:49 UTC
Backport to queens -https://review.openstack.org/#/c/570958/

Comment 12 Rajini Karthik 2018-05-29 16:11:07 UTC
https://review.openstack.org/#/c/570958/

Comment 27 Sean Merrow 2018-08-01 15:00:59 UTC
Hi Rajini, 

Can Dell EMC perform QA on this?

Thanks,
Sean

Comment 28 Rajini Karthik 2018-08-01 15:39:18 UTC
If there is an rpm or something, I can ask the VNX storage team to QA it.

Comment 29 Sean Merrow 2018-08-01 16:52:51 UTC
Can we get the rpm to Rajini for testing?

Comment 30 Rajini Karthik 2018-08-01 16:53:20 UTC
We can verify this bug fix,if you can provide tripleo-heat-template and puppet-* rpms.

Comment 31 Sean Merrow 2018-08-01 17:17:00 UTC
Dustin, can you help get Rajini the packages she would need to test?

Thanks,
Sean

Comment 32 Dustin Schoenbrun 2018-08-01 19:40:55 UTC
Done! Sent them over to Rajini at her Dell email address. Let me know if you need anything else!

Comment 33 Rajini Karthik 2018-08-03 17:27:29 UTC
Hi Rajini,

I tried to deploy Manila VNX today, but failed, the issue is same as before:
--------------------------------------------------
Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item manila::backend::dellemc_vnx::share_backend_name in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/manila/share.pp:215:41 on node osp12ctl0.localdomain
+ rc=1
+ set -e
+ '[' 1 -ne 2 -a 1 -ne 0 ']'
+ exit 1
--------------------------------------------------

My test steps as:
1.	Upgrade openstack-tripleo-heat-templates-8.0.3-2.el7ost.noarch.rpm on undercloud
2.	Upgrade puppet-tripleo-8.3.4-2.el7ost.noarch.rpm on undercloud
3.	Deploy overcloud

I think Red Hat should also provide new overcloud image which with new puppet-* rpms installed, the puppet-tripleo version on overcloud as:
--------------------------------------------------
[root@osp12ctl0 ~]# rpm -qa|grep tripleo
puppet-tripleo-8.3.2-7.el7ost.noarch
[root@osp12ctl0 ~]# rpm -qa|grep puppet|grep manila
puppet-manila-12.4.0-0.20180329035214.6c18418.el7ost.noarch
--------------------------------------------------


Thanks
Yong

Comment 34 Rajini Karthik 2018-08-03 17:34:05 UTC
Hi Dustin
This didn’t work. In the past we have tested upstream puppet modules using upload-puppet-script.
Is there a better way to test these rpms?

Using the upload-puppet-modules script we will be able to update the puppet modules when executing the overcloud deployment.
# From the undercloud 
mkdir puppet-modules
cd puppet-modules
git clone https://git.openstack.org/openstack/puppet-tripleo tripleo
# Edit as needed under the tripleo folder
cd
git clone https://git.openstack.org/openstack/tripleo-common
export PATH="$PATH:tripleo-common/scripts"
upload-puppet-modules --directory puppet-modules/

Thanks
Rajini

Comment 35 Alan Bishop 2018-08-03 17:48:05 UTC
(In reply to Rajini Karthik from comment #34)
> Hi Dustin
> This didn’t work. In the past we have tested upstream puppet modules using
> upload-puppet-script.
> Is there a better way to test these rpms?
> 
> Using the upload-puppet-modules script we will be able to update the puppet
> modules when executing the overcloud deployment.
> # From the undercloud 
> mkdir puppet-modules
> cd puppet-modules
> git clone https://git.openstack.org/openstack/puppet-tripleo tripleo

Instead of cloning the upstream repo, grab a copy of the files from the updated RPM on the undercloud. Try this after you created the puppet-modules directory:

% cp -r /usr/share/openstack-puppet/modules/tripleo ~/puppet-modules

You should now have a ~/puppet-modules/tripleo directory that contains the fix. Then run the upload-puppet-modules script, as you described.

Comment 36 Rajini Karthik 2018-08-06 14:10:45 UTC

I tried the deploy on Yong's env, but FAILED again.

Here are the steps:
1. Used the latest heat-templates and puppet-tripleo packages.
(undercloud) [stack@manager ~]$ rpm -qa | grep heat-template
openstack-tripleo-heat-templates-8.0.3-2.el7ost.noarch
(undercloud) [stack@manager ~]$ rpm -qa | grep puppet-tripleo
puppet-tripleo-8.3.4-2.el7ost.noarch
2. Ran `upload-puppet-modules` provided by Alan.
3. Deployed the overcloud.
But failed.
                
                Cinder failure:
                Error: Evaluation Error: Error while evaluating a Resource Statement, Cinder::Backend::Emc_vnx[tripleo_dellemc_vnx]:
                has no parameter named 'storage_vnx_pool_names'
                expects a value for parameter 'storage_vnx_pool_name'  at /etc/puppet/modules/tripleo/manifests/profile/base/cinder/volume/dellemc_vnx.pp:42 on node osp12ctl0.localdomain

                Manila failure:
                Error: Evaluation Error: Error while evaluating a Resource Statement, Manila::Backend::Dellemc_vnx[tripleo_manila_vnx]:
has no parameter named 'network_plugin_ipv6_enabled'
has no parameter named 'emc_ssl_cert_verify'
has no parameter named 'emc_ssl_cert_path'  at /etc/puppet/modules/tripleo/manifests/profile/base/manila/share.pp:215 on node osp12ctl0.localdomain

Root causes:
I found the puppet-cinder and puppet-manila are out-of-date on the undercloud.
(undercloud) [stack@manager ~]$ rpm -qa | grep puppet-cinder
puppet-cinder-12.4.1-0.20180329071637.4011a82.el7ost.noarch
(undercloud) [stack@manager ~]$ rpm -qa | grep puppet-manila
puppet-manila-12.4.0-0.20180329035214.6c18418.el7ost.noarch
The fixes for above failures were merged: https://review.openstack.org/#/c/562532/ and https://review.openstack.org/#/c/574504/.

My questions:
1.	Could you share me the related rpm packages of puppet-cinder and puppet-manila containing above fixes?
2.	And after getting these two packages, I need to run `upload-puppet-modules` on them together with puppet-tripleo, right?

Thanks!
-Ryan

Comment 37 Alan Bishop 2018-08-06 15:02:56 UTC
I just emailed the latest puppet-cinder and puppet-manila RPMs for OSP-13 (will ship in 13z2) to Rajini.

BTW, I thought Dell EMC's DCI has access to the latest OSP-13 puddles. Is this not in place? If it is, then the latest RPMs should already be available. It will certainly be more expedient to get RPMs from DCI than to request them individually.

Comment 38 Rajini Karthik 2018-08-07 14:01:15 UTC
This has been tested successfully.



I installed the latest puppet-cinder and puppet-manila rpms, and deployed Cinder and Manila SUCCESSFULLY.

The version of rpms:
    openstack-tripleo-heat-templates-8.0.3-2.el7ost.noarch.rpm
    puppet-tripleo-8.3.4-2.el7ost.noarch.rpm
    puppet-cinder-12.4.1-0.20180628102250.641e036.el7ost.noarch.rpm
    puppet-manila-12.4.0-2.el7ost.noarch.rpm

(overcloud) [stack@manager ~]$ openstack volume service list
+------------------+-------------------------------+------+---------+-------+----------------------------+
| Binary           | Host                          | Zone | Status  | State | Updated At                 |
+------------------+-------------------------------+------+---------+-------+----------------------------+
| cinder-scheduler | osp12ctl0                     | nova | enabled | up    | 2018-08-07T07:55:02.000000 |
| cinder-volume    | hostgroup@tripleo_dellemc_vnx | nova | enabled | up    | 2018-08-07T07:55:02.000000 |
+------------------+-------------------------------+------+---------+-------+----------------------------+
(overcloud) [stack@manager ~]$ manila service-list
+----+------------------+------------------------------+------+---------+-------+----------------------------+
| Id | Binary           | Host                         | Zone | Status  | State | Updated_at                 |
+----+------------------+------------------------------+------+---------+-------+----------------------------+
| 1  | manila-scheduler | hostgroup                    | nova | enabled | up    | 2018-08-07T07:55:26.000000 |
| 2  | manila-share     | hostgroup@tripleo_manila_vnx | nova | enabled | up    | 2018-08-07T07:55:25.000000 |
+----+------------------+------------------------------+------+---------+-------+----------------------------+

Thanks!
-Ryan

Comment 39 Sean Merrow 2018-08-08 13:20:08 UTC
Setting Verified by Partner.

Comment 40 Joanne O'Flynn 2018-08-15 07:55:57 UTC
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.

Comment 42 errata-xmlrpc 2018-08-29 16:35:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2574


Note You need to log in before you can comment on or make changes to this bug.