Bug 1648114

Summary:	[OSP13] [puppet-manila] THT deploys manila.conf with deprecated settings for neutron and nova
Product:	Red Hat OpenStack	Reporter:	David Vallee Delisle <dvd>
Component:	puppet-manila	Assignee:	Victoria Martinez de la Cruz <vimartin>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Jason Grosso <jgrosso>
Severity:	high	Docs Contact:	Tana <tberry>
Priority:	high
Version:	13.0 (Queens)	CC:	arkady_kanevsky, cschwede, dvd, gouthamr, jgrosso, jjoyce, jschluet, kurt_hey, marjones, mburns, pgrist, rajini.karthik, slinaber, tbarron, tenobreg, tvignaud, vimartin
Target Milestone:	z5	Keywords:	TestOnly, Triaged, ZStream
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	puppet-manila-12.5.0-3	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1684619 (view as bug list)		Environment:
Last Closed:	2019-03-19 10:34:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1688322

Description David Vallee Delisle 2018-11-08 21:04:04 UTC

Description
===========
Since this commit [1], keystone doesn't support v2.0 requests anymore. When deployed with THT, manila.conf is configured with neutron_admin_* and nova_admin_* which are not supporting the use of a domain for keystone v3 auth.

[1] https://review.openstack.org/#/c/504465/

Steps to reproduce
==================
Deploy manila-share
Create a manila-share

Expected result
===============
We should have a share

Actual result
=============
Manila-share is getting a 404 on keystone /v2.0/tokens

Environment
===========
Queens

Fix
===

When using these settings, it works:

~~~
[nova]
catalog_info=compute:nova:publicURL
catalog_admin_info=compute:nova:adminURL
api_insecure=False
username=nova
password=xxx
project_name=service
project_domain_name=Default
user_domain_name=Default
auth_url=http://192.168.1.6:5000/

[neutron]
auth_type=password
url=http://192.168.1.6:9696
url_timeout=30
username=neutron
password=xxx
project_name=service
project_domain_name=Default
user_domain_name=Default
auth_url=http://192.168.1.6:5000/
api_insecure=False
auth_strategy=keystone
~~~

Comment 11 Paul Grist 2019-01-23 23:52:26 UTC

https://review.openstack.org/#/c/629485/ is merged

Comment 29 Jason Grosso 2019-02-27 23:34:41 UTC

After installing 14z1 on sealusa23 it appear we have an issue with setting all the options for manila  correctly

The deployment was with NetApp with the following parameters:

---
tripleo_heat_templates:
    - /usr/share/openstack-tripleo-heat-templates/environments/manila-netapp-config.yaml
 
custom_templates: 
    parameter_defaults:
        ManilaNetappLogin: 'admin'
        ManilaNetappPassword: 'netapp123'
        ManilaNetappServerHostname: '10.9.76.133'
        ManilaNetappRootVolumeAggr: 'aggr1_n1'
        ManilaNetappTraceFlags: 'api'
        ManilaNetappDriverHandlesShareServers: 'true'


check the following commands to verify the setup 


(overcloud) [stack@undercloud-0 ~]$ manila pool-list
+-----------------------------------+-----------+----------------+----------+
| Name                              | Host      | Backend        | Pool     |
+-----------------------------------+-----------+----------------+----------+
| hostgroup@tripleo_netapp#aggr1_n1 | hostgroup | tripleo_netapp | aggr1_n1 |
| hostgroup@tripleo_netapp#aggr1_n2 | hostgroup | tripleo_netapp | aggr1_n2 |
+-----------------------------------+-----------+----------------+----------+
(overcloud) [stack@undercloud-0 ~]$ manila service-list
+----+------------------+--------------------------+------+---------+-------+----------------------------+
| Id | Binary           | Host                     | Zone | Status  | State | Updated_at                 |
+----+------------------+--------------------------+------+---------+-------+----------------------------+
| 2  | manila-scheduler | hostgroup                | nova | enabled | up    | 2019-02-27T19:44:09.000000 |
| 5  | manila-share     | hostgroup@tripleo_netapp | nova | enabled | up    | 2019-02-27T19:44:10.000000 |
+----+------------------+--------------------------+------+---------+-------+----------------------------+


for Neutron:
[keystone_authtoken]
www_authenticate_uri=http://172.17.1.10:5000/
auth_uri=http://172.17.1.10:5000/
auth_type=password
auth_url=http://172.17.1.10:5000
username=manila
password=pah4b9XPkJcD01qohzZGNPw9I
user_domain_name=Default
project_name=service
project_domain_name=Default
[matchmaker_redis]
[neutron]
auth_url=http://172.17.1.10:5000/
[nova]
auth_url=http://172.17.1.10:5000/
from controller-0

for Nova:  from /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf on controller-0 on seal23

[keystone_authtoken]
www_authenticate_uri=http://172.17.1.10:5000
auth_uri=http://172.17.1.10:5000
memcached_servers=172.17.1.18:11211,172.17.1.29:11211,172.17.1.20:11211
auth_type=password
auth_url=http://172.17.1.10:5000
username=nova
password=X5oHeR3EMzmZNzughNT77lMaC
user_domain_name=Default
project_name=service
project_domain_name=Default


it looks like we have the following issues : Thanks Goutham :)

$ manila message-list
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| ID                                   | Resource Type | Resource ID                          | Action ID | User Message                                                                                                                                                                                                                                                                                                | Detail ID | Created At                 |
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| 901046cf-a37c-4c3f-8246-92f3c5b73637 | SHARE         | aa192844-0332-4c33-821d-56c4e24d9880 | 002       | create: Could not find an existing share server or allocate one on the share network provided. You may use a different share network, or verify the network details in the share network and retry your request. If this doesn't work, contact your administrator to troubleshoot issues with your network. | 004       | 2019-02-27T22:05:00.000000 |
| eb848d05-85a0-4422-bfc0-53f2b32dc458 | SHARE         | c4ddfadb-807b-43f4-8a7a-af41f5632c18 | 002       | create: Could not find an existing share server or allocate one on the share network provided. You may use a different share network, or verify the network details in the share network and retry your request. If this doesn't work, contact your administrator to troubleshoot issues with your network. | 004       | 2019-02-27T21:48:12.000000 |
| 4092fc0e-6553-4519-8c0f-68567ba07cd7 | SHARE         | 92f9638a-f2b8-4951-91d3-44186bab8140 | 002       | create: Could not find an existing share server or allocate one on the share network provided. You may use a different share network, or verify the network details in the share network and retry your request. If this doesn't work, contact your administrator to troubleshoot issues with your network. | 004       | 2019-02-27T21:39:48.000000 |
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+








2019-02-27 22:05:00.808 26 DEBUG oslo_concurrency.lockutils [req-6e60dfae-780c-4f82-9901-1ed50f8350d6 f328015adf2e43ee839377ffe222052c d1b9a692d87a4437b633f4370adbb65a - - -] Lock "share_manager_b3abe4fc-c077-4b17-b70c-a863845418c5" released by "manila.share.manager._wrapped_provide_share_server_for_share" :: held 0.227s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2019-02-27 22:05:00.808 26 ERROR manila.share.manager [req-6e60dfae-780c-4f82-9901-1ed50f8350d6 f328015adf2e43ee839377ffe222052c d1b9a692d87a4437b633f4370adbb65a - - -] Creation of share instance 09e20b10-350e-4506-a7a3-dd93c6df9bc2 failed: failed to get share server.: Unauthorized: Cannot load auth plugin for neutron
2019-02-27 22:05:00.830 26 INFO manila.message.api [req-6e60dfae-780c-4f82-9901-1ed50f8350d6 f328015adf2e43ee839377ffe222052c d1b9a692d87a4437b633f4370adbb65a - - -] Creating message record for request_id = req-6e60dfae-780c-4f82-9901-1ed50f8350d6
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server [req-6e60dfae-780c-4f82-9901-1ed50f8350d6 f328015adf2e43ee839377ffe222052c d1b9a692d87a4437b633f4370adbb65a - - -] Exception during message handling: Unauthorized: Cannot load auth plugin for neutron
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 187, in wrapped
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/utils.py", line 598, in wrapper
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1718, in create_share_instance
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     detail=message_field.Detail.NO_SHARE_SERVER)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1700, in create_share_instance
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     share_group=share_group_ref,
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 620, in _provide_share_server_for_share
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     return _wrapped_provide_share_server_for_share()
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 616, in _wrapped_provide_share_server_for_share
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 632, in _create_share_server_in_backend
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     metadata=metadata)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 3513, in _setup_server
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     self.driver.deallocate_network(context, share_server['id'])
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 3433, in _setup_server
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     self.driver.allocate_network(context, share_server, share_network)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/driver.py", line 846, in allocate_network
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     context, share_server, share_network, **kwargs)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/network/neutron/neutron_network_plugin.py", line 147, in allocate_network
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     if not self._has_provider_network_extension():
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/network/neutron/neutron_network_plugin.py", line 242, in _has_provider_network_extension
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     extensions = self.neutron_api.list_extensions()
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/network/neutron/api.py", line 225, in list_extensions
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     extensions_list = self.client.list_extensions().get('extensions')
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/network/neutron/api.py", line 100, in client
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     return self.get_client(context.get_admin_context())
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/network/neutron/api.py", line 113, in get_client
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     region_name=CONF[NEUTRON_GROUP].region_name,
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/common/client_auth.py", line 92, in get_client
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     self.admin_auth = self._load_auth_plugin()
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/common/client_auth.py", line 78, in _load_auth_plugin
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server     raise self.exception_module.Unauthorized(message=msg)
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server Unauthorized: Cannot load auth plugin for neutron
2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server
2019-02-27 22:06:11.938 26 DEBUG oslo_service.periodic_task [req-86370280-5e15-489c-bc34-021ee4607b4c - - - - -] Running periodic task ShareManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
2019-02-27 22:06:11.938 26 DEBUG manila.manager [req-86370280-5e15-489c-bc34-021ee4607b4c - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/manila/manager.py:142




[heat-admin@controller-0 ~]$ sudo grep -v '^$\|^\s*\#' /var/lib/config-data/puppet-generated/manila/etc/manila/manila.conf                                                                             
[DEFAULT]
state_path=/var/lib/manila
host=hostgroup
storage_availability_zone=nova
default_share_type=default
rootwrap_config=/etc/manila/rootwrap.conf
auth_strategy=keystone
enabled_share_backends=tripleo_netapp
enabled_share_protocols=NFS,CIFS
network_api_class=manila.network.neutron.neutron_network_plugin.NeutronNetworkPlugin
network_plugin_ipv4_enabled=True
network_plugin_ipv6_enabled=False
osapi_share_listen=172.17.1.18
osapi_share_workers=4
debug=True
log_dir=/var/log/manila
transport_url=rabbit://guest:XClUy96ERufeUUwhGew8RTxGf.localdomain:5672,guest:XClUy96ERufeUUwhGew8RTxGf.localdomain:5672,guest:XClUy96ERufeUUwhGew8RTxGf@contr
oller-2.internalapi.localdomain:5672/?ssl=0
control_exchange=openstack
api_paste_config=/etc/manila/api-paste.ini
[cinder]
[cors]
[database]
connection=mysql+pymysql://manila:pah4b9XPkJcD01qohzZGNPw9I.1.10/manila?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf
max_retries=-1
db_max_retries=-1
[healthcheck]
[keystone_authtoken]
www_authenticate_uri=http://172.17.1.10:5000/
auth_uri=http://172.17.1.10:5000/
auth_type=password
auth_url=http://172.17.1.10:5000
username=manila
password=pah4b9XPkJcD01qohzZGNPw9I
user_domain_name=Default
project_name=service
project_domain_name=Default
[matchmaker_redis]
[neutron]
auth_url=http://172.17.1.10:5000/
[nova]
auth_url=http://172.17.1.10:5000/
[oslo_concurrency]
lock_path=/tmp/manila/manila_locks
[oslo_messaging_amqp]
container_name=guest
idle_timeout=0
trace=False
allow_insecure_clients=False
server_request_prefix=exclusive
broadcast_prefix=broadcast
group_request_prefix=unicast
[oslo_messaging_kafka]
[oslo_messaging_notifications]
driver=messagingv2
transport_url=rabbit://guest:XClUy96ERufeUUwhGew8RTxGf.localdomain:5672,guest:XClUy96ERufeUUwhGew8RTxGf.localdomain:5672,guest:XClUy96ERufeUUwhGew8RTxGf.localdomain:5672/?ssl=0
[oslo_messaging_rabbit]
ssl=False
[oslo_messaging_zmq]
[oslo_middleware]
enable_proxy_headers_parsing=True
[oslo_policy]
policy_file=/etc/manila/policy.json
[ssl]
[tripleo_netapp]
share_driver=manila.share.drivers.netapp.common.NetAppDriver
driver_handles_share_servers=True
netapp_login=admin
netapp_password=*******
netapp_server_hostname=*******
share_backend_name=tripleo_netapp
netapp_transport_type=http
netapp_storage_family=ontap_cluster
netapp_server_port=80
netapp_volume_name_template=share_%(share_id)s
netapp_vserver=
netapp_vserver_name_template=os_%s
netapp_lif_name_template=os_%(net_allocation_id)s
netapp_aggregate_name_search_pattern=(.*)
netapp_root_volume_aggregate=aggr1_n1
netapp_root_volume_name=root
netapp_port_name_search_pattern=(.*)
netapp_trace_flags=api

Comment 31 Jason Grosso 2019-03-01 17:16:22 UTC

I apologize this was mislabeled on my comment [29] this was tested on 13 z5  http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/z5/RH7-RHOS-13.0/   I had been switching between 14z1 and 13 z5 bugs.

Comment 36 Rajini Karthik 2019-03-13 20:34:06 UTC

We are hitting this bug on unity manila backend. Will this be released in the Z5

Comment 37 Paul Grist 2019-03-13 21:47:34 UTC

Hi Rajini,

This fix needed one additional patch that is not in z5 errata release:  https://review.openstack.org/#/c/641436/

We are working to expedite the a release for this and you can monitor the BZ for updates and we can follow up with an email as well.

Thanks,
Paul

Comment 38 Goutham Pacha Ravi 2019-03-14 00:57:53 UTC

The bugzilla tracking the additional patch Paul mentioned has been linked to this BZ (See: "Blocks"): https://bugzilla.redhat.com/show_bug.cgi?id=1688322

Comment 39 Jason Grosso 2019-03-18 17:39:15 UTC

verified as part of https://bugzilla.redhat.com/show_bug.cgi?id=1688322

Comment 40 Lon Hohberger 2019-03-19 10:34:31 UTC

According to our records, this should be resolved by puppet-manila-12.5.0-3.el7ost.  This build is available now.