Description =========== Since this commit [1], keystone doesn't support v2.0 requests anymore. When deployed with THT, manila.conf is configured with neutron_admin_* and nova_admin_* which are not supporting the use of a domain for keystone v3 auth. [1] https://review.openstack.org/#/c/504465/ Steps to reproduce ================== Deploy manila-share Create a manila-share Expected result =============== We should have a share Actual result ============= Manila-share is getting a 404 on keystone /v2.0/tokens Environment =========== Queens Fix === When using these settings, it works: ~~~ [nova] catalog_info=compute:nova:publicURL catalog_admin_info=compute:nova:adminURL api_insecure=False username=nova password=xxx project_name=service project_domain_name=Default user_domain_name=Default auth_url=http://192.168.1.6:5000/ [neutron] auth_type=password url=http://192.168.1.6:9696 url_timeout=30 username=neutron password=xxx project_name=service project_domain_name=Default user_domain_name=Default auth_url=http://192.168.1.6:5000/ api_insecure=False auth_strategy=keystone ~~~
https://review.openstack.org/#/c/629485/ is merged
After installing 14z1 on sealusa23 it appear we have an issue with setting all the options for manila correctly The deployment was with NetApp with the following parameters: --- tripleo_heat_templates: - /usr/share/openstack-tripleo-heat-templates/environments/manila-netapp-config.yaml custom_templates: parameter_defaults: ManilaNetappLogin: 'admin' ManilaNetappPassword: 'netapp123' ManilaNetappServerHostname: '10.9.76.133' ManilaNetappRootVolumeAggr: 'aggr1_n1' ManilaNetappTraceFlags: 'api' ManilaNetappDriverHandlesShareServers: 'true' check the following commands to verify the setup (overcloud) [stack@undercloud-0 ~]$ manila pool-list +-----------------------------------+-----------+----------------+----------+ | Name | Host | Backend | Pool | +-----------------------------------+-----------+----------------+----------+ | hostgroup@tripleo_netapp#aggr1_n1 | hostgroup | tripleo_netapp | aggr1_n1 | | hostgroup@tripleo_netapp#aggr1_n2 | hostgroup | tripleo_netapp | aggr1_n2 | +-----------------------------------+-----------+----------------+----------+ (overcloud) [stack@undercloud-0 ~]$ manila service-list +----+------------------+--------------------------+------+---------+-------+----------------------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | +----+------------------+--------------------------+------+---------+-------+----------------------------+ | 2 | manila-scheduler | hostgroup | nova | enabled | up | 2019-02-27T19:44:09.000000 | | 5 | manila-share | hostgroup@tripleo_netapp | nova | enabled | up | 2019-02-27T19:44:10.000000 | +----+------------------+--------------------------+------+---------+-------+----------------------------+ for Neutron: [keystone_authtoken] www_authenticate_uri=http://172.17.1.10:5000/ auth_uri=http://172.17.1.10:5000/ auth_type=password auth_url=http://172.17.1.10:5000 username=manila password=pah4b9XPkJcD01qohzZGNPw9I user_domain_name=Default project_name=service project_domain_name=Default [matchmaker_redis] [neutron] auth_url=http://172.17.1.10:5000/ [nova] auth_url=http://172.17.1.10:5000/ from controller-0 for Nova: from /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf on controller-0 on seal23 [keystone_authtoken] www_authenticate_uri=http://172.17.1.10:5000 auth_uri=http://172.17.1.10:5000 memcached_servers=172.17.1.18:11211,172.17.1.29:11211,172.17.1.20:11211 auth_type=password auth_url=http://172.17.1.10:5000 username=nova password=X5oHeR3EMzmZNzughNT77lMaC user_domain_name=Default project_name=service project_domain_name=Default it looks like we have the following issues : Thanks Goutham :) $ manila message-list +--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+ | ID | Resource Type | Resource ID | Action ID | User Message | Detail ID | Created At | +--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+ | 901046cf-a37c-4c3f-8246-92f3c5b73637 | SHARE | aa192844-0332-4c33-821d-56c4e24d9880 | 002 | create: Could not find an existing share server or allocate one on the share network provided. You may use a different share network, or verify the network details in the share network and retry your request. If this doesn't work, contact your administrator to troubleshoot issues with your network. | 004 | 2019-02-27T22:05:00.000000 | | eb848d05-85a0-4422-bfc0-53f2b32dc458 | SHARE | c4ddfadb-807b-43f4-8a7a-af41f5632c18 | 002 | create: Could not find an existing share server or allocate one on the share network provided. You may use a different share network, or verify the network details in the share network and retry your request. If this doesn't work, contact your administrator to troubleshoot issues with your network. | 004 | 2019-02-27T21:48:12.000000 | | 4092fc0e-6553-4519-8c0f-68567ba07cd7 | SHARE | 92f9638a-f2b8-4951-91d3-44186bab8140 | 002 | create: Could not find an existing share server or allocate one on the share network provided. You may use a different share network, or verify the network details in the share network and retry your request. If this doesn't work, contact your administrator to troubleshoot issues with your network. | 004 | 2019-02-27T21:39:48.000000 | +--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+ 2019-02-27 22:05:00.808 26 DEBUG oslo_concurrency.lockutils [req-6e60dfae-780c-4f82-9901-1ed50f8350d6 f328015adf2e43ee839377ffe222052c d1b9a692d87a4437b633f4370adbb65a - - -] Lock "share_manager_b3abe4fc-c077-4b17-b70c-a863845418c5" released by "manila.share.manager._wrapped_provide_share_server_for_share" :: held 0.227s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285 2019-02-27 22:05:00.808 26 ERROR manila.share.manager [req-6e60dfae-780c-4f82-9901-1ed50f8350d6 f328015adf2e43ee839377ffe222052c d1b9a692d87a4437b633f4370adbb65a - - -] Creation of share instance 09e20b10-350e-4506-a7a3-dd93c6df9bc2 failed: failed to get share server.: Unauthorized: Cannot load auth plugin for neutron 2019-02-27 22:05:00.830 26 INFO manila.message.api [req-6e60dfae-780c-4f82-9901-1ed50f8350d6 f328015adf2e43ee839377ffe222052c d1b9a692d87a4437b633f4370adbb65a - - -] Creating message record for request_id = req-6e60dfae-780c-4f82-9901-1ed50f8350d6 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server [req-6e60dfae-780c-4f82-9901-1ed50f8350d6 f328015adf2e43ee839377ffe222052c d1b9a692d87a4437b633f4370adbb65a - - -] Exception during message handling: Unauthorized: Cannot load auth plugin for neutron 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 187, in wrapped 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/utils.py", line 598, in wrapper 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1718, in create_share_instance 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server detail=message_field.Detail.NO_SHARE_SERVER) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1700, in create_share_instance 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server share_group=share_group_ref, 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 620, in _provide_share_server_for_share 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server return _wrapped_provide_share_server_for_share() 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 616, in _wrapped_provide_share_server_for_share 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 632, in _create_share_server_in_backend 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server metadata=metadata) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 3513, in _setup_server 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server self.driver.deallocate_network(context, share_server['id']) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 3433, in _setup_server 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server self.driver.allocate_network(context, share_server, share_network) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/driver.py", line 846, in allocate_network 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server context, share_server, share_network, **kwargs) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/network/neutron/neutron_network_plugin.py", line 147, in allocate_network 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server if not self._has_provider_network_extension(): 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/network/neutron/neutron_network_plugin.py", line 242, in _has_provider_network_extension 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server extensions = self.neutron_api.list_extensions() 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/network/neutron/api.py", line 225, in list_extensions 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server extensions_list = self.client.list_extensions().get('extensions') 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/network/neutron/api.py", line 100, in client 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server return self.get_client(context.get_admin_context()) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/network/neutron/api.py", line 113, in get_client 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server region_name=CONF[NEUTRON_GROUP].region_name, 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/common/client_auth.py", line 92, in get_client 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server self.admin_auth = self._load_auth_plugin() 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/common/client_auth.py", line 78, in _load_auth_plugin 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server raise self.exception_module.Unauthorized(message=msg) 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server Unauthorized: Cannot load auth plugin for neutron 2019-02-27 22:05:00.845 26 ERROR oslo_messaging.rpc.server 2019-02-27 22:06:11.938 26 DEBUG oslo_service.periodic_task [req-86370280-5e15-489c-bc34-021ee4607b4c - - - - -] Running periodic task ShareManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 2019-02-27 22:06:11.938 26 DEBUG manila.manager [req-86370280-5e15-489c-bc34-021ee4607b4c - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/manila/manager.py:142 [heat-admin@controller-0 ~]$ sudo grep -v '^$\|^\s*\#' /var/lib/config-data/puppet-generated/manila/etc/manila/manila.conf [DEFAULT] state_path=/var/lib/manila host=hostgroup storage_availability_zone=nova default_share_type=default rootwrap_config=/etc/manila/rootwrap.conf auth_strategy=keystone enabled_share_backends=tripleo_netapp enabled_share_protocols=NFS,CIFS network_api_class=manila.network.neutron.neutron_network_plugin.NeutronNetworkPlugin network_plugin_ipv4_enabled=True network_plugin_ipv6_enabled=False osapi_share_listen=172.17.1.18 osapi_share_workers=4 debug=True log_dir=/var/log/manila transport_url=rabbit://guest:XClUy96ERufeUUwhGew8RTxGf.localdomain:5672,guest:XClUy96ERufeUUwhGew8RTxGf.localdomain:5672,guest:XClUy96ERufeUUwhGew8RTxGf@contr oller-2.internalapi.localdomain:5672/?ssl=0 control_exchange=openstack api_paste_config=/etc/manila/api-paste.ini [cinder] [cors] [database] connection=mysql+pymysql://manila:pah4b9XPkJcD01qohzZGNPw9I.1.10/manila?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf max_retries=-1 db_max_retries=-1 [healthcheck] [keystone_authtoken] www_authenticate_uri=http://172.17.1.10:5000/ auth_uri=http://172.17.1.10:5000/ auth_type=password auth_url=http://172.17.1.10:5000 username=manila password=pah4b9XPkJcD01qohzZGNPw9I user_domain_name=Default project_name=service project_domain_name=Default [matchmaker_redis] [neutron] auth_url=http://172.17.1.10:5000/ [nova] auth_url=http://172.17.1.10:5000/ [oslo_concurrency] lock_path=/tmp/manila/manila_locks [oslo_messaging_amqp] container_name=guest idle_timeout=0 trace=False allow_insecure_clients=False server_request_prefix=exclusive broadcast_prefix=broadcast group_request_prefix=unicast [oslo_messaging_kafka] [oslo_messaging_notifications] driver=messagingv2 transport_url=rabbit://guest:XClUy96ERufeUUwhGew8RTxGf.localdomain:5672,guest:XClUy96ERufeUUwhGew8RTxGf.localdomain:5672,guest:XClUy96ERufeUUwhGew8RTxGf.localdomain:5672/?ssl=0 [oslo_messaging_rabbit] ssl=False [oslo_messaging_zmq] [oslo_middleware] enable_proxy_headers_parsing=True [oslo_policy] policy_file=/etc/manila/policy.json [ssl] [tripleo_netapp] share_driver=manila.share.drivers.netapp.common.NetAppDriver driver_handles_share_servers=True netapp_login=admin netapp_password=******* netapp_server_hostname=******* share_backend_name=tripleo_netapp netapp_transport_type=http netapp_storage_family=ontap_cluster netapp_server_port=80 netapp_volume_name_template=share_%(share_id)s netapp_vserver= netapp_vserver_name_template=os_%s netapp_lif_name_template=os_%(net_allocation_id)s netapp_aggregate_name_search_pattern=(.*) netapp_root_volume_aggregate=aggr1_n1 netapp_root_volume_name=root netapp_port_name_search_pattern=(.*) netapp_trace_flags=api
I apologize this was mislabeled on my comment [29] this was tested on 13 z5 http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/z5/RH7-RHOS-13.0/ I had been switching between 14z1 and 13 z5 bugs.
We are hitting this bug on unity manila backend. Will this be released in the Z5
Hi Rajini, This fix needed one additional patch that is not in z5 errata release: https://review.openstack.org/#/c/641436/ We are working to expedite the a release for this and you can monitor the BZ for updates and we can follow up with an email as well. Thanks, Paul
The bugzilla tracking the additional patch Paul mentioned has been linked to this BZ (See: "Blocks"): https://bugzilla.redhat.com/show_bug.cgi?id=1688322
verified as part of https://bugzilla.redhat.com/show_bug.cgi?id=1688322
According to our records, this should be resolved by puppet-manila-12.5.0-3.el7ost. This build is available now.