Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2246563

Summary:	[RHOSP17.1] TripleO files missing for Manila FlashBlade driver
Product:	Red Hat OpenStack	Reporter:	Simon Dodsley <simon>
Component:	openstack-tripleo-heat-templates	Assignee:	Goutham Pacha Ravi <gouthamr>
Status:	CLOSED ERRATA	QA Contact:	vhariria
Severity:	urgent	Docs Contact:
Priority:	high
Version:	17.1 (Wallaby)	CC:	abishop, ashrodri, atiwary, gbrinn, gouthamr, jschluet, mariel, mburns, pgrist, prgutier, slinaber, vhariria
Target Milestone:	z2	Keywords:	OtherQA, TestOnly, Triaged
Target Release:	17.1
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-14.3.1-17.1.20231103010826.el9ost	Doc Type:	Bug Fix
Doc Text:	Before this update, Director did not include the puppet modules and the heat templates that you needed to configure the Pure Flashblade driver with your Red Hat Openstack Shared File System Service (manila). With this update, Director now includes the necessary puppet modules and heat templates for your configuration.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2024-01-16 14:31:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Simon Dodsley 2023-10-27 13:42:31 UTC

As per BZ #2033811 the FlashBlade Manila driver and associated components were supposed to have been backported from Xena from Wallaby.

It appears that only the manila driver code itself was backported and all the required puppet and triple components were not backported.

The relation chain is as follows:

tripleo-heat-templates: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/792850
puppet-tripleo: https://review.opendev.org/c/openstack/puppet-tripleo/+/792849
puppet-manila: https://review.opendev.org/c/openstack/puppet-manila/+/792848

Due to this we cannot start the Manila FlashBlade driver and therefore cannot continue with the Manila Certification process for RHOSP 17.1.

In our RHOSP 17.1 deployment we are seeing the following packages that do not contain the required files:
puppet-manila-18.5.2-1.20230127001703.a72a7d5.el9ost.noarch
puppet-tripleo-14.2.3-1.20230517011019.el9ost.noarch
openstack-tripleo-heat-templates-14.3.1-1.20230519151024.el9ost.noarch

Please complete the backport of all required componenets.

Comment 1 Goutham Pacha Ravi 2023-10-27 21:21:11 UTC

(In reply to Simon Dodsley from comment #0)
> As per BZ #2033811 the FlashBlade Manila driver and associated components
> were supposed to have been backported from Xena from Wallaby.
> 
> It appears that only the manila driver code itself was backported and all
> the required puppet and triple components were not backported.
> 
> The relation chain is as follows:
> 
> tripleo-heat-templates:
> https://review.opendev.org/c/openstack/tripleo-heat-templates/+/792850
> puppet-tripleo:
> https://review.opendev.org/c/openstack/puppet-tripleo/+/792849
> puppet-manila: https://review.opendev.org/c/openstack/puppet-manila/+/792848
> 
> Due to this we cannot start the Manila FlashBlade driver and therefore
> cannot continue with the Manila Certification process for RHOSP 17.1.
> 
> In our RHOSP 17.1 deployment we are seeing the following packages that do
> not contain the required files:
> puppet-manila-18.5.2-1.20230127001703.a72a7d5.el9ost.noarch
> puppet-tripleo-14.2.3-1.20230517011019.el9ost.noarch
> openstack-tripleo-heat-templates-14.3.1-1.20230519151024.el9ost.noarch
> 
> Please complete the backport of all required componenets.


Hmm, was under the impression these were backported upstream - but its been a while. thanks for the report; I'll propose the downstream-only backports. We'll try to get this into 17.1z3 at this point.

Comment 10 Goutham Pacha Ravi 2023-12-01 19:38:36 UTC

While testing backports we discovered a couple of bugs in the TripleO/Puppet code that landed in Xena:


Tripleo-heat-templates:

diff --git a/deployment/manila/manila-backend-flashblade-puppet.yaml b/deployment/manila/manila-backend-flashblade-puppet.yaml
index 643d3c38a..3025e18a1 100644
--- a/deployment/manila/manila-backend-flashblade-puppet.yaml
+++ b/deployment/manila/manila-backend-flashblade-puppet.yaml
@@ -66,10 +66,11 @@ outputs:
       service_name: manila_backend_flashblade
       config_settings:
         map_merge:
-          - manila::backend::flashblade::share_backend_name: {get_param: ManilaFlashBladeBackendName}
+          - manila::backend::flashblade::title: {get_param: ManilaFlashBladeBackendName}
+            manila::backend::flashblade::share_backend_name: {get_param: ManilaFlashBladeBackendName}
             manila::backend::flashblade::flashblade_mgmt_vip: {get_param: ManilaFlashBladeMgmtIp}
             manila::backend::flashblade::flashblade_data_vip: {get_param: ManilaFlashBladeDataIp}
-            manila::backend::flashblade::flashblade_api_token: {get_param: ManilaFlashBladeAPIToken}
+            manila::backend::flashblade::flashblade_api: {get_param: ManilaFlashBladeAPIToken}
             manila::backend::flashblade::flashblade_eradicate : {get_param: ManilaFlashBladeEradicate}
           - if:
               - { equals: [ { get_param: ManilaFlashBladeAvailabilityZone }, '' ] }




Puppet-TripleO:

diff --git a/manifests/profile/base/manila/share.pp b/manifests/profile/base/manila/share.pp
index 7557d9d07..39a3d991c 100644
--- a/manifests/profile/base/manila/share.pp
+++ b/manifests/profile/base/manila/share.pp
@@ -284,7 +284,7 @@ class tripleo::profile::base::manila::share (
         'flashblade_mgmt_vip'          => hiera('manila::backend::flashblade::flashblade_mgmt_vip', undef),
         'backend_availability_zone'    => hiera('manila::backend::flashblade::backend_availability_zone', undef),
         'flashblade_data_vip'          => hiera('manila::backend::flashblade::flashblade_data_vip', undef),
-        'flashblade_api_token'         => hiera('manila::backend::flashblade::flashblade_api_token', undef),
+        'flashblade_api'               => hiera('manila::backend::flashblade::flashblade_api', undef),
         'flashblade_eradicate'         => hiera('manila::backend::flashblade::flashblade_eradicate', undef),
       })})
     }


These changes have no workarounds.

Comment 18 Goutham Pacha Ravi 2023-12-18 22:16:14 UTC

We were able to sanity test this:


$ cat overcloud_deploy.sh
#!/bin/bash

openstack overcloud deploy \
--timeout 240 \
--templates /usr/share/openstack-tripleo-heat-templates \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/manila-flashblade-config.yaml \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.corp.redhat.com \
--deployed-server \
-e /home/stack/templates/overcloud-vip-deployed.yaml \
-e /home/stack/templates/overcloud-networks-deployed.yaml \
-e /home/stack/templates/overcloud-baremetal-deployed.yaml \
--networks-file /home/stack/virt/network/network_data_v2.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovs-dvr.yaml \
-e /home/stack/virt/network/network-environment_v2.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/ntp_pool.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/virt/pure_storage.yaml \
--log-file overcloud_deployment_59.log


$ cat /home/stack/virt/pure_storage.yaml
parameter_defaults:
  ContainerManilaShareImage: registry.connect.redhat.com/purestorage/openstack-manila-share-pure-rhosp-17-1:17.1.0
  ManilaFlashBladeBackendName: 'tripleo_flashblade'
  ManilaFlashBladeMgmtIp: '192.168.10.19'
  ManilaFlashBladeDataIp: '172.63.0.12'
  ManilaFlashBladeAPIToken: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
  ManilaFlashBladeEradicate: true


From the controller node, post deployment:

$ sudo grep -v '^$\|^\s*\#'  /var/lib/config-data/puppet-generated/manila/etc/manila/manila.conf
[DEFAULT]
state_path=/var/lib/manila
host=hostgroup
storage_availability_zone=nova
default_share_type=default
rootwrap_config=/etc/manila/rootwrap.conf
auth_strategy=keystone
enabled_share_backends=tripleo_flashblade
enabled_share_protocols=NFS,CIFS
network_api_class=manila.network.neutron.neutron_network_plugin.NeutronNetworkPlugin
network_plugin_ipv4_enabled=True
network_plugin_ipv6_enabled=False
osapi_share_listen=172.17.1.83
osapi_share_workers=4
debug=True
log_dir=/var/log/manila
transport_url=rabbit://guest:yiTZvmXHZv3FcZNyftn4HWhAn.redhat.local:5672,guest:yiTZvmXHZv3FcZNyftn4HWhAn.redhat.local:5672,guest:yiTZvmXHZv3FcZNyftn4HWhAn.redhat.local:5672/?ssl=0
control_exchange=openstack
api_paste_config=/etc/manila/api-paste.ini
[cinder]
[cors]
[database]
connection=mysql+pymysql://manila:42kMpgOhDGYvO1jHC1Ag3QHAG.1.57/manila?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo
max_retries=-1
db_max_retries=-1
[glance]
[healthcheck]
[keystone_authtoken]
www_authenticate_uri=http://10.0.0.102:5000/
interface=internal
region_name=regionOne
memcached_servers=controller-0.internalapi.redhat.local:11211,controller-1.internalapi.redhat.local:11211,controller-2.internalapi.redhat.local:11211
memcache_use_advanced_pool=True
auth_type=password
auth_url=http://172.17.1.57:5000
username=manila
password=42kMpgOhDGYvO1jHC1Ag3QHAG
user_domain_name=Default
project_name=service
project_domain_name=Default
[neutron]
region_name=regionOne
auth_url=http://172.17.1.57:5000
auth_type=password
password=42kMpgOhDGYvO1jHC1Ag3QHAG
project_domain_name=Default
project_name=service
user_domain_name=Default
username=manila
[nova]
region_name=regionOne
auth_url=http://172.17.1.57:5000
auth_type=password
password=42kMpgOhDGYvO1jHC1Ag3QHAG
project_domain_name=Default
project_name=service
user_domain_name=Default
username=manila
[oslo_concurrency]
lock_path=/tmp/manila/manila_locks
[oslo_messaging_amqp]
container_name=guest
idle_timeout=0
trace=False
server_request_prefix=exclusive
broadcast_prefix=broadcast
group_request_prefix=unicast
[oslo_messaging_kafka]
[oslo_messaging_notifications]
driver=noop
transport_url=rabbit://guest:yiTZvmXHZv3FcZNyftn4HWhAn.redhat.local:5672,guest:yiTZvmXHZv3FcZNyftn4HWhAn.redhat.local:5672,guest:yiTZvmXHZv3FcZNyftn4HWhAn.redhat.local:5672/?ssl=0
[oslo_messaging_rabbit]
heartbeat_in_pthread=False
[oslo_middleware]
enable_proxy_headers_parsing=True
[oslo_policy]
policy_file=/etc/manila/policy.yaml
[ssl]
[tripleo_flashblade]
share_driver=manila.share.drivers.purestorage.flashblade.FlashBladeShareDriver
driver_handles_share_servers=False
flashblade_eradicate=True
flashblade_api=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
flashblade_mgmt_vip=192.168.10.19
flashblade_data_vip=172.63.0.12
share_backend_name=tripleo_flashblade



We can see that the config made the manila share manager service attempt to connect with the fake values we provided, and fail... an excerpt from manila-share.log:


2023-12-18 22:04:01.365 12 ERROR manila.share.manager [req-6710c432-a9ee-4d3e-89da-8090b4c5c17d - - - - -] Error encountered during initialization of driver FlashBladeShareDriver@hostgroup@tripleo_flashblade: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.10.19', port=443): Max retries exceeded with url: /api/api_version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f88281a9df0>: Failed to establish a new connection: [Errno 110] ETIMEDOUT'))
2023-12-18 22:04:01.365 12 ERROR manila.share.manager Traceback (most recent call last):
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     conn = connection.create_connection(
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     raise err
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     sock.connect(sa)
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/eventlet/greenio/base.py", line 253, in connect
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     socket_checkerr(fd)
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/eventlet/greenio/base.py", line 51, in socket_checkerr
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     raise socket.error(err, errno.errorcode[err])
2023-12-18 22:04:01.365 12 ERROR manila.share.manager TimeoutError: [Errno 110] ETIMEDOUT
2023-12-18 22:04:01.365 12 ERROR manila.share.manager
2023-12-18 22:04:01.365 12 ERROR manila.share.manager During handling of the above exception, another exception occurred:
2023-12-18 22:04:01.365 12 ERROR manila.share.manager
2023-12-18 22:04:01.365 12 ERROR manila.share.manager Traceback (most recent call last):
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     httplib_response = self._make_request(
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     self._validate_conn(conn)
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     conn.connect()
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/connection.py", line 353, in connect
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     conn = self._new_conn()
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     raise NewConnectionError(
2023-12-18 22:04:01.365 12 ERROR manila.share.manager urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f88281a9df0>: Failed to establish a new connection: [Errno 110] ETIMEDOUT
2023-12-18 22:04:01.365 12 ERROR manila.share.manager
2023-12-18 22:04:01.365 12 ERROR manila.share.manager During handling of the above exception, another exception occurred:
2023-12-18 22:04:01.365 12 ERROR manila.share.manager
2023-12-18 22:04:01.365 12 ERROR manila.share.manager Traceback (most recent call last):
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/manila/share/manager.py", line 346, in _driver_setup
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     self.driver.do_setup(ctxt)
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/manila/share/drivers/purestorage/flashblade.py", line 124, in do_setup
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     self._sys = purity_fb.PurityFb(self.management_address)
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/purity_fb/fb_client.py", line 68, in __init__
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     self._version = self._choose_rest_version()
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/purity_fb/fb_client.py", line 187, in _choose_rest_version
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     versions = self.list_versions()
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/purity_fb/fb_client.py", line 95, in list_versions
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.api_version.list_versions().versions
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/purity_fb/api_client/version_api.py", line 64, in list_versions
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     (data) = self.list_versions_with_http_info(**kwargs)
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/purity_fb/api_client/version_api.py", line 120, in list_versions_with_http_info
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.api_client.call_api('/api_version', 'GET',
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/purity_fb/api_client/api_client.py", line 327, in call_api
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.__call_api(resource_path, method,
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/purity_fb/api_client/api_client.py", line 151, in __call_api
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     response_data = self.request(method, url,
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/purity_fb/api_client/api_client.py", line 350, in request
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.rest_client.GET(url,
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/purity_fb/api_client/rest.py", line 240, in GET
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.request("GET", url,
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.request("GET", url,
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/purity_fb/api_client/rest.py", line 203, in request
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     r = self.pool_manager.request(method, url,
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.request_encode_url(
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.urlopen(method, url, **extra_kw)
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     response = conn.urlopen(method, u.request_uri, **kw)
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.urlopen(
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.urlopen(
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     return self.urlopen(
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     retries = retries.increment(
2023-12-18 22:04:01.365 12 ERROR manila.share.manager   File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
2023-12-18 22:04:01.365 12 ERROR manila.share.manager     raise MaxRetryError(_pool, url, error or ResponseError(cause))
2023-12-18 22:04:01.365 12 ERROR manila.share.manager urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.10.19', port=443): Max retries exceeded with url: /api/api_version (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f88281a9df0>: Failed to establish a new connection: [Errno 110] ETIMEDOUT'))
2023-12-18 22:04:01.365 12 ERROR manila.share.manager
2023-12-18 22:04:01.368 12 DEBUG manila.utils [req-6710c432-a9ee-4d3e-89da-8090b4c5c17d - - - - -] Failed attempt 8 _print_stop /usr/lib/python3.9/site-packages/manila/utils.py:498
2023-12-18 22:04:01.368 12 DEBUG manila.utils [req-6710c432-a9ee-4d3e-89da-8090b4c5c17d - - - - -] Have been at this for 4696.039 seconds _print_stop /usr/lib/python3.9/site-packages/manila/utils.py:499
2023-12-18 22:04:01.368 12 DEBUG manila.utils [req-6710c432-a9ee-4d3e-89da-8090b4c5c17d - - - - -] Sleeping for 512.0 seconds. _backoff_sleep /usr/lib/python3.9/site-packages/manila/utils.py:493





This needs further verification from Pure Storage..

Comment 19 Simon Dodsley 2023-12-18 22:47:04 UTC

This looks correct. The errors you are seeing are exactly as we would expect given you do not have a physical FlashBlade.

The backend stanza looks to have been formatted correctly.

We (Pure Storage) are happy for these changes to be adopted into the next RHOSP 17.1.x release.

Comment 29 errata-xmlrpc 2024-01-16 14:31:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 17.1.2 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:0209