Bug 2138184

Summary:	Re-deployment fails if octavia or ceph is enabled
Product:	Red Hat OpenStack	Reporter:	Takashi Kajinami <tkajinam>
Component:	openstack-tripleo-common	Assignee:	Takashi Kajinami <tkajinam>
Status:	CLOSED ERRATA	QA Contact:	David Rosenfeld <drosenfe>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	16.1 (Train)	CC:	alfrgarc, cjeanner, drosenfe, jelynch, jslagle, mburns, ramishra, slinaber, spower
Target Milestone:	z9	Keywords:	Triaged
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-common-11.4.1-1.20220926013655.75bd92a.el8ost	Doc Type:	Bug Fix
Doc Text:	RHSA-2022:6969 introduced the process to clean up files in the /var/lib/mistral directory in the undercloud but the process consistently failed when the Load-balancing service (octavia) or Red Hat Ceph Storage was enabled because these services created additional directories, which the cleanup process could not properly remove. Some deployment actions, such as scale out, consistently failed if the Load-balancing service or Ceph Storage was enabled. With this update, Mistral no longer executes the cleanup. Users must manually delete files if they want to enforce the reduced permission of the files in the /var/lib/mistral directory. Deployment actions no longer fail because of a permission error.	Story Points:	---
Clone Of:	2137484	Environment:
Last Closed:	2022-12-07 20:27:53 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2137484
Bug Blocks:

Description Takashi Kajinami 2022-10-27 14:20:04 UTC

+++ This bug was initially created as a clone of Bug #2137484 +++

Description of problem:

This was found in bz 2136393 initially.

Updating overcloud by running the deployment command fails because of the following workflow error if octavia or ceph is enabled.
~~~
Waiting for messages on queue 'tripleo' with no timeout.
The action raised an exception
[action_ex_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, msg='[Errno 13] Permission denied: 'local_dir'', action_cls='<class 'mistral.actions.action_factory.DownloadConfigAction'>', attributes='{}', params='{'work_dir': '/var/lib/mistral/overcloud', 'container_config':
'overcloud-config'}']
~~~

This is the regression caused by the fix for bz 2125078 .
The change introduced the step to purge files in /var/lib/mistral/<stack name> to enforce the proper permission
but the deployment tasks for octavia/ceph create files/directories owned by tripleo-admin in the directory
and the cleanup process fails with permission error.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Deploy overcloud with Octavia enabled
2. Run the same deployment command

Actual results:
The 2nd deployment fails because of the workflow error

Expected results:
The 2nd deployment should not fail.

Additional info:

Comment 7 David Rosenfeld 2022-11-09 15:04:29 UTC

The redeploy permission denied error is still seen in Phase 3 regression of RHOS-16.1-RHEL-8-20221108.n.1. Moving to on_dev although maybe the correct state is modified. 

While the undercloud has the correct openstack-tripleo-common package:

[stack@undercloud-0 ~]$ sudo yum list installed | grep openstack-tripleo-common
openstack-tripleo-common.noarch               11.4.1-1.20220926013655.75bd92a.el8ost           @rhelosp-16.1            
openstack-tripleo-common-containers.noarch    11.4.1-1.20220926013655.75bd92a.el8ost           @rhelosp-16.1

The mistral containers do not:

[stack@undercloud-0 ~]$ sudo podman exec mistral_api yum list installed | grep openstack-tripleo-common
2022-11-09 14:58:55,152 [ERROR] yum:1255:MainThread @logutil.py:194 - [Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' - Further logging output will be written to stderr
openstack-tripleo-common.noarch                11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1559188                                 
openstack-tripleo-common-container-base.noarch 11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1558923                                 
openstack-tripleo-common-containers.noarch     11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1559188

Comment 13 David Rosenfeld 2022-11-16 15:07:08 UTC

Using RHOS-16.1-RHEL-8-20221115.n.1 stack update is not working. Still not seeing updated package in mistral containers:

[stack@undercloud-0 ~]$ sudo yum list installed | grep openstack-tripleo-common
openstack-tripleo-common.noarch               11.4.1-1.20220926013655.75bd92a.el8ost           @rhelosp-16.1            
openstack-tripleo-common-containers.noarch    11.4.1-1.20220926013655.75bd92a.el8ost           @rhelosp-16.1 

          
[stack@undercloud-0 ~]$ sudo podman exec mistral_executor yum list installed | grep openstack-tripleo-common
2022-11-16 15:04:52,386 [ERROR] yum:11804:MainThread @logutil.py:194 - [Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' - Further logging output will be written to stderr
openstack-tripleo-common.noarch                11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1559188                                 
openstack-tripleo-common-container-base.noarch 11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1558923                                 
openstack-tripleo-common-containers.noarch     11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1559188

Comment 20 errata-xmlrpc 2022-12-07 20:27:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795