Bug 2138184 - Re-deployment fails if octavia or ceph is enabled
Summary: Re-deployment fails if octavia or ceph is enabled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z9
: 16.1 (Train on RHEL 8.2)
Assignee: Takashi Kajinami
QA Contact: David Rosenfeld
URL:
Whiteboard:
Depends On: 2137484
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-10-27 14:20 UTC by Takashi Kajinami
Modified: 2023-11-17 19:42 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-common-11.4.1-1.20220926013655.75bd92a.el8ost
Doc Type: Bug Fix
Doc Text:
RHSA-2022:6969 introduced the process to clean up files in the /var/lib/mistral directory in the undercloud but the process consistently failed when the Load-balancing service (octavia) or Red Hat Ceph Storage was enabled because these services created additional directories, which the cleanup process could not properly remove. Some deployment actions, such as scale out, consistently failed if the Load-balancing service or Ceph Storage was enabled. With this update, Mistral no longer executes the cleanup. Users must manually delete files if they want to enforce the reduced permission of the files in the /var/lib/mistral directory. Deployment actions no longer fail because of a permission error.
Clone Of: 2137484
Environment:
Last Closed: 2022-12-07 20:27:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 862556 0 None stable/train: MERGED tripleo-common: Train-only: Do not attempt to remove config-download files (Ib819c40862302065b6b52f68f0460f3d533d2194) 2022-11-10 15:16:37 UTC
Red Hat Issue Tracker OSP-19751 0 None None None 2022-10-27 14:22:55 UTC
Red Hat Product Errata RHBA-2022:8795 0 None None None 2022-12-07 20:28:31 UTC

Description Takashi Kajinami 2022-10-27 14:20:04 UTC
+++ This bug was initially created as a clone of Bug #2137484 +++

Description of problem:

This was found in bz 2136393 initially.

Updating overcloud by running the deployment command fails because of the following workflow error if octavia or ceph is enabled.
~~~
Waiting for messages on queue 'tripleo' with no timeout.
The action raised an exception
[action_ex_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, msg='[Errno 13] Permission denied: 'local_dir'', action_cls='<class 'mistral.actions.action_factory.DownloadConfigAction'>', attributes='{}', params='{'work_dir': '/var/lib/mistral/overcloud', 'container_config':
'overcloud-config'}']
~~~

This is the regression caused by the fix for bz 2125078 .
The change introduced the step to purge files in /var/lib/mistral/<stack name> to enforce the proper permission
but the deployment tasks for octavia/ceph create files/directories owned by tripleo-admin in the directory
and the cleanup process fails with permission error.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Deploy overcloud with Octavia enabled
2. Run the same deployment command

Actual results:
The 2nd deployment fails because of the workflow error

Expected results:
The 2nd deployment should not fail.

Additional info:

Comment 7 David Rosenfeld 2022-11-09 15:04:29 UTC
The redeploy permission denied error is still seen in Phase 3 regression of RHOS-16.1-RHEL-8-20221108.n.1. Moving to on_dev although maybe the correct state is modified. 

While the undercloud has the correct openstack-tripleo-common package:

[stack@undercloud-0 ~]$ sudo yum list installed | grep openstack-tripleo-common
openstack-tripleo-common.noarch               11.4.1-1.20220926013655.75bd92a.el8ost           @rhelosp-16.1            
openstack-tripleo-common-containers.noarch    11.4.1-1.20220926013655.75bd92a.el8ost           @rhelosp-16.1

The mistral containers do not:

[stack@undercloud-0 ~]$ sudo podman exec mistral_api yum list installed | grep openstack-tripleo-common
2022-11-09 14:58:55,152 [ERROR] yum:1255:MainThread @logutil.py:194 - [Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' - Further logging output will be written to stderr
openstack-tripleo-common.noarch                11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1559188                                 
openstack-tripleo-common-container-base.noarch 11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1558923                                 
openstack-tripleo-common-containers.noarch     11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1559188

Comment 13 David Rosenfeld 2022-11-16 15:07:08 UTC
Using RHOS-16.1-RHEL-8-20221115.n.1 stack update is not working. Still not seeing updated package in mistral containers:

[stack@undercloud-0 ~]$ sudo yum list installed | grep openstack-tripleo-common
openstack-tripleo-common.noarch               11.4.1-1.20220926013655.75bd92a.el8ost           @rhelosp-16.1            
openstack-tripleo-common-containers.noarch    11.4.1-1.20220926013655.75bd92a.el8ost           @rhelosp-16.1 

          
[stack@undercloud-0 ~]$ sudo podman exec mistral_executor yum list installed | grep openstack-tripleo-common
2022-11-16 15:04:52,386 [ERROR] yum:11804:MainThread @logutil.py:194 - [Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' - Further logging output will be written to stderr
openstack-tripleo-common.noarch                11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1559188                                 
openstack-tripleo-common-container-base.noarch 11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1558923                                 
openstack-tripleo-common-containers.noarch     11.4.1-1.20220926013654.75bd92a.el8ost           @odcs-1559188

Comment 20 errata-xmlrpc 2022-12-07 20:27:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795


Note You need to log in before you can comment on or make changes to this bug.