Bug 2149283

Description of problem:

The command used was:

openstack overcloud update converge           --stack overcloud           --templates /usr/share/openstack-tripleo-heat-templates           -r /home/stack/templates/roles_data.yaml           -n /home/stack/templates/network_data.yaml           -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml           -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml           -e /home/stack/templates/cloudname-environment.yaml           -e /home/stack/containers-prepare-parameter.yaml           -e /home/stack/templates/enable-tls-environment.yaml           -e /home/stack/templates/extraconfig-environment.yaml           -e /home/stack/templates/inject-trust-anchor-hiera-environment.yaml           -e /home/stack/templates/network-environment.yaml           -e /home/stack/templates/octavia-environment.yaml           -e /home/stack/templates/scale-environment.yaml           -e /home/stack/templates/storage-environment.yaml           -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yam
~~~

Upgrading from 16.1 to 16.2 --> 16.2 upgrade Process described here: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html-single/keeping_red_hat_openstack_platform_updated/index#proc_finalizing-the-update_updating-overcloud


Error:

~~~
2022-11-26 11:56:51.854 746655 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout.
The action raised an exception [action_ex_id=17641340-359c-4902-9b7f-cf65275ee00f, msg='[Errno 13] Permission denied: '/var/lib/mistral/overcloud/ceph-ansible'', action_cls='<class 'mistral.actions.action_factory.DownloadConfigAction'>', attributes='{}', params='{'work_dir': '/var/lib/mistral/overcloud', 'container_config': 'overcloud-config'}']Overcloud Endpoint: https://185.71.177.214:13000
Overcloud Horizon Dashboard URL: https://xx.xx.xx.xx/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed with error
2022-11-26 11:58:55.734 746655 ERROR openstack [-] Overcloud configuration failed.: tripleoclient.exceptions.DeploymentError: Overcloud configuration failed.
2022-11-26 11:58:55.735 746655 INFO osc_lib.shell [-] END return value: 1
~~~


Permissions 
~~~
[root@director ~]# ls -larth /var/lib/mistral
total 80K
drwx------.  2 42430 42430   31 Jul 13  2020 .ssh
drwxr-xr-x.  3 42430 42430   78 Jul 13  2020 .novaclient
drwxr-xr-x.  2 42430 42430 4.0K Jul 17  2020 ansible_fact_cache
drwxr-xr-x. 11 42430 42430 4.0K Aug 17  2020 0cec11f8-427a-44f2-9b49-b5aeba886b74
drwxr-xr-x. 11 42430 42430 4.0K Aug 17  2020 0757f71e-1032-4fb3-a5e7-85f8c818476c
drwxr-xr-x. 11 42430 42430 4.0K Aug 17  2020 41cb7800-bd28-48e1-97c6-ecd243b02f07
drwxr-xr-x. 11 42430 42430 4.0K Aug 18  2020 51239b39-2fbb-43fa-9ffc-8691d5fae95a
-r--r--r--.  1 42430 42430  16K Nov 24 19:35 undercloud.conf
drwxrwx---. 11 42430 42430 4.0K Nov 25 08:34 9dbe6020-1f1c-49bb-a0b9-b4b97fa6ef53
drwxrwx---. 11 42430 42430 4.0K Nov 25 09:14 1900f9ff-b11b-464f-b427-4851f0e78712
drwxrwx---. 11 42430 42430 4.0K Nov 25 11:02 fccba2f1-129a-4532-a62b-776e528bd3fa
drwxrwx---. 11 42430 42430 4.0K Nov 25 11:22 6ef7af79-5c58-4f3e-bd58-0b4a3611fb87
drwxrwx---. 11 42430 42430 4.0K Nov 25 19:49 210f191d-0de2-46fe-b9e0-103bc3ff65a2
drwxrwx---. 10 42430 42430 4.0K Nov 25 22:10 8fe793b0-c421-4c81-808d-6132dcaf1274
drwxrwx---. 10 42430 42430 4.0K Nov 25 22:12 92b99385-15e9-4084-8654-16ce3782a3ab
drwxrwx---. 12 42430 42430 4.0K Nov 25 22:33 343bee8b-5001-4e53-9d52-005a0e1cede0
drwxrwx---. 11 42430 42430 4.0K Nov 25 22:36 b0613e32-85ba-4597-8767-943ad01c200a
lrwxrwxrwx.  1 42430 42430   26 Nov 25 22:53 config-download-latest -> /var/lib/mistral/overcloud
drwxr-x--x. 19 42430 42430 4.0K Nov 25 22:53 .
drwxrwx---.  3 42430 42430   53 Nov 26 10:06 overcloud
drwxr-xr-x. 97 root  root  4.0K Nov 28 08:47 ..
 [root@director ~]# ls -larth /var/lib/mistral/overcloud/ceph-ansible
total 6.9M
drwxr-x---. 2 tripleo-admin root     6 Nov 25 22:57 fetch_dir
lrwxrwxrwx. 1 root          root    57 Nov 25 22:57 inventory.yml -> /var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml
-rw-r--r--. 1 root          root   231 Nov 25 22:57 extra_vars.yml
-rw-r--r--. 1 root          root     2 Nov 25 22:57 nodes_uuid_data.json
-rw-r--r--. 1 root          root  1.1K Nov 25 22:57 nodes_uuid_playbook.yml
drwxr-x---. 2 tripleo-admin root   146 Nov 25 22:59 group_vars
drwxr-x---. 2 tripleo-admin root   223 Nov 25 22:59 host_vars
-rw-r--r--. 1 root          root   12K Nov 25 22:59 nodes_uuid_command.log
-rw-r-----. 1 root          root   789 Nov 25 22:59 create_ceph_ansible_remote_tmp.yml
-rw-r--r--. 1 root          root  2.7K Nov 25 22:59 create_ceph_ansible_remote_tmp.log
-rwxr-x---. 1 root          root  1.2K Nov 25 22:59 ceph_ansible_command.sh
drwxr-x---. 5 tripleo-admin root  4.0K Nov 25 22:59 .
-rw-r--r--. 1 root          root  6.9M Nov 25 23:03 ceph_ansible_command.log
drwxrwx---. 3         42430 42430   53 Nov 26 10:06 ..
[tripleo-admin@director ~]$ logout
 [root@director ~]# su - tripleo-admin
bash: 2022-11-22: command not found...
[tripleo-admin@director ~]$ ls -larth /var/lib/mistral/config-download-latest/ceph-ansible
ls: cannot access '/var/lib/mistral/config-download-latest/ceph-ansible': Permission denied
[tripleo-admin@director ~]$ ls -larth /var/lib/mistral/overcloud/ceph-ansible
ls: cannot access '/var/lib/mistral/overcloud/ceph-ansible': Permission denied
[tripleo-admin@director ~]$ 
~~~


Version-Release number of selected component (if applicable):

16.2.2

How reproducible:


Steps to Reproduce:
1. run openstack overcloud update converge
2.
3.

Actual results:

The action raised an exception  msg='[Errno 13] Permission denied: '/var/lib/mistral/overcloud/ceph-ansible''

Expected results:

No Error

Additional info:

- This BZ could be related to this other BZ https://bugzilla.redhat.com/show_bug.cgi?id=1824266

- SOS reports are available