Description of problem: The command used was: openstack overcloud update converge --stack overcloud --templates /usr/share/openstack-tripleo-heat-templates -r /home/stack/templates/roles_data.yaml -n /home/stack/templates/network_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml -e /home/stack/templates/cloudname-environment.yaml -e /home/stack/containers-prepare-parameter.yaml -e /home/stack/templates/enable-tls-environment.yaml -e /home/stack/templates/extraconfig-environment.yaml -e /home/stack/templates/inject-trust-anchor-hiera-environment.yaml -e /home/stack/templates/network-environment.yaml -e /home/stack/templates/octavia-environment.yaml -e /home/stack/templates/scale-environment.yaml -e /home/stack/templates/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yam ~~~ Upgrading from 16.1 to 16.2 --> 16.2 upgrade Process described here: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html-single/keeping_red_hat_openstack_platform_updated/index#proc_finalizing-the-update_updating-overcloud Error: ~~~ 2022-11-26 11:56:51.854 746655 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout. The action raised an exception [action_ex_id=17641340-359c-4902-9b7f-cf65275ee00f, msg='[Errno 13] Permission denied: '/var/lib/mistral/overcloud/ceph-ansible'', action_cls='<class 'mistral.actions.action_factory.DownloadConfigAction'>', attributes='{}', params='{'work_dir': '/var/lib/mistral/overcloud', 'container_config': 'overcloud-config'}']Overcloud Endpoint: https://185.71.177.214:13000 Overcloud Horizon Dashboard URL: https://xx.xx.xx.xx/dashboard Overcloud rc file: /home/stack/overcloudrc Overcloud Deployed with error 2022-11-26 11:58:55.734 746655 ERROR openstack [-] Overcloud configuration failed.: tripleoclient.exceptions.DeploymentError: Overcloud configuration failed. 2022-11-26 11:58:55.735 746655 INFO osc_lib.shell [-] END return value: 1 ~~~ Permissions ~~~ [root@director ~]# ls -larth /var/lib/mistral total 80K drwx------. 2 42430 42430 31 Jul 13 2020 .ssh drwxr-xr-x. 3 42430 42430 78 Jul 13 2020 .novaclient drwxr-xr-x. 2 42430 42430 4.0K Jul 17 2020 ansible_fact_cache drwxr-xr-x. 11 42430 42430 4.0K Aug 17 2020 0cec11f8-427a-44f2-9b49-b5aeba886b74 drwxr-xr-x. 11 42430 42430 4.0K Aug 17 2020 0757f71e-1032-4fb3-a5e7-85f8c818476c drwxr-xr-x. 11 42430 42430 4.0K Aug 17 2020 41cb7800-bd28-48e1-97c6-ecd243b02f07 drwxr-xr-x. 11 42430 42430 4.0K Aug 18 2020 51239b39-2fbb-43fa-9ffc-8691d5fae95a -r--r--r--. 1 42430 42430 16K Nov 24 19:35 undercloud.conf drwxrwx---. 11 42430 42430 4.0K Nov 25 08:34 9dbe6020-1f1c-49bb-a0b9-b4b97fa6ef53 drwxrwx---. 11 42430 42430 4.0K Nov 25 09:14 1900f9ff-b11b-464f-b427-4851f0e78712 drwxrwx---. 11 42430 42430 4.0K Nov 25 11:02 fccba2f1-129a-4532-a62b-776e528bd3fa drwxrwx---. 11 42430 42430 4.0K Nov 25 11:22 6ef7af79-5c58-4f3e-bd58-0b4a3611fb87 drwxrwx---. 11 42430 42430 4.0K Nov 25 19:49 210f191d-0de2-46fe-b9e0-103bc3ff65a2 drwxrwx---. 10 42430 42430 4.0K Nov 25 22:10 8fe793b0-c421-4c81-808d-6132dcaf1274 drwxrwx---. 10 42430 42430 4.0K Nov 25 22:12 92b99385-15e9-4084-8654-16ce3782a3ab drwxrwx---. 12 42430 42430 4.0K Nov 25 22:33 343bee8b-5001-4e53-9d52-005a0e1cede0 drwxrwx---. 11 42430 42430 4.0K Nov 25 22:36 b0613e32-85ba-4597-8767-943ad01c200a lrwxrwxrwx. 1 42430 42430 26 Nov 25 22:53 config-download-latest -> /var/lib/mistral/overcloud drwxr-x--x. 19 42430 42430 4.0K Nov 25 22:53 . drwxrwx---. 3 42430 42430 53 Nov 26 10:06 overcloud drwxr-xr-x. 97 root root 4.0K Nov 28 08:47 .. [root@director ~]# ls -larth /var/lib/mistral/overcloud/ceph-ansible total 6.9M drwxr-x---. 2 tripleo-admin root 6 Nov 25 22:57 fetch_dir lrwxrwxrwx. 1 root root 57 Nov 25 22:57 inventory.yml -> /var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml -rw-r--r--. 1 root root 231 Nov 25 22:57 extra_vars.yml -rw-r--r--. 1 root root 2 Nov 25 22:57 nodes_uuid_data.json -rw-r--r--. 1 root root 1.1K Nov 25 22:57 nodes_uuid_playbook.yml drwxr-x---. 2 tripleo-admin root 146 Nov 25 22:59 group_vars drwxr-x---. 2 tripleo-admin root 223 Nov 25 22:59 host_vars -rw-r--r--. 1 root root 12K Nov 25 22:59 nodes_uuid_command.log -rw-r-----. 1 root root 789 Nov 25 22:59 create_ceph_ansible_remote_tmp.yml -rw-r--r--. 1 root root 2.7K Nov 25 22:59 create_ceph_ansible_remote_tmp.log -rwxr-x---. 1 root root 1.2K Nov 25 22:59 ceph_ansible_command.sh drwxr-x---. 5 tripleo-admin root 4.0K Nov 25 22:59 . -rw-r--r--. 1 root root 6.9M Nov 25 23:03 ceph_ansible_command.log drwxrwx---. 3 42430 42430 53 Nov 26 10:06 .. [tripleo-admin@director ~]$ logout [root@director ~]# su - tripleo-admin bash: 2022-11-22: command not found... [tripleo-admin@director ~]$ ls -larth /var/lib/mistral/config-download-latest/ceph-ansible ls: cannot access '/var/lib/mistral/config-download-latest/ceph-ansible': Permission denied [tripleo-admin@director ~]$ ls -larth /var/lib/mistral/overcloud/ceph-ansible ls: cannot access '/var/lib/mistral/overcloud/ceph-ansible': Permission denied [tripleo-admin@director ~]$ ~~~ Version-Release number of selected component (if applicable): 16.2.2 How reproducible: Steps to Reproduce: 1. run openstack overcloud update converge 2. 3. Actual results: The action raised an exception msg='[Errno 13] Permission denied: '/var/lib/mistral/overcloud/ceph-ansible'' Expected results: No Error Additional info: - This BZ could be related to this other BZ https://bugzilla.redhat.com/show_bug.cgi?id=1824266 - SOS reports are available
Yes, this is indeed a duplicate of the bug you mentioned. Since you have raised it against a different version to the other one, I'll leave it up to the storage team to triage and decide how they want to handle it.
I believe this is duplicate of bz 2137484 instead of bz 1824266 . As I suggested in the private comment in the bug, downgrading the mistral-engine container in undercloud and moving the /var/lib/mistral/overcloud directory would be the easiest workaround for now.
*** This bug has been marked as a duplicate of bug 2137484 ***