Description of problem:

Overcloud deployment fails to start nova due to permission issues:

2023-02-27T12:47:42.742857759+00:00 stderr F + echo 'Running command: '\''/var/lib/nova/delay-nova-compute --delay 180 --nova-binary /usr/bin/nova-compute '\'''
2023-02-27T12:47:42.742867419+00:00 stdout F Running command: '/var/lib/nova/delay-nova-compute --delay 180 --nova-binary /usr/bin/nova-compute '
2023-02-27T12:47:42.742892897+00:00 stderr F + umask 0022
2023-02-27T12:47:42.742921514+00:00 stderr F + exec /var/lib/nova/delay-nova-compute --delay 180 --nova-binary /usr/bin/nova-compute
2023-02-27T12:50:42.889091848+00:00 stderr F Traceback (most recent call last):
2023-02-27T12:50:42.889091848+00:00 stderr F   File "/var/lib/nova/delay-nova-compute", line 42, in <module>
2023-02-27T12:50:42.889280592+00:00 stderr F     open(args.state_file, 'a').close()
2023-02-27T12:50:42.889293622+00:00 stderr F PermissionError: [Errno 13] Permission denied: '/run/nova-compute-delayed'


Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230216.n.1 (worked in RHOS-17.1-RHEL-9-20230131.n.2)


How reproducible: always


Steps to Reproduce:
1. Deploy OSP with HCI


Actual results:
Overcloud deployment logs:

++ cat /run_command^M
+ CMD='/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py'^M
+ ARGS=^M
+ [[ ! -n '' ]]^M
+ . kolla_extend_start^M
+ echo 'Running command: '\''/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py'\'''^M
+ umask 0022^M
+ exec /container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py^M
+ command -v python3^M
+ python3 /container-config-scripts/nova_wait_for_compute_service.py
2023-02-27 12:57:55.305939 | 5254003d-657e-4874-3f17-00000000e373 |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_5 | compute-0 | error={"changed": false, "msg": "Failed containers: nova_wait_for_compute_service"}


Expected results: successful overcloud deployment


Additional info:
Job: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/osasinfra/view/shiftstack_ci/job/DFG-osasinfra-shiftstack_ci-osp_verification-osp17.1-passed_phase2-ocp4-stable/5/

Logs: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-osasinfra-shiftstack_ci-osp_verification-osp17.1-passed_phase2-ocp4-stable/5/

$ metalsmith list
+--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+
| UUID                                 | Node Name    | Allocation UUID                      | Hostname     | State  | IP Addresses           |
+--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+
| 51de2246-9d8e-444b-b08e-a98490e983c1 | compute-0    | 1779fd08-03c0-4697-8b26-b4981cf24e69 | compute-0    | ACTIVE | ctlplane=192.168.24.19 |
| 1c012626-7198-46d6-964e-7a6ae06ff9b9 | controller-0 | 989fbd0e-3d1a-438e-8306-7877f2ada048 | controller-0 | ACTIVE | ctlplane=192.168.24.39 |
| 18c4ed99-b61a-4d96-83ed-0af067164bdb | controller-1 | 934d956e-582b-4c4a-aae1-f97636bc268f | controller-1 | ACTIVE | ctlplane=192.168.24.48 |
| 2860fe61-034f-407d-bbe3-b88613016d2c | controller-2 | 32e95b33-a41c-4955-9284-5fb5b4a17d30 | controller-2 | ACTIVE | ctlplane=192.168.24.20 |
+--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+

Overcloud deployment command:
openstack overcloud deploy \
  --timeout 100 \
  --templates /usr/share/openstack-tripleo-heat-templates \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-mds.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
  --libvirt-type kvm \
  --stack overcloud \
  -r /home/stack/hybrid_templates/roles/roles_data.yaml \
  -e /home/stack/hybrid_templates/roles/nodes.yaml \
  --deployed-server \
  -e /home/stack/templates/overcloud-vip-deployed.yaml \
  -e /home/stack/templates/overcloud-networks-deployed.yaml \
  -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
  -e /home/stack/templates/overcloud-ceph-deployed.yaml \
  --disable-protected-resource-types \
  --networks-file /home/stack/hybrid_templates/network/network_data_v2.yaml \
-e /home/stack/hybrid_templates/titan27-storage-config.yml \
--networks-file /home/stack/hybrid_templates/network/network_data_v2.yaml \
-e /home/stack/hybrid_templates/network/network-environment_v2.yaml \
-e /home/stack/hybrid_templates/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \
-e /home/stack/hybrid_templates/custom-storage.yaml \
-e /home/stack/hybrid_templates/debug.yaml \
-e /home/stack/hybrid_templates/ntp_pool.yaml \
-e /home/stack/hybrid_templates/config_heat.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/hybrid_templates/manila-cephganesha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
-e /home/stack/hybrid_templates/cloud-names.yaml \
-e /home/stack/hybrid_templates/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
-e /home/stack/hybrid_templates/ipaservices-baremetal-ansible.yaml \
--log-file overcloud_deployment_84.log