Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2173939

Summary: Overcloud deployment fails starting nova due to file permission
Product: Red Hat OpenStack Reporter: Jon Uriarte <juriarte>
Component: openstack-tripleo-heat-templatesAssignee: Bogdan Dobrelya <bdobreli>
Status: CLOSED ERRATA QA Contact: Joe H. Rahme <jhakimra>
Severity: high Docs Contact:
Priority: high    
Version: 17.1 (Wallaby)CC: alfrgarc, alifshit, bbowen, bdobreli, ekuris, gfidente, mburns, mdemaced, mkrcmari, pgrist, prgutier, skovili, smooney, tkajinam
Target Milestone: rcKeywords: AutomationBlocker, Triaged
Target Release: 17.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-14.3.1-1.20230519151015.el9ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-16 01:13:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1498621, 2174346    

Description Jon Uriarte 2023-02-28 13:21:51 UTC
Description of problem:

Overcloud deployment fails to start nova due to permission issues:

2023-02-27T12:47:42.742857759+00:00 stderr F + echo 'Running command: '\''/var/lib/nova/delay-nova-compute --delay 180 --nova-binary /usr/bin/nova-compute '\'''
2023-02-27T12:47:42.742867419+00:00 stdout F Running command: '/var/lib/nova/delay-nova-compute --delay 180 --nova-binary /usr/bin/nova-compute '
2023-02-27T12:47:42.742892897+00:00 stderr F + umask 0022
2023-02-27T12:47:42.742921514+00:00 stderr F + exec /var/lib/nova/delay-nova-compute --delay 180 --nova-binary /usr/bin/nova-compute
2023-02-27T12:50:42.889091848+00:00 stderr F Traceback (most recent call last):
2023-02-27T12:50:42.889091848+00:00 stderr F   File "/var/lib/nova/delay-nova-compute", line 42, in <module>
2023-02-27T12:50:42.889280592+00:00 stderr F     open(args.state_file, 'a').close()
2023-02-27T12:50:42.889293622+00:00 stderr F PermissionError: [Errno 13] Permission denied: '/run/nova-compute-delayed'


Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230216.n.1 (worked in RHOS-17.1-RHEL-9-20230131.n.2)


How reproducible: always


Steps to Reproduce:
1. Deploy OSP with HCI


Actual results:
Overcloud deployment logs:

++ cat /run_command^M
+ CMD='/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py'^M
+ ARGS=^M
+ [[ ! -n '' ]]^M
+ . kolla_extend_start^M
+ echo 'Running command: '\''/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py'\'''^M
+ umask 0022^M
+ exec /container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py^M
+ command -v python3^M
+ python3 /container-config-scripts/nova_wait_for_compute_service.py
2023-02-27 12:57:55.305939 | 5254003d-657e-4874-3f17-00000000e373 |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_5 | compute-0 | error={"changed": false, "msg": "Failed containers: nova_wait_for_compute_service"}


Expected results: successful overcloud deployment


Additional info:
Job: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/osasinfra/view/shiftstack_ci/job/DFG-osasinfra-shiftstack_ci-osp_verification-osp17.1-passed_phase2-ocp4-stable/5/

Logs: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-osasinfra-shiftstack_ci-osp_verification-osp17.1-passed_phase2-ocp4-stable/5/

$ metalsmith list
+--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+
| UUID                                 | Node Name    | Allocation UUID                      | Hostname     | State  | IP Addresses           |
+--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+
| 51de2246-9d8e-444b-b08e-a98490e983c1 | compute-0    | 1779fd08-03c0-4697-8b26-b4981cf24e69 | compute-0    | ACTIVE | ctlplane=192.168.24.19 |
| 1c012626-7198-46d6-964e-7a6ae06ff9b9 | controller-0 | 989fbd0e-3d1a-438e-8306-7877f2ada048 | controller-0 | ACTIVE | ctlplane=192.168.24.39 |
| 18c4ed99-b61a-4d96-83ed-0af067164bdb | controller-1 | 934d956e-582b-4c4a-aae1-f97636bc268f | controller-1 | ACTIVE | ctlplane=192.168.24.48 |
| 2860fe61-034f-407d-bbe3-b88613016d2c | controller-2 | 32e95b33-a41c-4955-9284-5fb5b4a17d30 | controller-2 | ACTIVE | ctlplane=192.168.24.20 |
+--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+

Overcloud deployment command:
openstack overcloud deploy \
  --timeout 100 \
  --templates /usr/share/openstack-tripleo-heat-templates \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-mds.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
  --libvirt-type kvm \
  --stack overcloud \
  -r /home/stack/hybrid_templates/roles/roles_data.yaml \
  -e /home/stack/hybrid_templates/roles/nodes.yaml \
  --deployed-server \
  -e /home/stack/templates/overcloud-vip-deployed.yaml \
  -e /home/stack/templates/overcloud-networks-deployed.yaml \
  -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
  -e /home/stack/templates/overcloud-ceph-deployed.yaml \
  --disable-protected-resource-types \
  --networks-file /home/stack/hybrid_templates/network/network_data_v2.yaml \
-e /home/stack/hybrid_templates/titan27-storage-config.yml \
--networks-file /home/stack/hybrid_templates/network/network_data_v2.yaml \
-e /home/stack/hybrid_templates/network/network-environment_v2.yaml \
-e /home/stack/hybrid_templates/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \
-e /home/stack/hybrid_templates/custom-storage.yaml \
-e /home/stack/hybrid_templates/debug.yaml \
-e /home/stack/hybrid_templates/ntp_pool.yaml \
-e /home/stack/hybrid_templates/config_heat.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/hybrid_templates/manila-cephganesha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
-e /home/stack/hybrid_templates/cloud-names.yaml \
-e /home/stack/hybrid_templates/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
-e /home/stack/hybrid_templates/ipaservices-baremetal-ansible.yaml \
--log-file overcloud_deployment_84.log

Comment 6 Bogdan Dobrelya 2023-04-11 14:17:38 UTC
submitted downstream patches before upstream merged (low cores attention, faily unrelated upstream CI jobs)

Comment 7 Marian Krcmarik 2023-04-19 12:43:44 UTC
*** Bug 2182545 has been marked as a duplicate of this bug. ***

Comment 8 Bogdan Dobrelya 2023-04-25 15:47:46 UTC
merged dowsntream for 17.1 trunk patches

Comment 9 Bogdan Dobrelya 2023-05-04 12:11:00 UTC
Reverted downstream

Comment 10 Bogdan Dobrelya 2023-05-04 13:29:52 UTC
Updated the upstream patch

Comment 38 errata-xmlrpc 2023-08-16 01:13:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577