Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2250696

Summary: Overcloud deployment fails starting nova in HCI deployment
Product: Red Hat OpenStack Reporter: Jon Uriarte <juriarte>
Component: openstack-tripleo-heat-templatesAssignee: OSP Team <rhos-maint>
Status: CLOSED DUPLICATE QA Contact: Joe H. Rahme <jhakimra>
Severity: high Docs Contact:
Priority: unspecified    
Version: 17.1 (Wallaby)CC: bshephar, mburns
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-21 09:12:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jon Uriarte 2023-11-20 15:34:19 UTC
Description of problem:

Overcloud deployment fails when starting nova compute.

Nova compute logs:
2023-11-18T00:58:40.583423942+00:00 stderr F + sudo -E kolla_set_configs
2023-11-18T00:58:41.009294610+00:00 stderr F INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
2023-11-18T00:58:41.009471400+00:00 stderr F INFO:__main__:Validating config file
2023-11-18T00:58:41.009525405+00:00 stderr F INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
2023-11-18T00:58:41.009557722+00:00 stderr F INFO:__main__:Copying service configuration files
2023-11-18T00:58:41.010299371+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/auth.conf to /etc/libvirt/auth.conf
2023-11-18T00:58:41.010684148+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/passwd.db to /etc/libvirt/passwd.db
2023-11-18T00:58:41.010965782+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/qemu.conf
2023-11-18T00:58:41.011075149+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/qemu.conf to /etc/libvirt/qemu.conf
2023-11-18T00:58:41.011382093+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtlogd.conf
2023-11-18T00:58:41.011447237+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtlogd.conf to /etc/libvirt/virtlogd.conf
2023-11-18T00:58:41.011716963+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtnodedevd.conf
2023-11-18T00:58:41.011780084+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtnodedevd.conf to /etc/libvirt/virtnodedevd.conf
2023-11-18T00:58:41.012060695+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtproxyd.conf
2023-11-18T00:58:41.012121528+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtproxyd.conf to /etc/libvirt/virtproxyd.conf
2023-11-18T00:58:41.012406361+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtqemud.conf
2023-11-18T00:58:41.012467513+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtqemud.conf to /etc/libvirt/virtqemud.conf
2023-11-18T00:58:41.012751013+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtsecretd.conf
2023-11-18T00:58:41.012812259+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtsecretd.conf to /etc/libvirt/virtsecretd.conf
2023-11-18T00:58:41.013219532+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtstoraged.conf
2023-11-18T00:58:41.013329408+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtstoraged.conf to /etc/libvirt/virtstoraged.conf
2023-11-18T00:58:41.013935617+00:00 stderr F INFO:__main__:Deleting /etc/nova/migration/authorized_keys
2023-11-18T00:58:41.014002524+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/authorized_keys to /etc/nova/migration/authorized_keys
2023-11-18T00:58:41.014282169+00:00 stderr F INFO:__main__:Deleting /etc/nova/migration/identity
2023-11-18T00:58:41.014357416+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/identity to /etc/nova/migration/identity
2023-11-18T00:58:41.014636106+00:00 stderr F INFO:__main__:Creating directory /etc/nova/provider_config
2023-11-18T00:58:41.014820723+00:00 stderr F INFO:__main__:Deleting /etc/nova/nova.conf
2023-11-18T00:58:41.014882992+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/nova.conf to /etc/nova/nova.conf
2023-11-18T00:58:41.015267140+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/secret.xml to /etc/nova/secret.xml
2023-11-18T00:58:41.015677789+00:00 stderr F INFO:__main__:Deleting /etc/sasl2/libvirt.conf
2023-11-18T00:58:41.015742202+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sasl2/libvirt.conf to /etc/sasl2/libvirt.conf
2023-11-18T00:58:41.016095585+00:00 stderr F INFO:__main__:Deleting /etc/ssh/sshd_config
2023-11-18T00:58:41.016158442+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ssh/sshd_config to /etc/ssh/sshd_config
2023-11-18T00:58:41.016574894+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/libvirtd to /etc/sysconfig/libvirtd
2023-11-18T00:58:41.016819867+00:00 stderr F INFO:__main__:Deleting /etc/login.defs
2023-11-18T00:58:41.016885316+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/login.defs to /etc/login.defs
2023-11-18T00:58:41.017521159+00:00 stderr F INFO:__main__:Deleting /var/lib/nova/.ssh/config
2023-11-18T00:58:41.017588442+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/var/lib/nova/.ssh/config to /var/lib/nova/.ssh/config
2023-11-18T00:58:41.017928214+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-iscsid/initiatorname.iscsi to /etc/iscsi/initiatorname.iscsi
2023-11-18T00:58:41.018245865+00:00 stderr F INFO:__main__:Deleting /etc/iscsi/iscsid.conf
2023-11-18T00:58:41.018311255+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-iscsid/iscsid.conf to /etc/iscsi/iscsid.conf
2023-11-18T00:58:41.018863037+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.openstack.keyring to /etc/ceph/ceph.client.openstack.keyring
2023-11-18T00:58:41.019120831+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.radosgw.keyring to /etc/ceph/ceph.client.radosgw.keyring
2023-11-18T00:58:41.019429162+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.manila.keyring to /etc/ceph/ceph.client.manila.keyring
2023-11-18T00:58:41.019681269+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.conf to /etc/ceph/ceph.conf
2023-11-18T00:58:41.019921265+00:00 stderr F INFO:__main__:Writing out command to execute
2023-11-18T00:58:41.020408852+00:00 stderr F INFO:__main__:Setting permission for /var/log/nova
2023-11-18T00:58:41.020555060+00:00 stderr F INFO:__main__:Setting permission for /etc/ceph/ceph.client.openstack.keyring
2023-11-18T00:58:41.028566402+00:00 stderr F ++ cat /run_command
2023-11-18T00:58:41.030303711+00:00 stderr F + CMD='/var/lib/nova/delay-nova-compute --delay 180 --nova-binary '
2023-11-18T00:58:41.030332441+00:00 stderr F + ARGS=
2023-11-18T00:58:41.030359245+00:00 stderr F + [[ ! -n '' ]]
2023-11-18T00:58:41.030368270+00:00 stderr F + . kolla_extend_start
2023-11-18T00:58:41.030481065+00:00 stderr F + echo 'Running command: '\''/var/lib/nova/delay-nova-compute --delay 180 --nova-binary '\'''
2023-11-18T00:58:41.030490399+00:00 stdout F Running command: '/var/lib/nova/delay-nova-compute --delay 180 --nova-binary '
2023-11-18T00:58:41.030504471+00:00 stderr F + umask 0022
2023-11-18T00:58:41.030521965+00:00 stderr F + exec /var/lib/nova/delay-nova-compute --delay 180 --nova-binary
2023-11-18T00:58:41.075899514+00:00 stderr F usage: delay-nova-compute [-h] [--config-file NOVA_CONFIG]
2023-11-18T00:58:41.075899514+00:00 stderr F                           [--nova-binary NOVA_BINARY] [--delay DELAY]
2023-11-18T00:58:41.075899514+00:00 stderr F                           [--state-dir STATE_DIR] [--state-file STATE_FILE]
2023-11-18T00:58:41.075992245+00:00 stderr F delay-nova-compute: error: argument --nova-binary: expected one argument


Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20231117.n.1


How reproducible: always


Steps to Reproduce:
1. Deploy OSP with HCI

Actual results:

[...]
2023-11-18 01:08:53.476046 |                                      |    WARNING | ERROR: Can't run container nova_wait_for_compute_service^M
stderr: + sudo -E kolla_set_configs^M
sudo: unable to send audit message: Operation not permitted^M
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json^M
INFO:__main__:Validating config file^M
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS^M
INFO:__main__:Copying service configuration files^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/auth.conf to /etc/libvirt/auth.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/passwd.db to /etc/libvirt/passwd.db^M
INFO:__main__:Deleting /etc/libvirt/qemu.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/qemu.conf to /etc/libvirt/qemu.conf^M
INFO:__main__:Deleting /etc/libvirt/virtlogd.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtlogd.conf to /etc/libvirt/virtlogd.conf^M
INFO:__main__:Deleting /etc/libvirt/virtnodedevd.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtnodedevd.conf to /etc/libvirt/virtnodedevd.conf^M
INFO:__main__:Deleting /etc/libvirt/virtproxyd.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtproxyd.conf to /etc/libvirt/virtproxyd.conf^M
INFO:__main__:Deleting /etc/libvirt/virtqemud.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtqemud.conf to /etc/libvirt/virtqemud.conf^M
INFO:__main__:Deleting /etc/libvirt/virtsecretd.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtsecretd.conf to /etc/libvirt/virtsecretd.conf^M
INFO:__main__:Deleting /etc/libvirt/virtstoraged.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtstoraged.conf to /etc/libvirt/virtstoraged.conf^M
INFO:__main__:Deleting /etc/nova/migration/authorized_keys^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/authorized_keys to /etc/nova/migration/authorized_keys^M
INFO:__main__:Deleting /etc/nova/migration/identity^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/identity to /etc/nova/migration/identity^M
INFO:__main__:Creating directory /etc/nova/provider_config^M
INFO:__main__:Deleting /etc/nova/nova.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/nova.conf to /etc/nova/nova.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/secret.xml to /etc/nova/secret.xml^M
INFO:__main__:Deleting /etc/sasl2/libvirt.conf^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sasl2/libvirt.conf to /etc/sasl2/libvirt.conf^M
INFO:__main__:Deleting /etc/ssh/sshd_config^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ssh/sshd_config to /etc/ssh/sshd_config^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/libvirtd to /etc/sysconfig/libvirtd^M
INFO:__main__:Deleting /etc/login.defs^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/login.defs to /etc/login.defs^M
INFO:__main__:Deleting /var/lib/nova/.ssh/config^M
INFO:__main__:Copying /var/lib/kolla/config_files/src/var/lib/nova/.ssh/config to /var/lib/nova/.ssh/config^M
INFO:__main__:Writing out command to execute^M
INFO:__main__:Setting permission for /var/log/nova^M
++ cat /run_command^M
+ CMD='/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py'^M
+ ARGS=^M
+ [[ ! -n '' ]]^M
+ . kolla_extend_start^M
+ echo 'Running command: '\''/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py'\'''^M
+ umask 0022^M
+ exec /container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py^M
+ command -v python3^M
+ python3 /container-config-scripts/nova_wait_for_compute_service.py
2023-11-18 01:08:53.478665 | 52540014-75de-ec22-69df-00000000d34b |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_5 | compute-0 | error={"changed": false, "msg": "Failed containers: nova_wait_for_compute_service"}
[...]


Expected results: successful overcloud deployment


Additional info:

$ metalsmith list
+--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+
| UUID                                 | Node Name    | Allocation UUID                      | Hostname     | State  | IP Addresses           |
+--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+
| 52cbc39c-ce68-4244-b7b8-95d33a03dbeb | compute-0    | a94e5178-c4d6-4bf3-9187-015aeabd36f8 | compute-0    | ACTIVE | ctlplane=192.168.24.36 |
| 091d01cb-d9f1-4239-a18c-62fd0d22eb00 | controller-0 | 2b8a5b47-aad6-41a9-8924-7f10a7efc685 | controller-0 | ACTIVE | ctlplane=192.168.24.24 |
| fa112387-e184-483d-8e7c-c46090ee0e87 | controller-1 | f2ba111b-8f97-45ce-a69b-c2c597c578a9 | controller-1 | ACTIVE | ctlplane=192.168.24.13 |
| a8169a9a-e12c-46e7-b26a-9cb2fa4ae10a | controller-2 | e62858ed-07d9-4c85-95f1-ec85dc6dcc67 | controller-2 | ACTIVE | ctlplane=192.168.24.50 |
+--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+

$ cat overcloud_deploy.sh 
#!/bin/bash

openstack overcloud deploy \
  --timeout 100 \
  --templates /usr/share/openstack-tripleo-heat-templates \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-mds.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
  --libvirt-type kvm \
  --stack overcloud \
  -r /home/stack/hybrid_templates/roles/roles_data.yaml \
  -e /home/stack/hybrid_templates/roles/nodes.yaml \
  --deployed-server \
  -e /home/stack/templates/overcloud-vip-deployed.yaml \
  -e /home/stack/templates/overcloud-networks-deployed.yaml \
  -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
  -e /home/stack/templates/overcloud-ceph-deployed.yaml \
  --disable-protected-resource-types \
  --networks-file /home/stack/hybrid_templates/network/network_data_v2.yaml \
-e /home/stack/hybrid_templates/titan35-storage-config.yml \
--networks-file /home/stack/hybrid_templates/network/network_data_v2.yaml \
-e /home/stack/hybrid_templates/network/network-environment_v2.yaml \
-e /home/stack/hybrid_templates/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \
-e /home/stack/hybrid_templates/custom-storage.yaml \
-e /home/stack/hybrid_templates/debug.yaml \
-e /home/stack/hybrid_templates/ntp_pool.yaml \
-e /home/stack/hybrid_templates/config_heat.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/hybrid_templates/manila-cephganesha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
-e /home/stack/hybrid_templates/cloud-names.yaml \
-e /home/stack/hybrid_templates/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
-e /home/stack/hybrid_templates/ipaservices-baremetal-ansible.yaml \
--log-file overcloud_deployment_89.log

Comment 2 Brendan Shephard 2023-11-21 00:02:52 UTC
So, it looks like the issue comes from here:
https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/deployment/nova/nova-compute-container-puppet.yaml#L1291

I think you probably actually want it to be like this right?:
                    template: '/var/lib/nova/delay-nova-compute --delay DELAY --nova-binary /usr/bin/nova-compute'


Moving over to dfg:Compute to triage and verify if my suggestion is indeed what we need here.

Comment 4 Jon Uriarte 2023-11-21 09:12:09 UTC
Look like this is a duplicate of bug 2250187, so I'm closing it.
Thanks Brendan for your help!

*** This bug has been marked as a duplicate of bug 2250187 ***