Bug 2250696
| Summary: | Overcloud deployment fails starting nova in HCI deployment | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jon Uriarte <juriarte> |
| Component: | openstack-tripleo-heat-templates | Assignee: | OSP Team <rhos-maint> |
| Status: | CLOSED DUPLICATE | QA Contact: | Joe H. Rahme <jhakimra> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 17.1 (Wallaby) | CC: | bshephar, mburns |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-11-21 09:12:09 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
So, it looks like the issue comes from here: https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/deployment/nova/nova-compute-container-puppet.yaml#L1291 I think you probably actually want it to be like this right?: template: '/var/lib/nova/delay-nova-compute --delay DELAY --nova-binary /usr/bin/nova-compute' Moving over to dfg:Compute to triage and verify if my suggestion is indeed what we need here. Look like this is a duplicate of bug 2250187, so I'm closing it. Thanks Brendan for your help! *** This bug has been marked as a duplicate of bug 2250187 *** |
Description of problem: Overcloud deployment fails when starting nova compute. Nova compute logs: 2023-11-18T00:58:40.583423942+00:00 stderr F + sudo -E kolla_set_configs 2023-11-18T00:58:41.009294610+00:00 stderr F INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json 2023-11-18T00:58:41.009471400+00:00 stderr F INFO:__main__:Validating config file 2023-11-18T00:58:41.009525405+00:00 stderr F INFO:__main__:Kolla config strategy set to: COPY_ALWAYS 2023-11-18T00:58:41.009557722+00:00 stderr F INFO:__main__:Copying service configuration files 2023-11-18T00:58:41.010299371+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/auth.conf to /etc/libvirt/auth.conf 2023-11-18T00:58:41.010684148+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/passwd.db to /etc/libvirt/passwd.db 2023-11-18T00:58:41.010965782+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/qemu.conf 2023-11-18T00:58:41.011075149+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/qemu.conf to /etc/libvirt/qemu.conf 2023-11-18T00:58:41.011382093+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtlogd.conf 2023-11-18T00:58:41.011447237+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtlogd.conf to /etc/libvirt/virtlogd.conf 2023-11-18T00:58:41.011716963+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtnodedevd.conf 2023-11-18T00:58:41.011780084+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtnodedevd.conf to /etc/libvirt/virtnodedevd.conf 2023-11-18T00:58:41.012060695+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtproxyd.conf 2023-11-18T00:58:41.012121528+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtproxyd.conf to /etc/libvirt/virtproxyd.conf 2023-11-18T00:58:41.012406361+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtqemud.conf 2023-11-18T00:58:41.012467513+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtqemud.conf to /etc/libvirt/virtqemud.conf 2023-11-18T00:58:41.012751013+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtsecretd.conf 2023-11-18T00:58:41.012812259+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtsecretd.conf to /etc/libvirt/virtsecretd.conf 2023-11-18T00:58:41.013219532+00:00 stderr F INFO:__main__:Deleting /etc/libvirt/virtstoraged.conf 2023-11-18T00:58:41.013329408+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtstoraged.conf to /etc/libvirt/virtstoraged.conf 2023-11-18T00:58:41.013935617+00:00 stderr F INFO:__main__:Deleting /etc/nova/migration/authorized_keys 2023-11-18T00:58:41.014002524+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/authorized_keys to /etc/nova/migration/authorized_keys 2023-11-18T00:58:41.014282169+00:00 stderr F INFO:__main__:Deleting /etc/nova/migration/identity 2023-11-18T00:58:41.014357416+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/identity to /etc/nova/migration/identity 2023-11-18T00:58:41.014636106+00:00 stderr F INFO:__main__:Creating directory /etc/nova/provider_config 2023-11-18T00:58:41.014820723+00:00 stderr F INFO:__main__:Deleting /etc/nova/nova.conf 2023-11-18T00:58:41.014882992+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/nova.conf to /etc/nova/nova.conf 2023-11-18T00:58:41.015267140+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/secret.xml to /etc/nova/secret.xml 2023-11-18T00:58:41.015677789+00:00 stderr F INFO:__main__:Deleting /etc/sasl2/libvirt.conf 2023-11-18T00:58:41.015742202+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sasl2/libvirt.conf to /etc/sasl2/libvirt.conf 2023-11-18T00:58:41.016095585+00:00 stderr F INFO:__main__:Deleting /etc/ssh/sshd_config 2023-11-18T00:58:41.016158442+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ssh/sshd_config to /etc/ssh/sshd_config 2023-11-18T00:58:41.016574894+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/libvirtd to /etc/sysconfig/libvirtd 2023-11-18T00:58:41.016819867+00:00 stderr F INFO:__main__:Deleting /etc/login.defs 2023-11-18T00:58:41.016885316+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/login.defs to /etc/login.defs 2023-11-18T00:58:41.017521159+00:00 stderr F INFO:__main__:Deleting /var/lib/nova/.ssh/config 2023-11-18T00:58:41.017588442+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/var/lib/nova/.ssh/config to /var/lib/nova/.ssh/config 2023-11-18T00:58:41.017928214+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-iscsid/initiatorname.iscsi to /etc/iscsi/initiatorname.iscsi 2023-11-18T00:58:41.018245865+00:00 stderr F INFO:__main__:Deleting /etc/iscsi/iscsid.conf 2023-11-18T00:58:41.018311255+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-iscsid/iscsid.conf to /etc/iscsi/iscsid.conf 2023-11-18T00:58:41.018863037+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.openstack.keyring to /etc/ceph/ceph.client.openstack.keyring 2023-11-18T00:58:41.019120831+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.radosgw.keyring to /etc/ceph/ceph.client.radosgw.keyring 2023-11-18T00:58:41.019429162+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.manila.keyring to /etc/ceph/ceph.client.manila.keyring 2023-11-18T00:58:41.019681269+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.conf to /etc/ceph/ceph.conf 2023-11-18T00:58:41.019921265+00:00 stderr F INFO:__main__:Writing out command to execute 2023-11-18T00:58:41.020408852+00:00 stderr F INFO:__main__:Setting permission for /var/log/nova 2023-11-18T00:58:41.020555060+00:00 stderr F INFO:__main__:Setting permission for /etc/ceph/ceph.client.openstack.keyring 2023-11-18T00:58:41.028566402+00:00 stderr F ++ cat /run_command 2023-11-18T00:58:41.030303711+00:00 stderr F + CMD='/var/lib/nova/delay-nova-compute --delay 180 --nova-binary ' 2023-11-18T00:58:41.030332441+00:00 stderr F + ARGS= 2023-11-18T00:58:41.030359245+00:00 stderr F + [[ ! -n '' ]] 2023-11-18T00:58:41.030368270+00:00 stderr F + . kolla_extend_start 2023-11-18T00:58:41.030481065+00:00 stderr F + echo 'Running command: '\''/var/lib/nova/delay-nova-compute --delay 180 --nova-binary '\''' 2023-11-18T00:58:41.030490399+00:00 stdout F Running command: '/var/lib/nova/delay-nova-compute --delay 180 --nova-binary ' 2023-11-18T00:58:41.030504471+00:00 stderr F + umask 0022 2023-11-18T00:58:41.030521965+00:00 stderr F + exec /var/lib/nova/delay-nova-compute --delay 180 --nova-binary 2023-11-18T00:58:41.075899514+00:00 stderr F usage: delay-nova-compute [-h] [--config-file NOVA_CONFIG] 2023-11-18T00:58:41.075899514+00:00 stderr F [--nova-binary NOVA_BINARY] [--delay DELAY] 2023-11-18T00:58:41.075899514+00:00 stderr F [--state-dir STATE_DIR] [--state-file STATE_FILE] 2023-11-18T00:58:41.075992245+00:00 stderr F delay-nova-compute: error: argument --nova-binary: expected one argument Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20231117.n.1 How reproducible: always Steps to Reproduce: 1. Deploy OSP with HCI Actual results: [...] 2023-11-18 01:08:53.476046 | | WARNING | ERROR: Can't run container nova_wait_for_compute_service^M stderr: + sudo -E kolla_set_configs^M sudo: unable to send audit message: Operation not permitted^M INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json^M INFO:__main__:Validating config file^M INFO:__main__:Kolla config strategy set to: COPY_ALWAYS^M INFO:__main__:Copying service configuration files^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/auth.conf to /etc/libvirt/auth.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/passwd.db to /etc/libvirt/passwd.db^M INFO:__main__:Deleting /etc/libvirt/qemu.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/qemu.conf to /etc/libvirt/qemu.conf^M INFO:__main__:Deleting /etc/libvirt/virtlogd.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtlogd.conf to /etc/libvirt/virtlogd.conf^M INFO:__main__:Deleting /etc/libvirt/virtnodedevd.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtnodedevd.conf to /etc/libvirt/virtnodedevd.conf^M INFO:__main__:Deleting /etc/libvirt/virtproxyd.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtproxyd.conf to /etc/libvirt/virtproxyd.conf^M INFO:__main__:Deleting /etc/libvirt/virtqemud.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtqemud.conf to /etc/libvirt/virtqemud.conf^M INFO:__main__:Deleting /etc/libvirt/virtsecretd.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtsecretd.conf to /etc/libvirt/virtsecretd.conf^M INFO:__main__:Deleting /etc/libvirt/virtstoraged.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtstoraged.conf to /etc/libvirt/virtstoraged.conf^M INFO:__main__:Deleting /etc/nova/migration/authorized_keys^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/authorized_keys to /etc/nova/migration/authorized_keys^M INFO:__main__:Deleting /etc/nova/migration/identity^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/identity to /etc/nova/migration/identity^M INFO:__main__:Creating directory /etc/nova/provider_config^M INFO:__main__:Deleting /etc/nova/nova.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/nova.conf to /etc/nova/nova.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/secret.xml to /etc/nova/secret.xml^M INFO:__main__:Deleting /etc/sasl2/libvirt.conf^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sasl2/libvirt.conf to /etc/sasl2/libvirt.conf^M INFO:__main__:Deleting /etc/ssh/sshd_config^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ssh/sshd_config to /etc/ssh/sshd_config^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/libvirtd to /etc/sysconfig/libvirtd^M INFO:__main__:Deleting /etc/login.defs^M INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/login.defs to /etc/login.defs^M INFO:__main__:Deleting /var/lib/nova/.ssh/config^M INFO:__main__:Copying /var/lib/kolla/config_files/src/var/lib/nova/.ssh/config to /var/lib/nova/.ssh/config^M INFO:__main__:Writing out command to execute^M INFO:__main__:Setting permission for /var/log/nova^M ++ cat /run_command^M + CMD='/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py'^M + ARGS=^M + [[ ! -n '' ]]^M + . kolla_extend_start^M + echo 'Running command: '\''/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py'\'''^M + umask 0022^M + exec /container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py^M + command -v python3^M + python3 /container-config-scripts/nova_wait_for_compute_service.py 2023-11-18 01:08:53.478665 | 52540014-75de-ec22-69df-00000000d34b | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_5 | compute-0 | error={"changed": false, "msg": "Failed containers: nova_wait_for_compute_service"} [...] Expected results: successful overcloud deployment Additional info: $ metalsmith list +--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+ | UUID | Node Name | Allocation UUID | Hostname | State | IP Addresses | +--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+ | 52cbc39c-ce68-4244-b7b8-95d33a03dbeb | compute-0 | a94e5178-c4d6-4bf3-9187-015aeabd36f8 | compute-0 | ACTIVE | ctlplane=192.168.24.36 | | 091d01cb-d9f1-4239-a18c-62fd0d22eb00 | controller-0 | 2b8a5b47-aad6-41a9-8924-7f10a7efc685 | controller-0 | ACTIVE | ctlplane=192.168.24.24 | | fa112387-e184-483d-8e7c-c46090ee0e87 | controller-1 | f2ba111b-8f97-45ce-a69b-c2c597c578a9 | controller-1 | ACTIVE | ctlplane=192.168.24.13 | | a8169a9a-e12c-46e7-b26a-9cb2fa4ae10a | controller-2 | e62858ed-07d9-4c85-95f1-ec85dc6dcc67 | controller-2 | ACTIVE | ctlplane=192.168.24.50 | +--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+ $ cat overcloud_deploy.sh #!/bin/bash openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-mds.yaml \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \ --libvirt-type kvm \ --stack overcloud \ -r /home/stack/hybrid_templates/roles/roles_data.yaml \ -e /home/stack/hybrid_templates/roles/nodes.yaml \ --deployed-server \ -e /home/stack/templates/overcloud-vip-deployed.yaml \ -e /home/stack/templates/overcloud-networks-deployed.yaml \ -e /home/stack/templates/overcloud-baremetal-deployed.yaml \ -e /home/stack/templates/overcloud-ceph-deployed.yaml \ --disable-protected-resource-types \ --networks-file /home/stack/hybrid_templates/network/network_data_v2.yaml \ -e /home/stack/hybrid_templates/titan35-storage-config.yml \ --networks-file /home/stack/hybrid_templates/network/network_data_v2.yaml \ -e /home/stack/hybrid_templates/network/network-environment_v2.yaml \ -e /home/stack/hybrid_templates/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \ -e /home/stack/hybrid_templates/custom-storage.yaml \ -e /home/stack/hybrid_templates/debug.yaml \ -e /home/stack/hybrid_templates/ntp_pool.yaml \ -e /home/stack/hybrid_templates/config_heat.yaml \ -e ~/containers-prepare-parameter.yaml \ -e /home/stack/hybrid_templates/manila-cephganesha.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \ -e /home/stack/hybrid_templates/cloud-names.yaml \ -e /home/stack/hybrid_templates/public_vip.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \ -e /home/stack/hybrid_templates/ipaservices-baremetal-ansible.yaml \ --log-file overcloud_deployment_89.log