Description of problem: Failing to deploy Octavia with Error no "post-deploy.conf" Version-Release number of selected component (if applicable): Red Hat OpenStack Platform release 16.2.3 (Train) openstack-tripleo-heat-templates-11.6.1-2.20220409014870.el8ost.noarch How reproducible: Deploy Octavia with RHOSP16.2 ( My test env is 3 controller nodes with 3 compute nodes) Steps to Reproduce: 1. Deloy with "octavia.yaml" ( When I test it "ACTIVE_STANDBY" was regardless with the issue) $ cat deploy_noceph.sh openstack overcloud deploy --templates \ -n /home/stack/templates/network_data.yaml \ -e /home/stack/templates/node_info.yaml\ -e /home/stack/containers-prepare-parameter.yaml \ -e /home/stack/openstack-tripleo-heat-templates-rendered/environments/net-single-nic-with-vlans.yaml \ -e /home/stack/openstack-tripleo-heat-templates-rendered/environments/network-isolation.yaml \ -e /home/stack/openstack-tripleo-heat-templates-rendered/environments/network-environment.yaml \ -e /home/stack/openstack-tripleo-heat-templates-rendered/environments/services/octavia.yaml \ -e /home/stack/templates/my-octavia-environment.yaml \ --ntp-server 192.168.24.1 --libvirt-type qemu $ cat /home/stack/templates/my-octavia-environment.yaml parameter_defaults: OctaviaLoadBalancerTopology: "ACTIVE_STANDBY" Actual results: - Deploy failed with the below error 2022-11-01 11:34:35,843 p=216835 u=mistral n=ansible | 2022-11-01 11:34:35.842887 | 5254000c-4156-b106-bd9b-000000006bb9 | TASK | Debug output for task: Start containers for step 5 2022-11-01 11:34:35,905 p=216835 u=mistral n=ansible | 2022-11-01 11:34:35.905092 | 5254000c-4156-b106-bd9b-000000006bb9 | OK | Debug output for task: Start containers for step 5 | overcloud-controller-0 | result={ "changed": false, "failed_when_result": false, "start_containers_outputs.stdout_lines | default([]) | union(start_containers_outputs.stderr_lines | default([]))": [ "b1e14ef3866a8583c7a58ec9146b0ad49fc901b3e67f2ce88b05fd99e854683b", "", "dbe41cb9f7202713f2accb166770bf78cd8fb8e8f490833782256bf36cad94b5", "9169658efa13cf0aef1d80b5a41d31b5746923909e6b9f8cdf3bf709defdead9", "Error executing ['podman', 'container', 'exists', 'octavia_health_manager']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_health_manager', '--filter', 'label=config_id=tripleo_step5', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_health_manager', '--format', '{{.Names}}']\"", "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_octavia_health_manager.service → /etc/systemd/system/tripleo_octavia_health_manager.service.", "Created symlink /etc/systemd/system/timers.target.wants/tripleo_octavia_health_manager_healthcheck.timer → /etc/systemd/system/tripleo_octavia_health_manager_healthcheck.timer.", "Created symlink /etc/systemd/system/tripleo_octavia_health_manager.service.requires/tripleo_octavia_health_manager_healthcheck.timer → /etc/systemd/system/tripleo_octavia_health_manager_healthcheck.timer.", "Error executing ['podman', 'container', 'exists', 'octavia_housekeeping']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_housekeeping', '--filter', 'label=config_id=tripleo_step5', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_housekeeping', '--format', '{{.Names}}']\"", "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_octavia_housekeeping.service → /etc/systemd/system/tripleo_octavia_housekeeping.service.", "Created symlink /etc/systemd/system/timers.target.wants/tripleo_octavia_housekeeping_healthcheck.timer → /etc/systemd/system/tripleo_octavia_housekeeping_healthcheck.timer.", "Created symlink /etc/systemd/system/tripleo_octavia_housekeeping.service.requires/tripleo_octavia_housekeeping_healthcheck.timer → /etc/systemd/system/tripleo_octavia_housekeeping_healthcheck.timer.", "Error executing ['podman', 'container', 'exists', 'octavia_worker']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_worker', '--filter', 'label=config_id=tripleo_step5', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_worker', '--format', '{{.Names}}']\"", "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_octavia_worker.service → /etc/systemd/system/tripleo_octavia_worker.service.", "Created symlink /etc/systemd/system/timers.target.wants/tripleo_octavia_worker_healthcheck.timer → /etc/systemd/system/tripleo_octavia_worker_healthcheck.timer.", "Created symlink /etc/systemd/system/tripleo_octavia_worker.service.requires/tripleo_octavia_worker_healthcheck.timer → /etc/systemd/system/tripleo_octavia_worker_healthcheck.timer." ] } 2022-11-01 11:34:35,974 p=216835 u=mistral n=ansible | 2022-11-01 11:34:35.974116 | 5254000c-4156-b106-bd9b-000000006bbb | TASK | Manage containers for step 5 with tripleo-ansible 2022-11-01 11:34:36,032 p=216835 u=mistral n=ansible | 2022-11-01 11:34:36.032029 | 5254000c-4156-b106-bd9b-000000006bbb | SKIPPED | Manage containers for step 5 with tripleo-ansible | overcloud-controller-0 .... Expected results: Deploy octavia with no issue. Additional info: - when it failed, Those containers (octavia_health_manager , octavia_housekeeping , octavia_worker) were kept restarted in the controllers. [root@overcloud-controller-0 ~]# podman ps |grep octa d6f852ca2c94 undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-octavia-api:16.2 kolla_start 3 hours ago Up 3 hours ago octavia_api 0a69f177243b undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-octavia-api:16.2 kolla_start 3 hours ago Up 3 hours ago octavia_driver_agent b1e14ef3866a undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-octavia-health-manager:16.2 kolla_start 3 hours ago Up 1 second ago octavia_health_manager dbe41cb9f720 undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-octavia-housekeeping:16.2 kolla_start 3 hours ago Up 1 second ago octavia_housekeeping 9169658efa13 undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-octavia-worker:16.2 kolla_start 3 hours ago Up 2 seconds ago octavia_worker [root@overcloud-controller-0 ~]# podman logs octavia_health_manager ... INFO:__main__:Setting permission for /var/log/octavia/driver-agent.log ++ cat /run_command + CMD='/usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/octavia-health-manager' + ARGS= + [[ ! -n '' ]] + . kolla_extend_start Running command: '/usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/octavia-health-manager' + echo 'Running command: '\''/usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/octavia-health-manager'\''' + exec /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/octavia-health-manager Traceback (most recent call last): File "/usr/bin/octavia-health-manager", line 10, in <module> sys.exit(main()) File "/usr/lib/python3.6/site-packages/octavia/cmd/health_manager.py", line 88, in main service.prepare_service(sys.argv) File "/usr/lib/python3.6/site-packages/octavia/common/service.py", line 25, in prepare_service config.init(argv[1:]) File "/usr/lib/python3.6/site-packages/octavia/common/config.py", line 814, in init **kwargs) File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2134, in __call__ raise ConfigFilesNotFoundError(self._namespace._files_not_found) oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/octavia/post-deploy.conf ... [root@overcloud-controller-0 ~]# podman logs octavia_housekeeping ... INFO:__main__:Setting permission for /var/log/octavia/driver-agent.log ++ cat /run_command Running command: '/usr/bin/octavia-housekeeping --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/housekeeping.log --config-dir /etc/octavia/conf.d/octavia-housekeeping' + CMD='/usr/bin/octavia-housekeeping --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/housekeeping.log --config-dir /etc/octavia/conf.d/octavia-housekeeping' + ARGS= + [[ ! -n '' ]] + . kolla_extend_start + echo 'Running command: '\''/usr/bin/octavia-housekeeping --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/housekeeping.log --config-dir /etc/octavia/conf.d/octavia-housekeeping'\''' + exec /usr/bin/octavia-housekeeping --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/housekeeping.log --config-dir /etc/octavia/conf.d/octavia-housekeeping Traceback (most recent call last): File "/usr/bin/octavia-housekeeping", line 10, in <module> sys.exit(main()) File "/usr/lib/python3.6/site-packages/octavia/cmd/house_keeping.py", line 99, in main service.prepare_service(sys.argv) File "/usr/lib/python3.6/site-packages/octavia/common/service.py", line 25, in prepare_service config.init(argv[1:]) File "/usr/lib/python3.6/site-packages/octavia/common/config.py", line 814, in init **kwargs) File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2134, in __call__ raise ConfigFilesNotFoundError(self._namespace._files_not_found) oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/octavia/post-deploy.conf + sudo -E kolla_set_configs sudo: unable to send audit message: Operation not permitted INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json ... [root@overcloud-controller-0 ~]# podman logs octavia_worker INFO:__main__:Setting permission for /var/log/octavia/driver-agent.log ++ cat /run_command Running command: '/usr/bin/octavia-worker --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/worker.log --config-dir /etc/octavia/conf.d/octavia-worker' + CMD='/usr/bin/octavia-worker --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/worker.log --config-dir /etc/octavia/conf.d/octavia-worker' + ARGS= + [[ ! -n '' ]] + . kolla_extend_start + echo 'Running command: '\''/usr/bin/octavia-worker --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/worker.log --config-dir /etc/octavia/conf.d/octavia-worker'\''' + exec /usr/bin/octavia-worker --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/worker.log --config-dir /etc/octavia/conf.d/octavia-worker Traceback (most recent call last): File "/usr/bin/octavia-worker", line 10, in <module> sys.exit(main()) File "/usr/lib/python3.6/site-packages/octavia/cmd/octavia_worker.py", line 31, in main octavia_service.prepare_service(sys.argv) File "/usr/lib/python3.6/site-packages/octavia/common/service.py", line 25, in prepare_service config.init(argv[1:]) File "/usr/lib/python3.6/site-packages/octavia/common/config.py", line 814, in init **kwargs) File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2134, in __call__ raise ConfigFilesNotFoundError(self._namespace._files_not_found) oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/octavia/post-deploy.conf + sudo -E kolla_set_configs sudo: unable to send audit message: Operation not permitted ... Seems relevant config files didn't generate. [root@overcloud-controller-0 ~]# ls -al /var/lib/config-data/puppet-generated/octavia/etc/* /var/lib/config-data/puppet-generated/octavia/etc/httpd: total 4 drwxr-xr-x. 4 root root 32 Oct 6 15:37 . drwxr-xr-x. 5 root root 50 Nov 1 02:14 .. drwxr-xr-x. 2 root root 42 Nov 1 02:14 conf drwxr-xr-x. 2 root root 4096 Nov 1 02:14 conf.d /var/lib/config-data/puppet-generated/octavia/etc/my.cnf.d: total 4 drwxr-xr-x. 2 root root 25 Nov 1 02:14 . drwxr-xr-x. 5 root root 50 Nov 1 02:14 .. -rw-r--r--. 1 root root 37 Nov 1 02:14 tripleo.cnf /var/lib/config-data/puppet-generated/octavia/etc/octavia: total 24 drwxr-xr-x. 3 root root 40 Nov 1 02:22 . drwxr-xr-x. 5 root root 50 Nov 1 02:14 .. drwxr-xr-x. 6 root root 105 Nov 1 02:22 conf.d -rw-r-----. 1 root 42437 23812 Nov 1 02:14 octavia.conf but I modifing "octavia-deployment-config.j2.yaml " for a test then the issue was gone. ........ external_post_deploy_tasks: - name: octavia_post_deploy block: - name: Set up group_vars set_fact: octavia_ansible_group_vars: { get_attr: [OctaviaVars, value, vars] } no_log: "{{ hide_sensitive_logs | bool }}" - name: Make needed directories on the undercloud become: true file: path: "{{item}}" state: directory owner: "{{ ansible_user }}" with_items: - "{{playbook_dir}}/octavia-ansible" - "{{ octavia_ansible_group_vars.octavia_local_tmpdir }}" - "{{ octavia_ansible_group_vars.octavia_group_vars_dir }}" - name: Write group_vars file become: true ### ADD here copy: dest: "{{ octavia_ansible_group_vars.octavia_group_vars_dir }}/octavia_vars.yaml" content: "{{ octavia_ansible_group_vars|to_nice_yaml }}" no_log: "{{ hide_sensitive_logs | bool }}" - name: Write octavia inventory become: true ### ADD here copy: dest: "{{playbook_dir}}/octavia-ansible/inventory.yaml" content: | octavia_nodes: hosts: {%- set octavia_groups = ['worker'] -%} {%- for octavia_group in octavia_groups -%} {%- if 'octavia_' ~ octavia_groups %} {% for host in groups['octavia_' ~ octavia_group] -%} {{ hostvars.raw_get(host)['ansible_facts']['hostname'] | lower}}: ansible_user: {{ hostvars.raw_get(host)['ansible_ssh_user'] | default('heat-admin') }} ansible_host: {{ hostvars.raw_get(host)['ansible_host'] | default(host) | lower }} canonical_hostname: {{ hostvars.raw_get(host)['canonical_hostname'] | default(host) | lower }} ansible_become: true {% endfor %} {%- endif -%} {%- endfor %} Undercloud: hosts: {% for host in groups['Undercloud'] -%} {{ hostvars.raw_get(host)['ansible_facts']['hostname'] | lower}}: ansible_host: {{ hostvars.raw_get(host)['ansible_host'] | default(host) | lower }} ansible_become: false ansible_connection: local {%- endfor -%} - name: Check for ssh_private_key in working directory become: true ### ADD here stat: path: "{{playbook_dir}}/ssh_private_key" register: detect_private_key_file .......