Bug 2139041

Description of problem:
Failing to deploy Octavia with Error no "post-deploy.conf"

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform release 16.2.3 (Train)
openstack-tripleo-heat-templates-11.6.1-2.20220409014870.el8ost.noarch

How reproducible:
Deploy Octavia with RHOSP16.2 ( My test env is 3 controller nodes with 3 compute nodes)

Steps to Reproduce:

1. Deloy with "octavia.yaml"
( When I test it "ACTIVE_STANDBY" was regardless with the issue)


$ cat deploy_noceph.sh
openstack overcloud deploy --templates \
-n /home/stack/templates/network_data.yaml \
-e /home/stack/templates/node_info.yaml\
-e /home/stack/containers-prepare-parameter.yaml \
        -e /home/stack/openstack-tripleo-heat-templates-rendered/environments/net-single-nic-with-vlans.yaml \
-e /home/stack/openstack-tripleo-heat-templates-rendered/environments/network-isolation.yaml \
        -e /home/stack/openstack-tripleo-heat-templates-rendered/environments/network-environment.yaml \
        -e /home/stack/openstack-tripleo-heat-templates-rendered/environments/services/octavia.yaml \
        -e /home/stack/templates/my-octavia-environment.yaml \
        --ntp-server 192.168.24.1 --libvirt-type qemu

$ cat /home/stack/templates/my-octavia-environment.yaml
parameter_defaults:
    OctaviaLoadBalancerTopology: "ACTIVE_STANDBY"




Actual results:

- Deploy failed with the below error


2022-11-01 11:34:35,843 p=216835 u=mistral n=ansible | 2022-11-01 11:34:35.842887 | 5254000c-4156-b106-bd9b-000000006bb9 |       TASK | Debug output for task: Start containers for step 5
2022-11-01 11:34:35,905 p=216835 u=mistral n=ansible | 2022-11-01 11:34:35.905092 | 5254000c-4156-b106-bd9b-000000006bb9 |         OK | Debug output for task: Start containers for step 5 | overcloud-controller-0 | result={
    "changed": false,
    "failed_when_result": false,
    "start_containers_outputs.stdout_lines | default([]) | union(start_containers_outputs.stderr_lines | default([]))": [
        "b1e14ef3866a8583c7a58ec9146b0ad49fc901b3e67f2ce88b05fd99e854683b",
        "",
        "dbe41cb9f7202713f2accb166770bf78cd8fb8e8f490833782256bf36cad94b5",
        "9169658efa13cf0aef1d80b5a41d31b5746923909e6b9f8cdf3bf709defdead9",
        "Error executing ['podman', 'container', 'exists', 'octavia_health_manager']: returned 1",
        "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_health_manager', '--filter', 'label=config_id=tripleo_step5', '--format', '{{.Names}}']\" - retrying without config_id",
        "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_health_manager', '--format', '{{.Names}}']\"",
        "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_octavia_health_manager.service → /etc/systemd/system/tripleo_octavia_health_manager.service.",
        "Created symlink /etc/systemd/system/timers.target.wants/tripleo_octavia_health_manager_healthcheck.timer → /etc/systemd/system/tripleo_octavia_health_manager_healthcheck.timer.",
        "Created symlink /etc/systemd/system/tripleo_octavia_health_manager.service.requires/tripleo_octavia_health_manager_healthcheck.timer → /etc/systemd/system/tripleo_octavia_health_manager_healthcheck.timer.",
        "Error executing ['podman', 'container', 'exists', 'octavia_housekeeping']: returned 1",
        "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_housekeeping', '--filter', 'label=config_id=tripleo_step5', '--format', '{{.Names}}']\" - retrying without config_id",
        "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_housekeeping', '--format', '{{.Names}}']\"",
        "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_octavia_housekeeping.service → /etc/systemd/system/tripleo_octavia_housekeeping.service.",
        "Created symlink /etc/systemd/system/timers.target.wants/tripleo_octavia_housekeeping_healthcheck.timer → /etc/systemd/system/tripleo_octavia_housekeeping_healthcheck.timer.",
        "Created symlink /etc/systemd/system/tripleo_octavia_housekeeping.service.requires/tripleo_octavia_housekeeping_healthcheck.timer → /etc/systemd/system/tripleo_octavia_housekeeping_healthcheck.timer.",
        "Error executing ['podman', 'container', 'exists', 'octavia_worker']: returned 1",
        "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_worker', '--filter', 'label=config_id=tripleo_step5', '--format', '{{.Names}}']\" - retrying without config_id",
        "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=octavia_worker', '--format', '{{.Names}}']\"",
        "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_octavia_worker.service → /etc/systemd/system/tripleo_octavia_worker.service.",
        "Created symlink /etc/systemd/system/timers.target.wants/tripleo_octavia_worker_healthcheck.timer → /etc/systemd/system/tripleo_octavia_worker_healthcheck.timer.",
        "Created symlink /etc/systemd/system/tripleo_octavia_worker.service.requires/tripleo_octavia_worker_healthcheck.timer → /etc/systemd/system/tripleo_octavia_worker_healthcheck.timer."
    ]
}
2022-11-01 11:34:35,974 p=216835 u=mistral n=ansible | 2022-11-01 11:34:35.974116 | 5254000c-4156-b106-bd9b-000000006bbb |       TASK | Manage containers for step 5 with tripleo-ansible
2022-11-01 11:34:36,032 p=216835 u=mistral n=ansible | 2022-11-01 11:34:36.032029 | 5254000c-4156-b106-bd9b-000000006bbb |    SKIPPED | Manage containers for step 5 with tripleo-ansible | overcloud-controller-0
....

Expected results:
Deploy octavia with no issue.

Additional info:

- when it failed, Those containers (octavia_health_manager ,  octavia_housekeeping , octavia_worker) were kept restarted in the controllers.

[root@overcloud-controller-0 ~]# podman ps |grep octa
d6f852ca2c94  undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-octavia-api:16.2             kolla_start           3 hours ago  Up 3 hours ago            octavia_api
0a69f177243b  undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-octavia-api:16.2             kolla_start           3 hours ago  Up 3 hours ago            octavia_driver_agent
b1e14ef3866a  undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-octavia-health-manager:16.2  kolla_start           3 hours ago  Up 1 second ago           octavia_health_manager
dbe41cb9f720  undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-octavia-housekeeping:16.2    kolla_start           3 hours ago  Up 1 second ago           octavia_housekeeping
9169658efa13  undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-octavia-worker:16.2          kolla_start           3 hours ago  Up 2 seconds ago          octavia_worker


[root@overcloud-controller-0 ~]# podman logs octavia_health_manager
...
INFO:__main__:Setting permission for /var/log/octavia/driver-agent.log
++ cat /run_command
+ CMD='/usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/octavia-health-manager'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
Running command: '/usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/octavia-health-manager'
+ echo 'Running command: '\''/usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/octavia-health-manager'\'''
+ exec /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/octavia-health-manager
Traceback (most recent call last):
  File "/usr/bin/octavia-health-manager", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/octavia/cmd/health_manager.py", line 88, in main
    service.prepare_service(sys.argv)
  File "/usr/lib/python3.6/site-packages/octavia/common/service.py", line 25, in prepare_service
    config.init(argv[1:])
  File "/usr/lib/python3.6/site-packages/octavia/common/config.py", line 814, in init
    **kwargs)
  File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2134, in __call__
    raise ConfigFilesNotFoundError(self._namespace._files_not_found)
oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/octavia/post-deploy.conf

...


[root@overcloud-controller-0 ~]# podman logs octavia_housekeeping
...
INFO:__main__:Setting permission for /var/log/octavia/driver-agent.log
++ cat /run_command
Running command: '/usr/bin/octavia-housekeeping --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/housekeeping.log --config-dir /etc/octavia/conf.d/octavia-housekeeping'
+ CMD='/usr/bin/octavia-housekeeping --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/housekeeping.log --config-dir /etc/octavia/conf.d/octavia-housekeeping'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
+ echo 'Running command: '\''/usr/bin/octavia-housekeeping --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/housekeeping.log --config-dir /etc/octavia/conf.d/octavia-housekeeping'\'''
+ exec /usr/bin/octavia-housekeeping --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/housekeeping.log --config-dir /etc/octavia/conf.d/octavia-housekeeping
Traceback (most recent call last):
  File "/usr/bin/octavia-housekeeping", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/octavia/cmd/house_keeping.py", line 99, in main
    service.prepare_service(sys.argv)
  File "/usr/lib/python3.6/site-packages/octavia/common/service.py", line 25, in prepare_service
    config.init(argv[1:])
  File "/usr/lib/python3.6/site-packages/octavia/common/config.py", line 814, in init
    **kwargs)
  File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2134, in __call__
    raise ConfigFilesNotFoundError(self._namespace._files_not_found)
oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/octavia/post-deploy.conf
+ sudo -E kolla_set_configs
sudo: unable to send audit message: Operation not permitted
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
...


[root@overcloud-controller-0 ~]# podman logs octavia_worker
INFO:__main__:Setting permission for /var/log/octavia/driver-agent.log
++ cat /run_command
Running command: '/usr/bin/octavia-worker --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/worker.log --config-dir /etc/octavia/conf.d/octavia-worker'
+ CMD='/usr/bin/octavia-worker --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/worker.log --config-dir /etc/octavia/conf.d/octavia-worker'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
+ echo 'Running command: '\''/usr/bin/octavia-worker --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/worker.log --config-dir /etc/octavia/conf.d/octavia-worker'\'''
+ exec /usr/bin/octavia-worker --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --config-file /etc/octavia/post-deploy.conf --log-file /var/log/octavia/worker.log --config-dir /etc/octavia/conf.d/octavia-worker
Traceback (most recent call last):
  File "/usr/bin/octavia-worker", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/octavia/cmd/octavia_worker.py", line 31, in main
    octavia_service.prepare_service(sys.argv)
  File "/usr/lib/python3.6/site-packages/octavia/common/service.py", line 25, in prepare_service
    config.init(argv[1:])
  File "/usr/lib/python3.6/site-packages/octavia/common/config.py", line 814, in init
    **kwargs)
  File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2134, in __call__
    raise ConfigFilesNotFoundError(self._namespace._files_not_found)
oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/octavia/post-deploy.conf
+ sudo -E kolla_set_configs
sudo: unable to send audit message: Operation not permitted
...

Seems relevant config files didn't generate.


[root@overcloud-controller-0 ~]# ls -al /var/lib/config-data/puppet-generated/octavia/etc/*
/var/lib/config-data/puppet-generated/octavia/etc/httpd:
total 4
drwxr-xr-x. 4 root root   32 Oct  6 15:37 .
drwxr-xr-x. 5 root root   50 Nov  1 02:14 ..
drwxr-xr-x. 2 root root   42 Nov  1 02:14 conf
drwxr-xr-x. 2 root root 4096 Nov  1 02:14 conf.d

/var/lib/config-data/puppet-generated/octavia/etc/my.cnf.d:
total 4
drwxr-xr-x. 2 root root 25 Nov  1 02:14 .
drwxr-xr-x. 5 root root 50 Nov  1 02:14 ..
-rw-r--r--. 1 root root 37 Nov  1 02:14 tripleo.cnf

/var/lib/config-data/puppet-generated/octavia/etc/octavia:
total 24
drwxr-xr-x. 3 root root     40 Nov  1 02:22 .
drwxr-xr-x. 5 root root     50 Nov  1 02:14 ..
drwxr-xr-x. 6 root root    105 Nov  1 02:22 conf.d
-rw-r-----. 1 root 42437 23812 Nov  1 02:14 octavia.conf



but I modifing "octavia-deployment-config.j2.yaml " for a test then the issue was gone.

........
      external_post_deploy_tasks:
        - name: octavia_post_deploy
          block:
            - name: Set up group_vars
              set_fact:
                octavia_ansible_group_vars: { get_attr: [OctaviaVars, value, vars] }
              no_log: "{{ hide_sensitive_logs | bool }}"
            - name: Make needed directories on the undercloud
              become: true
              file:
                path: "{{item}}"
                state: directory
                owner: "{{ ansible_user }}"
              with_items:
                - "{{playbook_dir}}/octavia-ansible"
                - "{{ octavia_ansible_group_vars.octavia_local_tmpdir }}"
                - "{{ octavia_ansible_group_vars.octavia_group_vars_dir }}"
            - name: Write group_vars file
              become: true ### ADD here
              copy:
                dest: "{{ octavia_ansible_group_vars.octavia_group_vars_dir }}/octavia_vars.yaml"
                content: "{{ octavia_ansible_group_vars|to_nice_yaml }}"
              no_log: "{{ hide_sensitive_logs | bool }}"
            - name: Write octavia inventory
              become: true ### ADD here
              copy:
                dest: "{{playbook_dir}}/octavia-ansible/inventory.yaml"
                content: |
                  octavia_nodes:
                    hosts:

                  {%- set octavia_groups = ['worker'] -%}
                  {%- for octavia_group in octavia_groups -%}
                  {%- if 'octavia_' ~ octavia_groups %}

                      {% for host in groups['octavia_' ~ octavia_group] -%}
                      {{ hostvars.raw_get(host)['ansible_facts']['hostname'] | lower}}:
                        ansible_user: {{ hostvars.raw_get(host)['ansible_ssh_user'] | default('heat-admin') }}
                        ansible_host: {{ hostvars.raw_get(host)['ansible_host'] | default(host) | lower }}
                        canonical_hostname: {{ hostvars.raw_get(host)['canonical_hostname'] | default(host) | lower }}
                        ansible_become: true
                      {% endfor %}

                  {%- endif -%}
                  {%- endfor %}

                  Undercloud:
                    hosts:
                      {% for host in groups['Undercloud'] -%}
                      {{ hostvars.raw_get(host)['ansible_facts']['hostname'] | lower}}:
                        ansible_host: {{ hostvars.raw_get(host)['ansible_host'] | default(host) | lower }}
                        ansible_become: false
                        ansible_connection: local

                      {%- endfor -%}
            - name: Check for ssh_private_key in working directory
              become: true ### ADD here
              stat:
                path: "{{playbook_dir}}/ssh_private_key"
              register: detect_private_key_file
.......