Problem description:

The overcloud deploy od RHOSP 16.1 (new installation) failed during the Octavia installation  with this error:

TASK [Write group_vars file] ***************************************************
Friday 18 November 2022  11:28:50 +0100 (0:00:01.100)       0:19:32.210 *******
fatal: [undercloud]: FAILED! => {"msg": "Failed to get information on remote file (/var/lib/mistral/overcloud/octavia-ansible/group_vars/octavia_vars.yaml): Permission denied"}

Diving a bit deeper into the error, it was possible to understand that the directory /var/lib/mistral/overcloud/octavia-ansible/ was created by the installer with tripleo-admin as owner and root as group.

Task task failed because the ansible_user is unable to read and write inside the /var/lib/mistral/overcloud/octavia-ansible/ folder. 

The playbook /usr/share/openstack-tripleo-heat-templates/deployment/octavia/octavia-deployment-config.j2.yaml (line 279), used to create the folder, there is a become:true instruction which is not present in the following task:

- name: Make needed directories on the undercloud
              become: true
              file:
                path: "{{item}}"
                state: directory
                owner: "{{ ansible_user }}"
              with_items:
                - "{{playbook_dir}}/octavia-ansible"
                - "{{ octavia_ansible_group_vars.octavia_local_tmpdir }}"
                - "{{ octavia_ansible_group_vars.octavia_group_vars_dir }}"
            - name: Write group_vars file
              copy:
                dest: "{{ octavia_ansible_group_vars.octavia_group_vars_dir }}/octavia_vars.yaml"
                content: "{{ octavia_ansible_group_vars|to_nice_yaml }}"

The deployment error was bypassed by adding the become:true to the whole block.

As result of this workaround, the Octavia is correctly installed along with all the overcloud but this solution has the side effect to make following stack update fail because the installer hasn't permission to clean /var/lib/mistral/overcloud/octavia-ansible.