Description of problem:

While adding a host, ansible code tries to do:

  - name: Copy vdsm and QEMU CSRs
    copy:
      content: "{{ item.stdout }}"
      dest: "{{ ovirt_pki_dir }}/{{ item.item.item.req_dir }}/{{ ovirt_vds_hostname }}.req"
      mode: 0644
    loop: '{{ csrs.results }}'
    delegate_to: localhost

This fails, with this in /var/log/ovirt-engine/ansible-runner-service.log :

    Destination /etc/pki/ovirt-engine/requests not writable

See e.g. [1][2].

This is on current CentOS Stream.

I suspect it's something related to httpd or ansible-runner dropping capabilities, or something like that.

Reproduced also locally, using OST. If I do:

# su - ovirt -s /bin/bash
$ python3 -c 'import os; print(os.access("/etc/pki/ovirt-engine/requests", os.W_OK))'
True

But apparently httpd runs it somewhat differently. If I strace httpd, I see:

45390 13:05:42.430945 stat("/etc/pki/ovirt-engine/requests", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
45390 13:05:42.431065 access("/etc/pki/ovirt-engine/requests", W_OK) = -1 EACCES (Permission denied)

[1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1881/
[2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1881/artifact/exported-artifacts/test_logs/he-basic-suite-master/post-he_deploy/lago-he-basic-suite-master-host-0/_var_log/ovirt-hosted-engine-setup/engine-logs-2021-01-12T08%3A10%3A34Z/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210112092316-lago-he-basic-suite-master-host-0.lago.local-115ec06b.log