Bug 1884599

Description of problem:

An ovirt-hosted-engine-cleanup was executed after a failed deployment and then the deployment is always failing with the error below while installing the HostedEngineLocal VM.

====
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["virt-install", "-n", "HostedEngineLocal", "--os-variant", "rhel8.0", "--virt-type", "kvm", "--memory", "2786", "--vcpus", "4", "--network", "network=default,mac=56:6f:88:58:00:00,model=virtio", "--disk", .....
 "msg": "non-zero return code", "rc": 1, "start": "2020-10-02 12:07:26.297519", "stderr": "ERROR    internal error: process exited while connecting to monitor: 2020-10-02T12:07:39.063524Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/vdsm/libvirt-vnc,endpoint=server,verify-peer=no: Unable to access credentials /etc/pki/vdsm/libvirt-vnc/ca-cert.pem: No such file or directory\nDomain installation does not appear to have been successful......... --connect qemu:///system start HostedEngineLocal", "otherwise, please restart your installation."], "stdout": "\nStarting install...", "stdout_lines": ["", "Starting install..."]}
====

The issue is because the cleanup script is not removing the below entries from the file "/etc/libvirt/qemu.conf".

===
tail -4 /etc/libvirt/qemu.conf
## beginning of configuration section for VNC encryption
vnc_tls=1
vnc_tls_x509_cert_dir="/etc/pki/vdsm/libvirt-vnc"
## end of configuration section for VNC encryption
===

The above entries are added if the cluster has "VNC encryption" enabled. The default cluster don't have "VNC encryption" enabled by default. However any new cluster which is created will be having "vnc encryption" by default. So the issue is only reproducible if we provide custom DC and cluster names.

The initial cleanup play is configured to cleanup data between beginning and end of "configuration section by vdsm-4.40.0".

===
  - name: Drop vdsm config statements
    command: >-
      sed -i
      '/## beginning of configuration section by
      vdsm-4.[0-9]\+.[0-9]\+/,/## end of configuration section by vdsm-4.[0-9]\+.[0-9]\+/d' {{ item }}
    environment: "{{ he_cmd_lang }}"
    args:
      warn: false
    with_items:
      - /etc/libvirt/libvirtd.conf
      - /etc/libvirt/qemu.conf
      - /etc/libvirt/qemu-sanlock.conf
      - /etc/sysconfig/libvirtd
===

However, the VNC encryption configuration which is added in qemu.conf is after "end of configuration section by vdsm".

===
grep -A 17 "beginning of configuration section by vdsm" /etc/libvirt/qemu.conf
## beginning of configuration section by vdsm-4.40.0
dynamic_ownership=1
group="qemu"
lock_manager="sanlock"
max_core="unlimited"
migrate_tls_x509_cert_dir="/etc/pki/vdsm/libvirt-migrate"
remote_display_port_max=6923
remote_display_port_min=5900
save_image_format="gzip"
spice_tls=1
spice_tls_x509_cert_dir="/etc/pki/vdsm/libvirt-spice"
user="qemu"
## end of configuration section by vdsm-4.40.0
## beginning of configuration section for VNC encryption
vnc_tls=1
vnc_tls_x509_cert_dir="/etc/pki/vdsm/libvirt-vnc"
## end of configuration section for VNC encryption
===

The user has to manually comment those lines for the deployment to proceed.


Version-Release number of selected component (if applicable):

ovirt-hosted-engine-setup-2.4.5-1.el8ev.noarch
vdsm-4.40.22-1.el8ev.x86_64

How reproducible:

100%

Steps to Reproduce:

1. During the hosted-engine deploy, provide custom name for DC and cluster.
2. Cancel the deployment once the host is added to the hosted-engine local VM.
3. Run ovirt-hosted-engine-cleanup.
4. Try to deploy the hosted-engine again and it will fail with the above mentioned error.

Actual results:

Hosted-engine deploy is failing with error "Unable to access credentials /etc/pki/vdsm/libvirt-vnc/ca-cert.pem" while it starts the engine VM

Expected results:

User should be able to re-deploy without changing the configuration files manually.

Additional info: