Bug 1951548

Description of problem:
This bug was hit while upgrading setup from 4.3.x to 4.4.x (last was to 4.4.5) in two different setups with NFS storage domains.
Setup fails with message:

] TASK [ovirt.ovirt.hosted_engine_setup : Activate storage domain]
[ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Domain format is different from master storage domain format]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Domain format is different from master storage domain format]\". HTTP response code is 400."}

This is because engine-setup ansible task creates V5 format storage domain, but my cluster had V4
It looks like storage domains in data center should have been upgraded to V5 when DC and
cluster compatibility version was updated to 4.3, but looks like it was implemented only in ovirt 4.3.3
and this setup was updated from 4.2 to 4.3 (with cluster compatibility update) before 4.3.3 was released,
so I ended up with 4.3 DCs and clusters with V4 storage domain format.
AFAIK there is no standard way to force update of storage format to V5 in hosted-engine cluster, so everybody who updated their cluster compatibility level to 4.3 before ovirt-4.3.3 may be affected.


Version-Release number of selected component (if applicable):
ovirt 4.4.5
ovirt-hosted-engine-setup.noarch                  2.4.9-1.el8
ovirt-ansible-collection.noarch                   1.4.1-1.el8

How reproducible:
Was reproduced on two different setups, both were upgraded from 4.2 to 4.3 in early 4.3 releases, then updated to latest 4.3(that was available at a time, first setup was updated in September,
second few weeks ago) before upgrading to 4.4.

Steps to Reproduce:
1. upgrade 4.2 to 4.3.0-4.3.2(?)
2. update cluster compatibility level to 4.3 (storage domains will be V4 format)
3. upgrade to latest 4.3.x
4. try to upgrade to 4.4

Actual results:
Update fails with error "Domain format is different from master storage domain format".

Expected results:
Upgrade succeeded.

Additional info:
I was able to finish both upgrades with some dirty workarounds:
! YOU ARE DOING IT AT YOUR OWN RISK !
! first workaround not working in last versions, this is only for info !
First time after first failure (when execution pauses and waiting for you to enter correct storage domain again)
I was able to login in to temporary engine portal via link provided by engine-setup in: "The bootstrap engine is temporary accessible over" step
(I've also answered 'yes' to the question "Pause the execution after adding this host to the engine?" at the beginning of engine-setup, I don't know if it matters)
and create hosted_storage domain manually, when creating domain manually engine correctly assigns V4 format, and after re-entering same information it accepted manually created
domain and setup finished correctly, after updating cluster compatibility level to 4.4 storage formats were correctly updated to V5


Second time update was done to 4.4.5 and was few weeks ago, first workaround didn't work this time, after creating domain manually I fails with new error:
"Cannot add Storage Connection. Storage connection 2074fe30-ff7c-4890-aa00-2440ef3ff682 already exists for hosted-storage."
The idea is to force setup to create V4 domain, or create empty expected V4 domain, that setup can use.
I started to dig into ansible playbooks and specifically /usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/create_storage_domain.yml
on deployment host, my domains are NFS so, we are looking at NFS domain creation code:
  - name: Add NFS storage domain
    ovirt_storage_domain:
      state: unattached
      name: "{{ he_storage_domain_name }}"
      host: "{{ he_host_name }}"
      data_center: "{{ datacenter_name }}"
      wait: true
      nfs:
        address: "{{ he_storage_domain_addr }}"
        path: "{{ he_storage_domain_path }}"
        mount_options: "{{ he_mount_options }}"
        version: "{{ he_nfs_version }}"
      auth: "{{ ovirt_auth }}"
    when: he_domain_type == "nfs"
    register: otopi_storage_domain_details_nfs

As I can see from ovirt_storage_domain module description ( https://docs.ansible.com/ansible/latest/collections/ovirt/ovirt/ovirt_storage_domain_module.html ) there is no way to
force format to V4 version ("format" parameter in ansible module means another functionality).
So first idea won't work, and I've tried second idea.
When execution pauses and waiting for you to enter correct storage domain again:
I've logged in into temporary portal, created hosted_storage domain, verified it was V4, removed it from engine (I don't remember if it's necessary) but don't clean data from it.
Changed ansible playbook for "Add NFS storage domain" from "state: unattached" to "state: imported" (I haven't wrote it down, may be it was "present", there were many experiments),
and supplied correct info for this storage domain to engine-setup, and everything finished correctly, after upgrading all hosts and updating cluster compatibility level to 4.5
storage formats were correctly updated to V5.

Something is broken when domain added by ansible? it incorrectly picks version, while web UI is working correctly.