Description of problem:

engine-setup fails while upgrading from 3.5 to 3.6 in SHE. it seems to be because _have_el6_hosted_engine_host() is wrong:

...
       vm_rows = dbstatement.execute(
            statement="""                                                                                                                                                                                                                     
                SELECT vds_group_id                                                                                                                                                                                                           
                FROM vm_static                                                                                                                                                                                                                
                WHERE vm_name='{vmname}'                                                                                                                                                                                                      
            """.format(vmname=HostedEngineVmName),
            ownConnection=True,
            transaction=False,
...
        vm_rows = dbstatement.execute(
            statement="""                                                                                                                                                                                                                     
                SELECT vds_group_id                                                                                                                                                                                                           
                FROM vm_static                                                                                                                                                                                                                
                WHERE vm_name='{vmname}'                                                                                                                                                                                                      
            """.format(vmname=HostedEngineVmName),
            ownConnection=True,
            transaction=False,
        )
...
        hosts = dbstatement.execute(
            statement="""                                                                                                                                                                                                                     
                SELECT vds_name,host_os                                                                                                                                                                                                       
                FROM vds_static s,vds_dynamic d                                                                                                                                                                                               
                WHERE                                                                                                                                                                                                                         
                    s.vds_id=d.vds_id                                                                                                                                                                                                         
                    AND vds_group_id=%(vds_group_id)s                                                                                                                                                                                         
            """,
            args=dict(
                vds_group_id=vm_rows[0]['vds_group_id'],
            ),
            ownConnection=True,
            transaction=False,
        )

^^ and here is wrong assumption, based on "static" data and not a reality.

engine=# select vm_guid from vm_static where vm_name = 'HostedEngine';
               vm_guid                
--------------------------------------
 ed2aca46-1854-4c67-89e6-87c17232cec0
(1 row)

engine=# select run_on_vds from vm_dynamic where vm_guid = 'ed2aca46-1854-4c67-89e6-87c17232cec0';
              run_on_vds              
--------------------------------------
 afba4a3d-f94d-40d6-8dda-8d1460ae9e0c
(1 row)

engine=# select vds_name,vds_id,vds_group_id,host_os,ha_configured,ha_active from vds;
    vds_name    |                vds_id                |             vds_group_id             |                 host_os                  | ha_configured | ha_active 
----------------+--------------------------------------+--------------------------------------+------------------------------------------+---------------+-----------
 jbelka-vhost1  | afba4a3d-f94d-40d6-8dda-8d1460ae9e0c | e1bcf40d-30a0-4bd0-8fb7-e35232ac5cf0 | RHEL - 7.3 - 0.2.el7                     | t             | t
 jbelka-vhost2  | 74b2898a-69b8-40f0-b00f-cb5fbb8d898f | e1bcf40d-30a0-4bd0-8fb7-e35232ac5cf0 | RHEL - 7.3 - 0.2.el7                     | t             | t
 dell-r210ii-13 | f4ff36dd-5a25-4693-84c4-a6380231377f | e1bcf40d-30a0-4bd0-8fb7-e35232ac5cf0 | RHEV Hypervisor - 7.3 - 20170615.0.el7ev | t             | t
 jbelka-vhost8  | 7c0b4a22-4d68-4d8d-824a-dabe2fdf4405 | 00000001-0001-0001-0001-0000000000c4 | RHEL - 6Server - 6.9.0.4.el6             | f             | f
 jbelka-vhost7  | 30a8e8fa-b4d2-44ef-b170-6626081bdbe3 | 00000001-0001-0001-0001-0000000000c4 | RHEL - 6Server - 6.9.0.4.el6             | f             | f
(5 rows)

So my current 3.5 env is like this:

- Default cluster with EL6 non-SHE hosts which had SHE EL6 hosts
- temp cluster with EL7 SHE hosts as a step in upgrade flow

I wanted to upgrade 3.5 engine to 3.6 version first so I could in 'InClusterUpgrade' policy on 3.6 to upgrade remaining EL6 to EL7 one by one in existing cluster and _only_ then to return all SHE hosts from temp to Default cluster.

Version-Release number of selected component (if applicable):
rhevm-setup-plugin-ovirt-engine-3.6.11.3-0.1.el6.noarch

How reproducible:
1 in 1

Steps to Reproduce:
1. have 3.5 she env with 2 el6 she host and 1 el6 non-she host in Default
   cluster
2. put 1 she host into temporary cluster, reinstall to rhevh7[1], upgrade to
   3.6 rhevh7,... when done reinstall remaining she el6 host to 3.6 ngn, upgrade
   all she hosts to latest version, double check they are now in temporary
   cluster
3. Default cluster now hosts only el6 host
4. upgrade engine

Actual results:
engine-setup fails and makes wrong assumption that SHE env is built on top of
EL6 hosts

Expected results:
engine-setup should check if SHE env is used, then what are SHE hosts' OS in reality (see 'ha_configured' in vds)

Additional info:
rhv-field-eng effort

[1] https://access.redhat.com/solutions/2351141