Bug 1489925

Summary: 3.6 engine-setup fails while upgrading from 3.5 to 3.6 in SHE
Product: Red Hat Enterprise Virtualization Manager Reporter: Jiri Belka <jbelka>
Component: ovirt-engineAssignee: Douglas Schilling Landgraf <dougsland>
Status: CLOSED DUPLICATE QA Contact: meital avital <mavital>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.6.0CC: ahino, bugs, dougsland, gveitmic, lsurette, rbalakri, Rhev-m-bugs, srevivo, ykaul, ylavi
Target Milestone: ovirt-3.6.z-async   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-17 15:37:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1477632, 1503446, 1529957    
Bug Blocks:    

Description Jiri Belka 2017-09-08 16:48:56 UTC
Description of problem:

engine-setup fails while upgrading from 3.5 to 3.6 in SHE. it seems to be because _have_el6_hosted_engine_host() is wrong:

...
       vm_rows = dbstatement.execute(
            statement="""                                                                                                                                                                                                                     
                SELECT vds_group_id                                                                                                                                                                                                           
                FROM vm_static                                                                                                                                                                                                                
                WHERE vm_name='{vmname}'                                                                                                                                                                                                      
            """.format(vmname=HostedEngineVmName),
            ownConnection=True,
            transaction=False,
...
        vm_rows = dbstatement.execute(
            statement="""                                                                                                                                                                                                                     
                SELECT vds_group_id                                                                                                                                                                                                           
                FROM vm_static                                                                                                                                                                                                                
                WHERE vm_name='{vmname}'                                                                                                                                                                                                      
            """.format(vmname=HostedEngineVmName),
            ownConnection=True,
            transaction=False,
        )
...
        hosts = dbstatement.execute(
            statement="""                                                                                                                                                                                                                     
                SELECT vds_name,host_os                                                                                                                                                                                                       
                FROM vds_static s,vds_dynamic d                                                                                                                                                                                               
                WHERE                                                                                                                                                                                                                         
                    s.vds_id=d.vds_id                                                                                                                                                                                                         
                    AND vds_group_id=%(vds_group_id)s                                                                                                                                                                                         
            """,
            args=dict(
                vds_group_id=vm_rows[0]['vds_group_id'],
            ),
            ownConnection=True,
            transaction=False,
        )

^^ and here is wrong assumption, based on "static" data and not a reality.

engine=# select vm_guid from vm_static where vm_name = 'HostedEngine';
               vm_guid                
--------------------------------------
 ed2aca46-1854-4c67-89e6-87c17232cec0
(1 row)

engine=# select run_on_vds from vm_dynamic where vm_guid = 'ed2aca46-1854-4c67-89e6-87c17232cec0';
              run_on_vds              
--------------------------------------
 afba4a3d-f94d-40d6-8dda-8d1460ae9e0c
(1 row)

engine=# select vds_name,vds_id,vds_group_id,host_os,ha_configured,ha_active from vds;
    vds_name    |                vds_id                |             vds_group_id             |                 host_os                  | ha_configured | ha_active 
----------------+--------------------------------------+--------------------------------------+------------------------------------------+---------------+-----------
 jbelka-vhost1  | afba4a3d-f94d-40d6-8dda-8d1460ae9e0c | e1bcf40d-30a0-4bd0-8fb7-e35232ac5cf0 | RHEL - 7.3 - 0.2.el7                     | t             | t
 jbelka-vhost2  | 74b2898a-69b8-40f0-b00f-cb5fbb8d898f | e1bcf40d-30a0-4bd0-8fb7-e35232ac5cf0 | RHEL - 7.3 - 0.2.el7                     | t             | t
 dell-r210ii-13 | f4ff36dd-5a25-4693-84c4-a6380231377f | e1bcf40d-30a0-4bd0-8fb7-e35232ac5cf0 | RHEV Hypervisor - 7.3 - 20170615.0.el7ev | t             | t
 jbelka-vhost8  | 7c0b4a22-4d68-4d8d-824a-dabe2fdf4405 | 00000001-0001-0001-0001-0000000000c4 | RHEL - 6Server - 6.9.0.4.el6             | f             | f
 jbelka-vhost7  | 30a8e8fa-b4d2-44ef-b170-6626081bdbe3 | 00000001-0001-0001-0001-0000000000c4 | RHEL - 6Server - 6.9.0.4.el6             | f             | f
(5 rows)

So my current 3.5 env is like this:

- Default cluster with EL6 non-SHE hosts which had SHE EL6 hosts
- temp cluster with EL7 SHE hosts as a step in upgrade flow

I wanted to upgrade 3.5 engine to 3.6 version first so I could in 'InClusterUpgrade' policy on 3.6 to upgrade remaining EL6 to EL7 one by one in existing cluster and _only_ then to return all SHE hosts from temp to Default cluster.

Version-Release number of selected component (if applicable):
rhevm-setup-plugin-ovirt-engine-3.6.11.3-0.1.el6.noarch

How reproducible:
1 in 1

Steps to Reproduce:
1. have 3.5 she env with 2 el6 she host and 1 el6 non-she host in Default
   cluster
2. put 1 she host into temporary cluster, reinstall to rhevh7[1], upgrade to
   3.6 rhevh7,... when done reinstall remaining she el6 host to 3.6 ngn, upgrade
   all she hosts to latest version, double check they are now in temporary
   cluster
3. Default cluster now hosts only el6 host
4. upgrade engine

Actual results:
engine-setup fails and makes wrong assumption that SHE env is built on top of
EL6 hosts

Expected results:
engine-setup should check if SHE env is used, then what are SHE hosts' OS in reality (see 'ha_configured' in vds)

Additional info:
rhv-field-eng effort

[1] https://access.redhat.com/solutions/2351141

Comment 3 Douglas Schilling Landgraf 2017-10-17 15:37:44 UTC
This is an old upgrade flow. The new design in bz#1477632 should cover this case, closing this bug for now. Fell free to re-open in case we face it again in the new upgrade flow.

*** This bug has been marked as a duplicate of bug 1477632 ***