Bug 1287126 - Host cannot be added to a cluster running in 3.6 compatibility mode
Host cannot be added to a cluster running in 3.6 compatibility mode
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: releng (Show other bugs)
3.6.0
x86_64 Linux
urgent Severity urgent
: ---
: ---
Assigned To: Joey Boggs
releng
:
: 1287299 (view as bug list)
Depends On:
Blocks: RHEV_36_HTB
  Show dependency treegraph
 
Reported: 2015-12-01 09:44 EST by Martin Tessun
Modified: 2016-05-23 07:03 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-11 04:23:44 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Rel-Eng
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Martin Tessun 2015-12-01 09:44:54 EST
Description of problem:
When installing a new RHEV 3.6 beta system (including RHEV-M and RHEL-H) the newly created hosts cannot be added to the Cluster

Version-Release number of selected component (if applicable):
- vdsm-4.17.10.1-0.el7ev.noarch
- rhevm-3.6.0.3-0.1.el6.noarch


How reproducible:
always

Steps to Reproduce:
1. Setup new RHEL-H with RHEL 7.2
2. Create a DC with Compatibility 3.6 and a Cluster with Compatibility 3.6
3. Cluster should have x86_64 Architecture and e.g. Sandy_Bridge as
   Compatibility level.
4. Add the newly created host to the Cluster

Actual results:
The host cannot join and goes to Non-Operational.
The Event shows:
Host <RHEL_H> does not comply with the cluster Default emulated machines. The current cluster compatibility level supports [pc-i440fx-rhel7.2.0, pseries-rhel7.2.0] and the host emulated machines are pc-i440fx-rhel7.1.0,rhel6.3.0,q35,rhel6.1.0,rhel6.6.0,rhel6.2.0,pc,pc-q35-rhel7.1.0,pc-q35-rhel7.0.0,rhel6.4.0,rhel6.0.0,rhel6.5.0,pc-i440fx-rhel7.0.0.

Expected results:
Host should join without any issues.

Additional info:
Additionally I am wondering about 3.6 needing pseries Emulator for x86_64 architecture:
pseries-rhel7.2.0 <== Should only be visible for pseries/Power Architecture.

Additionally vdsm should report pc-i440fx-rhel7.2.0 as well. Currently it doesn't:
# vdsClient -s 0 getVdsCapabilities
[...]
 emulated_machines = ['pc-i440fx-rhel7.1.0',
                      'rhel6.3.0',
                      'q35',
                      'rhel6.1.0',
                      'rhel6.6.0',
                      'rhel6.2.0',
                      'pc',
                      'pc-q35-rhel7.1.0',
                      'pc-q35-rhel7.0.0',
                      'rhel6.4.0',
                      'rhel6.0.0',
                      'rhel6.5.0',
                      'pc-i440fx-rhel7.0.0']
[...]

So maybe these are two bugs; one for vdsm and one for ovirt-engine.
vdsm: Not reporting pc-i440fx-rhel7.2.0
ovirt-engine: Requesting pseries-rhel7.2.0 even for x86 architecture

Also the setup of hosted engine fails as the Host cannot be added to the cluster during hosted-engine --deploy.
Comment 1 Martin Tessun 2015-12-01 09:47:36 EST
engin.log shows the following line:

2015-12-01 15:44:25,054 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [17d97417] Correlation ID: 17d97417, Job ID: 567a69a9-8767-4d51-af09-76ac6dd0849f, Call Stack: null, Custom Event ID: -1, Message: Host ovirt1 does not comply with the cluster Default emulated machines. The current cluster compatibility level supports [pc-i440fx-rhel7.2.0, pseries-rhel7.2.0] and the host emulated machines are pc-i440fx-rhel7.1.0,rhel6.3.0,q35,rhel6.1.0,rhel6.6.0,rhel6.2.0,pc,pc-q35-rhel7.1.0,pc-q35-rhel7.0.0,rhel6.4.0,rhel6.0.0,rhel6.5.0,pc-i440fx-rhel7.0.0.


Deploying hosted_engine stops at:
          Checking for oVirt-Engine status at ovirt.satellite.local...
[ INFO  ] Engine replied: DB Up!Welcome to Health Status!
[ INFO  ] Connecting to the Engine
          Enter the name of the cluster to which you want to add the host (Default) [Default]: 
[ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes...
          The host ovirt1 is in non-operational state.
          Please try to activate it via the engine webadmin UI.
          Retry checking host status or ignore this and continue (Retry, Ignore)[Retry]? 


In case the DC and Cluster compatibility is set to 3.5 the host can be added.
Comment 2 Martin Tessun 2015-12-01 09:50:31 EST
Sorry too fast submitted.
After updating the DC and cluster to 3.6 the host stays up. But as soon as the Host is set to maintenance and tried to enable again, the host also goes to Non Operational.
Comment 3 Julio Entrena Perez 2015-12-01 10:12:58 EST
Could it be that we're still missing bug 1210050 ?
Comment 4 Oved Ourfali 2015-12-03 00:56:51 EST
*** Bug 1287299 has been marked as a duplicate of this bug. ***
Comment 5 Michal Skrivanek 2015-12-03 02:55:01 EST
The released content doesn't correspond to what should be out there. Please check qemu-kvm-rhev version, it is likely an old 7.1-based one
Comment 7 Michal Skrivanek 2015-12-03 02:58:56 EST
note as per Fabian RHEV-H should work ok as it does include the right set of packages
Comment 8 Marina 2015-12-03 13:02:42 EST
Hi Martin, if needed, there is a workaround available - at the end of the description of this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1287299#c0
Comment 9 Marina 2015-12-03 13:05:36 EST
(In reply to Michal Skrivanek from comment #5)
> The released content doesn't correspond to what should be out there. Please
> check qemu-kvm-rhev version, it is likely an old 7.1-based one

Indeed, this is what I see on my machine:
~~~
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-2.1.2-23.el7_1.10.x86_64
~~~

However, I do not see why this should be a problem and why we just cannot update the default list of emulated machines on the engine to include the rhel7.1? (unless it is not a static list and actually generated on the fly, I do not believe it is.)
Comment 10 Martin Tessun 2015-12-04 02:14:30 EST
Hi Marina,

(In reply to Marina from comment #9)
> (In reply to Michal Skrivanek from comment #5)
> > The released content doesn't correspond to what should be out there. Please
> > check qemu-kvm-rhev version, it is likely an old 7.1-based one
> 
> Indeed, this is what I see on my machine:
> ~~~
> # rpm -q qemu-kvm-rhev
> qemu-kvm-rhev-2.1.2-23.el7_1.10.x86_64
> ~~~
> 
> However, I do not see why this should be a problem and why we just cannot
> update the default list of emulated machines on the engine to include the
> rhel7.1? (unless it is not a static list and actually generated on the fly,
> I do not believe it is.)

no it isn't. It is taken from the database table vdc_options:

engine=# select * from vdc_options where option_name='ClusterEmulatedMachines';
 option_id |       option_name       |             option_value              | version 
-----------+-------------------------+---------------------------------------+---------
        40 | ClusterEmulatedMachines | rhel6.2.0                             | 3.0
        41 | ClusterEmulatedMachines | rhel6.3.0                             | 3.1
        42 | ClusterEmulatedMachines | rhel6.4.0                             | 3.2
        43 | ClusterEmulatedMachines | rhel6.5.0                             | 3.3
        44 | ClusterEmulatedMachines | rhel6.5.0,pseries                     | 3.4
        45 | ClusterEmulatedMachines | rhel6.5.0,pseries                     | 3.5
        46 | ClusterEmulatedMachines | pc-i440fx-rhel7.2.0,pseries-rhel7.2.0 | 3.6
 (7 rows)


So one can easily change it also by running an update on the table:
engine=# update vdc_options set option_value='rhel6.5.0,pseries' where option_id=46;

Cheers,
Martin
Comment 11 Michal Skrivanek 2015-12-04 04:56:03 EST
(In reply to Marina from comment #9)
> (In reply to Michal Skrivanek from comment #5)
> > The released content doesn't correspond to what should be out there. Please
> > check qemu-kvm-rhev version, it is likely an old 7.1-based one
> 
> Indeed, this is what I see on my machine:
> ~~~
> # rpm -q qemu-kvm-rhev
> qemu-kvm-rhev-2.1.2-23.el7_1.10.x86_64
> ~~~
> 
> However, I do not see why this should be a problem and why we just cannot
> update the default list of emulated machines on the engine to include the
> rhel7.1? (unless it is not a static list and actually generated on the fly,
> I do not believe it is.)

Please don't do that. It will allow you to add the host all right, but you are running a _wrong_ QEMU, the one not supporting fully the 3.6 features (notably hotplug ram), not containing bugfixes, not tested by QE.

If you're fine with that (e.g. testing a new setup not intended for production) then feel free to go ahead, but you could as well add it in 3.5 compatibility mode
Comment 12 Marina 2015-12-04 08:24:33 EST
(In reply to Michal Skrivanek from comment #11)
> (In reply to Marina from comment #9)
> > (In reply to Michal Skrivanek from comment #5)
> > > The released content doesn't correspond to what should be out there. Please
> > > check qemu-kvm-rhev version, it is likely an old 7.1-based one
> > 
> > Indeed, this is what I see on my machine:
> > ~~~
> > # rpm -q qemu-kvm-rhev
> > qemu-kvm-rhev-2.1.2-23.el7_1.10.x86_64
> > ~~~
> > 
> > However, I do not see why this should be a problem and why we just cannot
> > update the default list of emulated machines on the engine to include the
> > rhel7.1? (unless it is not a static list and actually generated on the fly,
> > I do not believe it is.)
> 
> Please don't do that. It will allow you to add the host all right, but you
> are running a _wrong_ QEMU, the one not supporting fully the 3.6 features
> (notably hotplug ram), not containing bugfixes, not tested by QE.
> 
> If you're fine with that (e.g. testing a new setup not intended for
> production) then feel free to go ahead, but you could as well add it in 3.5
> compatibility mode

Michal, ok. I am 100% with you on the compatibility and testing.
HOWEVER: if vdsm requires that specific package, why didn't it update it when installed?

Note You need to log in before you can comment on or make changes to this bug.