Bug 1158050 - Host install fails due to apparent unsupported Virtualization flag
Summary: Host install fails due to apparent unsupported Virtualization flag
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: ovirt-host-deploy
Classification: oVirt
Component: Plugins.VDSM
Version: 1.3.0
Hardware: x86_64
OS: Unspecified
unspecified
urgent vote
Target Milestone: ---
: ---
Assignee: Alon Bar-Lev
QA Contact: sefi litmanovich
URL:
Whiteboard: infra
Depends On:
Blocks: rhev35rcblocker rhev35gablocker
TreeView+ depends on / blocked
 
Reported: 2014-10-28 12:46 UTC by Kevin Alon Goldblatt
Modified: 2016-02-10 19:16 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-12-09 12:41:08 UTC
oVirt Team: Infra
gklein: blocker?
kgoldbla: devel_ack?


Attachments (Terms of Use)
screenshots, server and vdsm logs (394.33 KB, application/x-gzip)
2014-10-28 12:46 UTC, Kevin Alon Goldblatt
no flags Details
host-deploy log added again (170.38 KB, text/plain)
2014-12-07 13:33 UTC, Kevin Alon Goldblatt
no flags Details
rhel6.5 engine log added (1.06 MB, text/plain)
2014-12-08 07:21 UTC, Kevin Alon Goldblatt
no flags Details
host-deploy (344.63 KB, text/plain)
2014-12-09 09:46 UTC, Aharon Canan
no flags Details

Description Kevin Alon Goldblatt 2014-10-28 12:46:53 UTC
Created attachment 951372 [details]
screenshots, server and vdsm logs

Description of problem:
Install host with upgraded libvirt rpms (libvirt rpms that support Live Merge)fails on an engine with Rhel6.6. Works fine with on an engine with Rhel6.5

Version-Release number of selected component (if applicable):
rhevm-3.5.0-0.17.beta.el6ev.noarch
vdsm-4.16.7.1-1.el7.x86_64

How reproducible:


Steps to Reproduce:
1. Installed engine with V3.5 with vt7 on host with Rhel6.6
2. Installed vdsm V3.5 vt7 on a host with Rhel7.0
3. Upgraded the libvirt rpms which support live merge for Rhel7.0
4. Tried adding the host to the engine >>>>> the installation failed with 
"Failed to install Host nott-vds1. Failed to execute stage 'Setup validation': Hardware does not support virtualization."
5. Tried adding the same host to an engine with Rhel6.5 >>>> This works fine


Actual results:
Failed to install host


Expected results:
Host should install

Additional info:
From engine log:
------------------------------------------
2014-10-28 11:43:04,022 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-4) [3768a15e] START, SetVdsStatusVDSCommand(HostName = nott-vds1, HostId = c2500171-11fb-47ea-bd61-182a83de8bfa, statu
s=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5f00e885
2014-10-28 11:43:04,092 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-4) [3768a15e] FINISH, SetVdsStatusVDSCommand, log id: 5f00e885
2014-10-28 11:43:04,418 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-4) [3768a15e] Installation 10.35.161.23: Connected to host 10.35.161.23 with SSH key fingerprint: 20:c4:2f:bf:df:38:e0:ee:87:a7:6
0:35:d3:6c:cc:41
2014-10-28 11:43:04,513 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-4) [3768a15e] Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host 
nott-vds1. Connected to host 10.35.161.23 with SSH key fingerprint: 20:c4:2f:bf:df:38:e0:ee:87:a7:60:35:d3:6c:cc:41.
2014-10-28 11:43:04,801 INFO  [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-4) [3768a15e] Installation of 10.35.161.23. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXX
XXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/setup DIALOG/dialect=str:machine DI
ALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar
2014-10-28 11:43:04,802 INFO  [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-7-thread-4) Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' is missing, creating
2014-10-28 11:43:05,511 INFO  [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-4) SSH execute root@10.35.161.23 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX
 \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True'
2014-10-28 11:43:06,952 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Initializing Host: nott-vds1
2014-10-28 11:43:10,775 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Initializing
2014-10-28 11:43:10,792 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Initializing.
2014-10-28 11:43:10,860 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Environment setup
2014-10-28 11:43:10,876 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Environment setu
p.
2014-10-28 11:43:10,912 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Environment packages setup
2014-10-28 11:43:10,930 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Environment pack
ages setup.
2014-10-28 11:43:15,361 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Programs detection
2014-10-28 11:43:15,376 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Programs detecti
on.
2014-10-28 11:43:15,391 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Environment customization
2014-10-28 11:43:15,403 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Environment cust
omization.
2014-10-28 11:43:15,900 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Kdump supported
2014-10-28 11:43:15,913 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Kdump supported.
2014-10-28 11:43:15,962 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Logs at host located at: '/tmp/ovirt-host-deploy-20141028114307-lus3pj.log'
2014-10-28 11:43:15,970 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Logs at host located at
: '/tmp/ovirt-host-deploy-20141028114307-lus3pj.log'.
2014-10-28 11:43:16,130 INFO  [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.35.161.23 reports unique id E5B0C284-604A-455C-B0EC-66A545EA6595
2014-10-28 11:43:16,245 INFO  [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id E5B0C284-604A-455C-B0EC-66A545EA6595 to Host 10.35.161.23
2014-10-28 11:43:16,526 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Disabling Kdump integration
2014-10-28 11:43:16,538 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Disabling Kdump integra
tion.
2014-10-28 11:43:16,633 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Setup validation
2014-10-28 11:43:16,650 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Setup validation
.
2014-10-28 11:43:16,795 WARN  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Machine does not support constant timestamp counter, this may effect performance
2014-10-28 11:43:16,809 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Host nott-vds1 installation in progress . Machine 
does not support constant timestamp counter, this may effect performance.
2014-10-28 11:43:16,834 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Failed to execute stage 'Setup validation': Hardware does not support virtualization
2014-10-28 11:43:16,841 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host nott-vds1. Failed to execute stage 'Setup validation': Hardware does not support virtualization.
2014-10-28 11:43:16,846 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Pre-termination
2014-10-28 11:43:16,854 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Pre-termination.
2014-10-28 11:43:16,902 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141028114316-10.35.161.23-3768a15e.log'
2014-10-28 11:43:16,915 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141028114316-10.35.161.23-3768a15e.log'.
2014-10-28 11:43:17,259 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Termination
2014-10-28 11:43:17,270 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Termination.
2014-10-28 11:43:17,450 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-4) SSH error running command root@10.35.161.23:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True': java.io.IOException: Command returned failure code 1 during SSH session 'root@10.35.161.23'

Comment 1 Kevin Alon Goldblatt 2014-10-28 12:47:46 UTC
I added a screenshot of the bios configuration which indicates that the Virtualization flag is set

Comment 2 Tal Nisan 2014-10-29 12:50:08 UTC
Adam, can you please have a look?

Comment 3 Alon Bar-Lev 2014-11-09 12:26:28 UTC
2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware._cpuid:85 cpuid: (2147483656, 1752462657, 1145913699, 1769238117)
2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware._cpu_has_svm_support:106 svm support: False

CPUID is 0x80000008
We require minimum of 0x8000000a

Comment 4 Nir Soffer 2014-11-11 09:29:00 UTC
Tal, this is not related to storage.

Comment 5 Alon Bar-Lev 2014-11-11 09:33:27 UTC
Removing Regression as not sure why it was added.

Comment 6 Alon Bar-Lev 2014-11-14 18:29:37 UTC
closing this, per comment#3.

Comment 7 Aharon Canan 2014-12-04 13:48:00 UTC
Hi

As you can see in the original description - it works fine when the engine is set over rhel 6.5 but fails when it is set over rhel 6.6

It is a bug for sure - 
If it is supported, it should be fixed on rhel 6.6 engine 
If not it should be blocked on rhel 6.5 engine

Comment 8 Aharon Canan 2014-12-04 13:49:31 UTC
and by the way, we are using this host for years - so regression.

Comment 9 Alon Bar-Lev 2014-12-04 13:57:32 UTC
I fail to understand.

same server - you install rhel-6.5 - works, then you reinstall server with rhel-6.6 - and it does not work?

can you please confirm by actually trying? if so this is regression of rhel-6.6, but we will discuss this later.

Comment 10 Aharon Canan 2014-12-04 14:00:24 UTC
(In reply to Alon Bar-Lev from comment #9)
> I fail to understand.
> 
> same server - you install rhel-6.5 - works, then you reinstall server with
> rhel-6.6 - and it does not work?
> 
> can you please confirm by actually trying? if so this is regression of
> rhel-6.6, but we will discuss this later.

The host installed with rhel 7
When trying to add it to engine over rhel 6.5 - works
When trying to add the same host to engine over rhel 6.6 - it fails.

Comment 11 Alon Bar-Lev 2014-12-04 14:05:35 UTC
(In reply to Aharon Canan from comment #10)
> (In reply to Alon Bar-Lev from comment #9)
> > I fail to understand.
> > 
> > same server - you install rhel-6.5 - works, then you reinstall server with
> > rhel-6.6 - and it does not work?
> > 
> > can you please confirm by actually trying? if so this is regression of
> > rhel-6.6, but we will discuss this later.
> 
> The host installed with rhel 7
> When trying to add it to engine over rhel 6.5 - works
> When trying to add the same host to engine over rhel 6.6 - it fails.

logs of and version of ovirt-host-deploy for each please... not sure why are not provided when a problem is reported.

the code runs on the host not on engine so the engine platform should not be relevant at all.

Comment 12 Aharon Canan 2014-12-04 14:43:14 UTC
Kevin - please provide

Comment 13 Kevin Alon Goldblatt 2014-12-07 13:33:00 UTC
Created attachment 965562 [details]
host-deploy log added again

Adding the ovirt host-deploy file again.

Version-Release number of selected component (if applicable):
rhevm-3.5.0-0.17.beta.el6ev.noarch
vdsm-4.16.7.1-1.el7.x86_64

Comment 14 Alon Bar-Lev 2014-12-07 14:05:39 UTC
ovirt-host-deploy-1.3.0-1.el6ev - not working

2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware._cpuid:85 cpuid: (2147483656, 1752462657, 1145913699, 1769238117)
2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware._cpu_has_svm_support:106 svm support: False
2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware._isVirtualizationEnabled:186 virtualization support AuthenticAMD (cpu: False, bios: True)
2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware.detect:203 Hardware does not support virtualization

please attach as requested several times the working rhel-6.5 version and log, on same host with no change on host's hardware/software with different engine.

Comment 15 Kevin Alon Goldblatt 2014-12-08 07:21:34 UTC
Created attachment 965712 [details]
rhel6.5 engine log added

I am adding the engine log from the Rhel6.5 engine that I was successfully able to add the Rhel7.0 Host. That engine was running:

rhel6.5
rhev3.5 vt7
rhevm-3.5.0-0.17.beta.el6ev.noarch

Comment 16 Alon Bar-Lev 2014-12-08 07:29:18 UTC
Kevin, if I do not get your cooperation we will never solve this issue and this bug will be closed as INSUFFICIENT_DATA.

Per comment#10 it is claimed that if you follow the following sequence then we have a regression.

1. install rhel-7 host on a specific host of QA.
2. install engine on rhel-6.5, try to deploy on (1) and it succeeds.
3. install engine on rhel-6.6, try to deploy on (1) without any change and it fails.

Please reproduce the EXACT above sequence from scratch, install everything from vanilla repositories, and attach host-deploy logs.

If I do not see a fresh installation, or I see that hosts  used in (2) and (3) are different no progress will be made.

Thank you.

Comment 17 Aharon Canan 2014-12-09 09:46:01 UTC
(In reply to Alon Bar-Lev from comment #3)
> 2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware
> hardware._cpuid:85 cpuid: (2147483656, 1752462657, 1145913699, 1769238117)
> 2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware
> hardware._cpu_has_svm_support:106 svm support: False
> 
> CPUID is 0x80000008
> We require minimum of 0x8000000a

from reading above, it sounds like the host should be blocked and we shouldn't be able to use it.

I am not sure how (probably bug) but we are using it for years and even now it is part of RHEV setup.

Checking and reproducing the flows from comment #16 is not relevant.

New host deploy logs of this host attached, please check how it is working - this is our problem according to you.

Also, you can check things live - 
engine - 10.35.161.23
host - 10.35.115.10

Comment 18 Aharon Canan 2014-12-09 09:46:31 UTC
Created attachment 966164 [details]
host-deploy

Comment 19 Alon Bar-Lev 2014-12-09 12:29:09 UTC
attachment#965712 [details]

2014-10-28 11:43:10 DEBUG otopi.plugins.otopi.system.info info._init:52 distribution ('Red Hat Enterprise Linux Server', '6.6', 'Santiago')
2014-10-28 11:43:10 DEBUG otopi.plugins.otopi.system.info info._init:53 host 'kevin-rhevm.scl.lab.tlv.redhat.com'

attachment#966164 [details]

2014-12-08 14:59:32 DEBUG otopi.plugins.otopi.system.info info._init:52 distribution ('Red Hat Enterprise Linux Server', '7.0', 'Maipo')
2014-12-08 14:59:32 DEBUG otopi.plugins.otopi.system.info info._init:53 host 'nott-vds1.qa.lab.tlv.redhat.com'

two different hosts, two different kernels, still trying to figure out how the same code running on same host produces different results.

I am closing this for now.

Comment 20 Aharon Canan 2014-12-09 12:36:25 UTC
(In reply to Alon Bar-Lev from comment #19)
> attachment#965712 [details]
> 
> 2014-10-28 11:43:10 DEBUG otopi.plugins.otopi.system.info info._init:52
> distribution ('Red Hat Enterprise Linux Server', '6.6', 'Santiago')
> 2014-10-28 11:43:10 DEBUG otopi.plugins.otopi.system.info info._init:53 host
> 'kevin-rhevm.scl.lab.tlv.redhat.com'
> 
> attachment#966164 [details]
> 
> 2014-12-08 14:59:32 DEBUG otopi.plugins.otopi.system.info info._init:52
> distribution ('Red Hat Enterprise Linux Server', '7.0', 'Maipo')
> 2014-12-08 14:59:32 DEBUG otopi.plugins.otopi.system.info info._init:53 host
> 'nott-vds1.qa.lab.tlv.redhat.com'
> 
> two different hosts, two different kernels, still trying to figure out how
> the same code running on same host produces different results.
> 
> I am closing this for now.

If you are still trying to figure out then is should remain open till you figure out afaik.

Anyway - 
What info is missing? we have the host up and running (Ip provided in comment #17), the logs (attached) and the setup running.
did you connect the host directly and checked the flag and how i is connected although it shouldn't?

I think it will be much better if we will talk (like Kevin already ask) and close gaps instead of "playing" with the bug status.

Comment 21 Alon Bar-Lev 2014-12-09 12:41:08 UTC
I am not trying to figure out anything... what you claim does not making any sense I left it open just to be polite.

You do not provide the data I requested, you do not try to reproduce your claim, I cannot help you.

Please reopen when you have new data as explicitly requested several time, do not reopen this in any other case.


Note You need to log in before you can comment on or make changes to this bug.