Created attachment 951372 [details] screenshots, server and vdsm logs Description of problem: Install host with upgraded libvirt rpms (libvirt rpms that support Live Merge)fails on an engine with Rhel6.6. Works fine with on an engine with Rhel6.5 Version-Release number of selected component (if applicable): rhevm-3.5.0-0.17.beta.el6ev.noarch vdsm-4.16.7.1-1.el7.x86_64 How reproducible: Steps to Reproduce: 1. Installed engine with V3.5 with vt7 on host with Rhel6.6 2. Installed vdsm V3.5 vt7 on a host with Rhel7.0 3. Upgraded the libvirt rpms which support live merge for Rhel7.0 4. Tried adding the host to the engine >>>>> the installation failed with "Failed to install Host nott-vds1. Failed to execute stage 'Setup validation': Hardware does not support virtualization." 5. Tried adding the same host to an engine with Rhel6.5 >>>> This works fine Actual results: Failed to install host Expected results: Host should install Additional info: From engine log: ------------------------------------------ 2014-10-28 11:43:04,022 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-4) [3768a15e] START, SetVdsStatusVDSCommand(HostName = nott-vds1, HostId = c2500171-11fb-47ea-bd61-182a83de8bfa, statu s=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5f00e885 2014-10-28 11:43:04,092 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-4) [3768a15e] FINISH, SetVdsStatusVDSCommand, log id: 5f00e885 2014-10-28 11:43:04,418 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-4) [3768a15e] Installation 10.35.161.23: Connected to host 10.35.161.23 with SSH key fingerprint: 20:c4:2f:bf:df:38:e0:ee:87:a7:6 0:35:d3:6c:cc:41 2014-10-28 11:43:04,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-4) [3768a15e] Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Connected to host 10.35.161.23 with SSH key fingerprint: 20:c4:2f:bf:df:38:e0:ee:87:a7:60:35:d3:6c:cc:41. 2014-10-28 11:43:04,801 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-4) [3768a15e] Installation of 10.35.161.23. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXX XXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DI ALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar 2014-10-28 11:43:04,802 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-7-thread-4) Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' is missing, creating 2014-10-28 11:43:05,511 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-4) SSH execute root.161.23 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2014-10-28 11:43:06,952 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Initializing Host: nott-vds1 2014-10-28 11:43:10,775 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Initializing 2014-10-28 11:43:10,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Initializing. 2014-10-28 11:43:10,860 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Environment setup 2014-10-28 11:43:10,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Environment setu p. 2014-10-28 11:43:10,912 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Environment packages setup 2014-10-28 11:43:10,930 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Environment pack ages setup. 2014-10-28 11:43:15,361 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Programs detection 2014-10-28 11:43:15,376 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Programs detecti on. 2014-10-28 11:43:15,391 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Environment customization 2014-10-28 11:43:15,403 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Environment cust omization. 2014-10-28 11:43:15,900 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Kdump supported 2014-10-28 11:43:15,913 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Kdump supported. 2014-10-28 11:43:15,962 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Logs at host located at: '/tmp/ovirt-host-deploy-20141028114307-lus3pj.log' 2014-10-28 11:43:15,970 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Logs at host located at : '/tmp/ovirt-host-deploy-20141028114307-lus3pj.log'. 2014-10-28 11:43:16,130 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.35.161.23 reports unique id E5B0C284-604A-455C-B0EC-66A545EA6595 2014-10-28 11:43:16,245 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id E5B0C284-604A-455C-B0EC-66A545EA6595 to Host 10.35.161.23 2014-10-28 11:43:16,526 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Disabling Kdump integration 2014-10-28 11:43:16,538 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Disabling Kdump integra tion. 2014-10-28 11:43:16,633 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Setup validation 2014-10-28 11:43:16,650 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Setup validation . 2014-10-28 11:43:16,795 WARN [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Machine does not support constant timestamp counter, this may effect performance 2014-10-28 11:43:16,809 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Host nott-vds1 installation in progress . Machine does not support constant timestamp counter, this may effect performance. 2014-10-28 11:43:16,834 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Failed to execute stage 'Setup validation': Hardware does not support virtualization 2014-10-28 11:43:16,841 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host nott-vds1. Failed to execute stage 'Setup validation': Hardware does not support virtualization. 2014-10-28 11:43:16,846 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Pre-termination 2014-10-28 11:43:16,854 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Pre-termination. 2014-10-28 11:43:16,902 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141028114316-10.35.161.23-3768a15e.log' 2014-10-28 11:43:16,915 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141028114316-10.35.161.23-3768a15e.log'. 2014-10-28 11:43:17,259 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.161.23: Stage: Termination 2014-10-28 11:43:17,270 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 3768a15e, Call Stack: null, Custom Event ID: -1, Message: Installing Host nott-vds1. Stage: Termination. 2014-10-28 11:43:17,450 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-4) SSH error running command root.161.23:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True': java.io.IOException: Command returned failure code 1 during SSH session 'root.161.23'
I added a screenshot of the bios configuration which indicates that the Virtualization flag is set
Adam, can you please have a look?
2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware._cpuid:85 cpuid: (2147483656, 1752462657, 1145913699, 1769238117) 2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware._cpu_has_svm_support:106 svm support: False CPUID is 0x80000008 We require minimum of 0x8000000a
Tal, this is not related to storage.
Removing Regression as not sure why it was added.
closing this, per comment#3.
Hi As you can see in the original description - it works fine when the engine is set over rhel 6.5 but fails when it is set over rhel 6.6 It is a bug for sure - If it is supported, it should be fixed on rhel 6.6 engine If not it should be blocked on rhel 6.5 engine
and by the way, we are using this host for years - so regression.
I fail to understand. same server - you install rhel-6.5 - works, then you reinstall server with rhel-6.6 - and it does not work? can you please confirm by actually trying? if so this is regression of rhel-6.6, but we will discuss this later.
(In reply to Alon Bar-Lev from comment #9) > I fail to understand. > > same server - you install rhel-6.5 - works, then you reinstall server with > rhel-6.6 - and it does not work? > > can you please confirm by actually trying? if so this is regression of > rhel-6.6, but we will discuss this later. The host installed with rhel 7 When trying to add it to engine over rhel 6.5 - works When trying to add the same host to engine over rhel 6.6 - it fails.
(In reply to Aharon Canan from comment #10) > (In reply to Alon Bar-Lev from comment #9) > > I fail to understand. > > > > same server - you install rhel-6.5 - works, then you reinstall server with > > rhel-6.6 - and it does not work? > > > > can you please confirm by actually trying? if so this is regression of > > rhel-6.6, but we will discuss this later. > > The host installed with rhel 7 > When trying to add it to engine over rhel 6.5 - works > When trying to add the same host to engine over rhel 6.6 - it fails. logs of and version of ovirt-host-deploy for each please... not sure why are not provided when a problem is reported. the code runs on the host not on engine so the engine platform should not be relevant at all.
Kevin - please provide
Created attachment 965562 [details] host-deploy log added again Adding the ovirt host-deploy file again. Version-Release number of selected component (if applicable): rhevm-3.5.0-0.17.beta.el6ev.noarch vdsm-4.16.7.1-1.el7.x86_64
ovirt-host-deploy-1.3.0-1.el6ev - not working 2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware._cpuid:85 cpuid: (2147483656, 1752462657, 1145913699, 1769238117) 2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware._cpu_has_svm_support:106 svm support: False 2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware._isVirtualizationEnabled:186 virtualization support AuthenticAMD (cpu: False, bios: True) 2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware hardware.detect:203 Hardware does not support virtualization please attach as requested several times the working rhel-6.5 version and log, on same host with no change on host's hardware/software with different engine.
Created attachment 965712 [details] rhel6.5 engine log added I am adding the engine log from the Rhel6.5 engine that I was successfully able to add the Rhel7.0 Host. That engine was running: rhel6.5 rhev3.5 vt7 rhevm-3.5.0-0.17.beta.el6ev.noarch
Kevin, if I do not get your cooperation we will never solve this issue and this bug will be closed as INSUFFICIENT_DATA. Per comment#10 it is claimed that if you follow the following sequence then we have a regression. 1. install rhel-7 host on a specific host of QA. 2. install engine on rhel-6.5, try to deploy on (1) and it succeeds. 3. install engine on rhel-6.6, try to deploy on (1) without any change and it fails. Please reproduce the EXACT above sequence from scratch, install everything from vanilla repositories, and attach host-deploy logs. If I do not see a fresh installation, or I see that hosts used in (2) and (3) are different no progress will be made. Thank you.
(In reply to Alon Bar-Lev from comment #3) > 2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware > hardware._cpuid:85 cpuid: (2147483656, 1752462657, 1145913699, 1769238117) > 2014-10-28 11:43:16 DEBUG otopi.ovirt_host_deploy.hardware > hardware._cpu_has_svm_support:106 svm support: False > > CPUID is 0x80000008 > We require minimum of 0x8000000a from reading above, it sounds like the host should be blocked and we shouldn't be able to use it. I am not sure how (probably bug) but we are using it for years and even now it is part of RHEV setup. Checking and reproducing the flows from comment #16 is not relevant. New host deploy logs of this host attached, please check how it is working - this is our problem according to you. Also, you can check things live - engine - 10.35.161.23 host - 10.35.115.10
Created attachment 966164 [details] host-deploy
attachment#965712 [details] 2014-10-28 11:43:10 DEBUG otopi.plugins.otopi.system.info info._init:52 distribution ('Red Hat Enterprise Linux Server', '6.6', 'Santiago') 2014-10-28 11:43:10 DEBUG otopi.plugins.otopi.system.info info._init:53 host 'kevin-rhevm.scl.lab.tlv.redhat.com' attachment#966164 [details] 2014-12-08 14:59:32 DEBUG otopi.plugins.otopi.system.info info._init:52 distribution ('Red Hat Enterprise Linux Server', '7.0', 'Maipo') 2014-12-08 14:59:32 DEBUG otopi.plugins.otopi.system.info info._init:53 host 'nott-vds1.qa.lab.tlv.redhat.com' two different hosts, two different kernels, still trying to figure out how the same code running on same host produces different results. I am closing this for now.
(In reply to Alon Bar-Lev from comment #19) > attachment#965712 [details] > > 2014-10-28 11:43:10 DEBUG otopi.plugins.otopi.system.info info._init:52 > distribution ('Red Hat Enterprise Linux Server', '6.6', 'Santiago') > 2014-10-28 11:43:10 DEBUG otopi.plugins.otopi.system.info info._init:53 host > 'kevin-rhevm.scl.lab.tlv.redhat.com' > > attachment#966164 [details] > > 2014-12-08 14:59:32 DEBUG otopi.plugins.otopi.system.info info._init:52 > distribution ('Red Hat Enterprise Linux Server', '7.0', 'Maipo') > 2014-12-08 14:59:32 DEBUG otopi.plugins.otopi.system.info info._init:53 host > 'nott-vds1.qa.lab.tlv.redhat.com' > > two different hosts, two different kernels, still trying to figure out how > the same code running on same host produces different results. > > I am closing this for now. If you are still trying to figure out then is should remain open till you figure out afaik. Anyway - What info is missing? we have the host up and running (Ip provided in comment #17), the logs (attached) and the setup running. did you connect the host directly and checked the flag and how i is connected although it shouldn't? I think it will be much better if we will talk (like Kevin already ask) and close gaps instead of "playing" with the bug status.
I am not trying to figure out anything... what you claim does not making any sense I left it open just to be polite. You do not provide the data I requested, you do not try to reproduce your claim, I cannot help you. Please reopen when you have new data as explicitly requested several time, do not reopen this in any other case.