Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1057275

Summary: Moving 3.3 host to maintainace after trying to move to 3.4 cluster leave host unusable
Product: [Retired] oVirt Reporter: Nir Soffer <nsoffer>
Component: ovirt-engine-coreAssignee: Mooli Tayer <mtayer>
Status: CLOSED DUPLICATE QA Contact: bugs <bugs>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.4CC: acathrow, bazulay, emesika, gklein, iheim, masayag, nsoffer, yeylon
Target Milestone: ---   
Target Release: 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-23 15:42:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine and vdsm logs none

Description Nir Soffer 2014-01-23 18:28:00 UTC 
Description of problem:

After moving host to Maintenance, host stuck in Preparing For Maintenance state or Down state, not usable.

Version-Release number of selected component (if applicable):

ovirt-engine: 3.4.0_beta1
vdsm: vdsm-4.13.2-1.el6

How reproducible:

Tried once.

Steps to Reproduce:
1. Checkout ovirt-3.4 beta1 tag from master and build and install ovirt-engine
2. Reinstall host in 3.3 cluster and wait until host is up
3. Move host to Maintenance (succeeds)
4. Create 3.4 cluster
5. Edit host and move it to new 3.4 cluster
6. Host get a red x icon and we see message in the ui log:
   Host jumbo is compatible with versions (3.0,3.1,3.2,3.3) and cannot join Cluster sasha-cluster-3.4 which is set to version 3.4.
7. Move host to Maintenance
8. Host never finish the move to Maintenance still in Preparing For Maintenance state
9. Restart ovirt-engine and vdsm
10. Confirm hos was rebooted
11. Host move to Down state, and cannot be activated

Actual results:

Host unusable, cannot remove it, cannot move to other cluster.

Expected results:

Host move to Maintenance, able to move it back to 3.3 cluster.

Additional info:

In engine log, we see this error many times:

2014-01-23 20:23:50,038 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) HostName = jumbo
2014-01-23 20:23:50,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Failed in GetCapabilitiesVDS method, for vds: jumbo; host: dhcp-3-151
2014-01-23 20:23:50,039 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = jumbo, HostId = afcb58e9-0330-4fe5-8b8c-caab35dbb9ed, vds=Host[jumbo]) execution failed. Exception: ClassCastException: java.lang.String cannot be cast to java.util.Map
2014-01-23 20:23:50,041 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Failed to refresh VDS , vds = afcb58e9-0330-4fe5-8b8c-caab35dbb9ed : jumbo, error = java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map, continuing.: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map
	at org.ovirt.engine.core.vdsbroker.vdsbroker.NetworkQosMapper.deserialize(NetworkQosMapper.java:31) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder.updateNetworkDetailsInInterface(VdsBrokerObjectsBuilder.java:1261) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder.addHostNetworksAndUpdateInterfaces(VdsBrokerObjectsBuilder.java:970) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder.updateNetworkData(VdsBrokerObjectsBuilder.java:940) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder.updateVDSDynamicData(VdsBrokerObjectsBuilder.java:326) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:17) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) [dal.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:537) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:335) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
	at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) [:1.7.0_51]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
Comment 1 Nir Soffer 2014-01-23 18:30:27 UTC 
Created attachment 854557 [details]
engine and vdsm logs
Comment 2 Itamar Heim 2014-01-26 08:11:13 UTC 
Setting target release to current version for consideration and review. please
do not push non-RFE bugs to an undefined target release to make sure bugs are
reviewed for relevancy, fix, closure, etc.
Comment 3 Nir Soffer 2014-01-30 15:21:04 UTC 
Workardound: remove vdsm 3.4, install vdsm from ovirt 3.3 branch. Host moves now to maintainance mode and can be moved back into 3.3 cluster.
Comment 4 Barak 2014-02-23 10:03:07 UTC 
(In reply to Nir Soffer from comment #0)
> Description of problem:
> 
> After moving host to Maintenance, host stuck in Preparing For Maintenance
> state or Down state, not usable.
> 
> Version-Release number of selected component (if applicable):
> 
> ovirt-engine: 3.4.0_beta1
> vdsm: vdsm-4.13.2-1.el6
> 
> How reproducible:
> 
> Tried once.
> 
> Steps to Reproduce:
> 1. Checkout ovirt-3.4 beta1 tag from master and build and install
> ovirt-engine
> 2. Reinstall host in 3.3 cluster and wait until host is up

I assume the host was installed also from 3.4 repos ?


> 3. Move host to Maintenance (succeeds)
> 4. Create 3.4 cluster
> 5. Edit host and move it to new 3.4 cluster
> 6. Host get a red x icon and we see message in the ui log:
>    Host jumbo is compatible with versions (3.0,3.1,3.2,3.3) and cannot join
> Cluster sasha-cluster-3.4 which is set to version 3.4.
> 7. Move host to Maintenance
> 8. Host never finish the move to Maintenance still in Preparing For
> Maintenance state
> 9. Restart ovirt-engine and vdsm
> 10. Confirm hos was rebooted
> 11. Host move to Down state, and cannot be activated
>
Comment 5 Nir Soffer 2014-02-23 11:34:49 UTC 
(In reply to Barak from comment #4)
> (In reply to Nir Soffer from comment #0)
> > Description of problem:
> > 
> > After moving host to Maintenance, host stuck in Preparing For Maintenance
> > state or Down state, not usable.
> > 
> > Version-Release number of selected component (if applicable):
> > 
> > ovirt-engine: 3.4.0_beta1
> > vdsm: vdsm-4.13.2-1.el6
> > 
> > How reproducible:
> > 
> > Tried once.
> > 
> > Steps to Reproduce:
> > 1. Checkout ovirt-3.4 beta1 tag from master and build and install
> > ovirt-engine
> > 2. Reinstall host in 3.3 cluster and wait until host is up
> 
> I assume the host was installed also from 3.4 repos ?

I cannot tell that now, but you can verify by vdsm version.

Anyway according to the engine log it is clear that engine is not handling the response from vdsm gracefully. Engine must cope with different versions of vdsm because the administrator has no control of vdsm version on the hosts; yum update will get you newer vdsm version regardless of your engine version.
Comment 6 Barak 2014-02-23 15:31:21 UTC 
Moti,

Please see stack trace above,

2014-01-23 20:23:50,041 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Failed to refresh VDS , vds = afcb58e9-0330-4fe5-8b8c-caab35dbb9ed : jumbo, error = java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map, continuing.: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map
	at org.ovirt.engine.core.vdsbroker.vdsbroker.NetworkQosMapper.deserialize(NetworkQosMapper.java:31) [vdsbroker.jar:]


Can you please take a look ?
Comment 7 Moti Asayag 2014-02-23 15:42:42 UTC 

*** This bug has been marked as a duplicate of bug 1056155 ***