Bug 1272084 - During migration: execution failed: java.lang.Boolean cannot be cast to java.util.Map
During migration: execution failed: java.lang.Boolean cannot be cast to java....
Status: CLOSED CURRENTRELEASE
Product: ovirt-engine
Classification: oVirt
Component: BLL.Virt (Show other bugs)
3.6.0.1
All All
medium Severity low (vote)
: ovirt-3.6.3
: 3.6.3
Assigned To: Milan Zamazal
Israel Pinto
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-15 08:48 EDT by Lukas Svaty
Modified: 2016-02-18 06:12 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-18 06:12:21 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Virt
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑3.6.z+
rule-engine: planning_ack+
rule-engine: devel_ack+
rule-engine: testing_ack+


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 51368 master MERGED core: Fix of MigrateDowntimeSupported option 2016-01-07 03:07 EST
oVirt gerrit 51491 ovirt-engine-3.6 MERGED core: Fix of MigrateDowntimeSupported option 2016-01-07 10:12 EST

  None (edit)
Description Lukas Svaty 2015-10-15 08:48:06 EDT
Description of problem:
During migration of VM  (between RHEL6 and RHEL7 hosts) error messeges appears in engine.log. See additional info:

Version-Release number of selected component (if applicable):
rhevm-3.6.0.1-0.1.el6.noarch

How reproducible:
100%

Steps to Reproduce:
1. Migrate VM
2. Check engine.log

Actual results:
Migration was successful, few error logs in engine.log

Expected results:
No error messages in engine log upon successful migration.

Additional info:
engine.log
(DefaultQuartzScheduler_Worker-24) [] START, MigrateStatusVDSCommand(HostName = yellow, MigrateStatusVDSCommandParameters:{runAsync='true', hostId='4b3c4719-c614-47a2-9eae-20ea4ea1e5e5', vmId='2e333b1f-2fe3-4945-89ac-8c9426bfb32d'}), log id: 14175164
2015-10-15 14:21:25,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [] Failed in 'MigrateStatusVDS' method, for vds: 'yellow'; host: '***.***.***.***': java.lang.Boolean cannot be cast to java.util.Map
2015-10-15 14:21:25,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [] Command 'MigrateStatusVDSCommand(HostName = yellow, MigrateStatusVDSCommandParameters:{runAsync='true', hostId='4b3c4719-c614-47a2-9eae-20ea4ea1e5e5', vmId='2e333b1f-2fe3-4945-89ac-8c9426bfb32d'})' execution failed: java.lang.Boolean cannot be cast to java.util.Map

[ ...omitting few INFO messages... ]

2015-10-15 14:21:41,220 ERROR [org.ovirt.engine.core.vdsbroker.VmsMonitoring] (DefaultQuartzScheduler_Worker-88) [] VM '2e333b1f-2fe3-4945-89ac-8c9426bfb32d' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='cc1552f7-d469-463d-b247-a344706ef13b', vmId='2e333b1f-2fe3-4945-89ac-8c9426bfb32d'}', device='spice', type='GRAPHICS', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}'
Comment 1 Omer Frenkel 2015-10-18 05:06:09 EDT
please attach engine and host logs
Comment 2 Red Hat Bugzilla Rules Engine 2015-10-19 06:51:01 EDT
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 3 Yaniv Lavi 2015-10-29 08:44:49 EDT
In oVirt testing is done on single release by default. Therefore I'm removing the 4.0 flag. If you think this bug must be tested in 4.0 as well, please re-add the flag. Please note we might not have testing resources to handle the 4.0 clone.
Comment 4 Michal Skrivanek 2015-11-13 11:09:32 EST
do you have the same cluster level in both clusters (3.5)?
Comment 5 Michal Skrivanek 2015-11-23 04:25:34 EST
ping?
Comment 6 Lukas Svaty 2015-11-24 06:40:04 EST
yes both hosts are in the same 3.5 clusters, both running rhel-6

libvirt-0.10.2-54.el6_7.2.x86_64
vdsm-4.16.27-1.el6ev.x86_64
Comment 7 Michal Skrivanek 2015-11-24 07:16:50 EST
Lukas, then "During migration of VM  (between RHEL6 and RHEL7 hosts)" doesn't make sense.
So what migration are you actually doing?
Comment 8 Lukas Svaty 2015-11-24 07:26:11 EST
previously it was tested between rhel6 and rhel7 host, not sure about scenario (maybe a mistake in description as it could have been two rhel6? not sure)

scenario from comment#6
using two rhel6 hosts in 3.5 cluster both with packages mentioned in comment#6
I have the environment set up at the moment so I can provide it for investigation
Comment 9 Michal Skrivanek 2015-11-24 07:31:07 EST
(In reply to Lukas Svaty from comment #8)
> previously it was tested between rhel6 and rhel7 host, not sure about
> scenario (maybe a mistake in description as it could have been two rhel6?
> not sure)
> 
> scenario from comment#6
> using two rhel6 hosts in 3.5 cluster both with packages mentioned in
> comment#6
> I have the environment set up at the moment so I can provide it for
> investigation

cool. So does it reproduce? if so and you can provide connection details then it's perfect
Comment 10 Lukas Svaty 2015-11-24 07:46:32 EST
yes, indeed this issue is still reproducible, provided environment connection info via mail
Comment 11 Lukas Svaty 2015-11-26 07:47:08 EST
After some investigation this issue is absent when JSONRPC communication is not enabled on host, thus a problem is most probably within JSONRPC.
Comment 12 Oved Ourfali 2015-11-29 04:04:34 EST
Piotr - can you take a look?
Comment 13 Piotr Kliczewski 2015-11-30 03:38:11 EST
The issue looks like classic marshaling issue for VM.getMigrationStatus. 

From the engine perspective there is "response" parameter which are assumed to be a map and it is causing the issue.

Here are list of patches which touched this part of code:
- engine
https://gerrit.ovirt.org/#/c/40100
- vdsm
https://gerrit.ovirt.org/#/c/39039

This API change was not reviewed by any infra member and it seems to be causing the issue.
Comment 14 Michal Skrivanek 2015-12-04 06:10:50 EST
note: this seem to be happening only to 3.5 hosts with JSON-RPC enabled.

Since it's harmless...just swallow the error on 3.6 engine side, with a comment that we can drop this in 4.0?
Comment 15 Milan Zamazal 2015-12-21 07:25:51 EST
The problem is in mismatch between the migrate downtime option name in the
source code and in the database.  Run the following SQL command in the database
and restart Engine and the problem with the type cast error message is
gone:

update vdc_options set option_name = 'MigrateDowntimeSupported' where option_name = 'MigrateDowntime';

I'm not sure whether it's better to rename the option in the database or in the
source code.  I'll ask Engine developers for opinion.
Comment 16 Israel Pinto 2016-02-08 06:24:42 EST
Verify with:
RHEVM Version: 3.6.3-0.1.el6 
vdsm: vdsm-4.17.19-0.el7ev

scenario:
1. Migration VM
2. Maintenance host
3. Check engine log

Results:
PASS not cast exception found.

Note You need to log in before you can comment on or make changes to this bug.