Description of problem: During migration of VM (between RHEL6 and RHEL7 hosts) error messeges appears in engine.log. See additional info: Version-Release number of selected component (if applicable): rhevm-3.6.0.1-0.1.el6.noarch How reproducible: 100% Steps to Reproduce: 1. Migrate VM 2. Check engine.log Actual results: Migration was successful, few error logs in engine.log Expected results: No error messages in engine log upon successful migration. Additional info: engine.log (DefaultQuartzScheduler_Worker-24) [] START, MigrateStatusVDSCommand(HostName = yellow, MigrateStatusVDSCommandParameters:{runAsync='true', hostId='4b3c4719-c614-47a2-9eae-20ea4ea1e5e5', vmId='2e333b1f-2fe3-4945-89ac-8c9426bfb32d'}), log id: 14175164 2015-10-15 14:21:25,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [] Failed in 'MigrateStatusVDS' method, for vds: 'yellow'; host: '***.***.***.***': java.lang.Boolean cannot be cast to java.util.Map 2015-10-15 14:21:25,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [] Command 'MigrateStatusVDSCommand(HostName = yellow, MigrateStatusVDSCommandParameters:{runAsync='true', hostId='4b3c4719-c614-47a2-9eae-20ea4ea1e5e5', vmId='2e333b1f-2fe3-4945-89ac-8c9426bfb32d'})' execution failed: java.lang.Boolean cannot be cast to java.util.Map [ ...omitting few INFO messages... ] 2015-10-15 14:21:41,220 ERROR [org.ovirt.engine.core.vdsbroker.VmsMonitoring] (DefaultQuartzScheduler_Worker-88) [] VM '2e333b1f-2fe3-4945-89ac-8c9426bfb32d' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='cc1552f7-d469-463d-b247-a344706ef13b', vmId='2e333b1f-2fe3-4945-89ac-8c9426bfb32d'}', device='spice', type='GRAPHICS', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}'
please attach engine and host logs
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
In oVirt testing is done on single release by default. Therefore I'm removing the 4.0 flag. If you think this bug must be tested in 4.0 as well, please re-add the flag. Please note we might not have testing resources to handle the 4.0 clone.
do you have the same cluster level in both clusters (3.5)?
ping?
yes both hosts are in the same 3.5 clusters, both running rhel-6 libvirt-0.10.2-54.el6_7.2.x86_64 vdsm-4.16.27-1.el6ev.x86_64
Lukas, then "During migration of VM (between RHEL6 and RHEL7 hosts)" doesn't make sense. So what migration are you actually doing?
previously it was tested between rhel6 and rhel7 host, not sure about scenario (maybe a mistake in description as it could have been two rhel6? not sure) scenario from comment#6 using two rhel6 hosts in 3.5 cluster both with packages mentioned in comment#6 I have the environment set up at the moment so I can provide it for investigation
(In reply to Lukas Svaty from comment #8) > previously it was tested between rhel6 and rhel7 host, not sure about > scenario (maybe a mistake in description as it could have been two rhel6? > not sure) > > scenario from comment#6 > using two rhel6 hosts in 3.5 cluster both with packages mentioned in > comment#6 > I have the environment set up at the moment so I can provide it for > investigation cool. So does it reproduce? if so and you can provide connection details then it's perfect
yes, indeed this issue is still reproducible, provided environment connection info via mail
After some investigation this issue is absent when JSONRPC communication is not enabled on host, thus a problem is most probably within JSONRPC.
Piotr - can you take a look?
The issue looks like classic marshaling issue for VM.getMigrationStatus. From the engine perspective there is "response" parameter which are assumed to be a map and it is causing the issue. Here are list of patches which touched this part of code: - engine https://gerrit.ovirt.org/#/c/40100 - vdsm https://gerrit.ovirt.org/#/c/39039 This API change was not reviewed by any infra member and it seems to be causing the issue.
note: this seem to be happening only to 3.5 hosts with JSON-RPC enabled. Since it's harmless...just swallow the error on 3.6 engine side, with a comment that we can drop this in 4.0?
The problem is in mismatch between the migrate downtime option name in the source code and in the database. Run the following SQL command in the database and restart Engine and the problem with the type cast error message is gone: update vdc_options set option_name = 'MigrateDowntimeSupported' where option_name = 'MigrateDowntime'; I'm not sure whether it's better to rename the option in the database or in the source code. I'll ask Engine developers for opinion.
Verify with: RHEVM Version: 3.6.3-0.1.el6 vdsm: vdsm-4.17.19-0.el7ev scenario: 1. Migration VM 2. Maintenance host 3. Check engine log Results: PASS not cast exception found.