Bug 1015887 - Migration status verb fails on engine side due to change in response format
Migration status verb fails on engine side due to change in response format
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.3.0
x86_64 Linux
high Severity high
: ---
: 3.3.0
Assigned To: Vinzenz Feenstra [evilissimo]
Pavel Novotny
virt
:
Depends On:
Blocks: 970645 3.3snap3
  Show dependency treegraph
 
Reported: 2013-10-06 10:12 EDT by GenadiC
Modified: 2014-01-21 11:17 EST (History)
11 users (show)

See Also:
Fixed In Version: is25
Doc Type: Bug Fix
Doc Text:
Cancelling a migration returned an error code in the wrong format. It has now been fixed, the error message now appears in the format of {'status': {'message': 'Migration canceled', 'code': 47}, 'progress': 1}
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-21 11:17:26 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
engine log (943.39 KB, text/x-log)
2013-10-06 10:12 EDT, GenadiC
no flags Details
origin vdsm log (11.80 MB, text/x-log)
2013-10-06 10:13 EDT, GenadiC
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 21250 None None None Never
oVirt gerrit 21700 None None None Never

  None (edit)
Description GenadiC 2013-10-06 10:12:07 EDT
Created attachment 808522 [details]
engine log

Description of problem:
Network migration fails on "Command MigrateStatusVDS execution failed. Exception: ClassCastException: java.util.HashMap cannot be cast to java.lang.Integer" when changing MTU to value different from default (9000 or 1480)

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create Network with MTU of 1480
2. Configure it as Display network and attach to Hosts with static IPs
3. Try to perform migration through that network

Actual results:
Migration fails after Timeout

Expected results:
Migration should work as expected over migration network

Additional info:
2013-10-06 16:54:12,519 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Rerun vm 0dcf947f-85aa-43a4-864b-d4abb71a1a5f. Called from vds silver-vdsb.qa.lab.tlv.redhat.com
2013-10-06 16:54:12,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-49) START, MigrateStatusVDSCommand(HostName = silver-vdsb.qa.lab.tlv.redhat.com, HostId = e937868c-88db-48c6-b714-295c52c2f798, vmId=0dcf947f-85aa-43a4-864b-d4abb71a1a5f), log id: 333c38cc
2013-10-06 16:54:12,530 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-49) Failed in MigrateStatusVDS method, for vds: silver-vdsb.qa.lab.tlv.redhat.com; host: 10.35.128.12
2013-10-06 16:54:12,531 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-49) Command MigrateStatusVDS execution failed. Exception: ClassCastException: java.util.HashMap cannot be cast to java.lang.Integer
2013-10-06 16:54:12,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-49) FINISH, MigrateStatusVDSCommand, log id: 333c38cc
2013-10-06 16:54:12,535 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-49) Correlation ID: 2dc985a1, Job ID: 392b15b2-2bab-43e8-a26d-557e6eb67d69, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Internal Engine Error. Trying to migrate to another Host (VM: VMTest1, Source: silver-vdsb.qa.lab.tlv.redhat.com, Destination: silver-vdsa.qa.lab.tlv.redhat.com).
2013-10-06 16:54:12,552 WARN  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-49) CanDoAction of action MigrateVm failed. Reasons:VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM
2013-10-06 16:54:12,552 INFO  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-49) Lock freed to object EngineLock [exclusiveLocks= key: 0dcf947f-85aa-43a4-864b-d4abb71a1a5f value: VM
, sharedLocks= ]
2013-10-06 16:54:12,557 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-49) Correlation ID: 2dc985a1, Job ID: 392b15b2-2bab-43e8-a26d-557e6eb67d69, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Internal Engine Error (VM: VMTest1, Source: silver-vdsb.qa.lab.tlv.redhat.com, Destination: silver-vdsa.qa.lab.tlv.redhat.com).
Comment 1 GenadiC 2013-10-06 10:13:35 EDT
Created attachment 808523 [details]
origin vdsm log
Comment 2 Moti Asayag 2013-10-10 10:03:22 EDT
vdsm version: vdsm-4.13.0-0.2.beta1.el6ev.x86_64

The error on engine.log:

"Command MigrateStatusVDS execution failed. Exception: ClassCastException: java.util.HashMap cannot be cast to java.lang.Integer" 

appears since the returned status from VSDM for migrateStatus verb was changed in http://gerrit.ovirt.org/#/c/11913/3/vdsm/vm.py:

Thread-9603::DEBUG::2013-10-10 14:36:12,042::BindingXMLRPC::981::vds::(wrapper) return vmGetMigrationStatus with {'status': {'message': 'Migration canceled', 'code': {'status': {'message': 
'Migration not in progress', 'code': 47}}}, 'progress': 1}

While the return value used to be simpler as status which contains message, code and progress.
Comment 3 Vinzenz Feenstra [evilissimo] 2013-11-14 02:53:28 EST
(In reply to Moti Asayag from comment #2)
> vdsm version: vdsm-4.13.0-0.2.beta1.el6ev.x86_64
> 
> The error on engine.log:
> 
> "Command MigrateStatusVDS execution failed. Exception: ClassCastException:
> java.util.HashMap cannot be cast to java.lang.Integer" 
> 
> appears since the returned status from VSDM for migrateStatus verb was
> changed in http://gerrit.ovirt.org/#/c/11913/3/vdsm/vm.py:
This bug fix did not introduce this issue
> 
> Thread-9603::DEBUG::2013-10-10
> 14:36:12,042::BindingXMLRPC::981::vds::(wrapper) return vmGetMigrationStatus
> with {'status': {'message': 'Migration canceled', 'code': {'status':
> {'message': 
> 'Migration not in progress', 'code': 47}}}, 'progress': 1}
> 
> While the return value used to be simpler as status which contains message,
> code and progress.

The content of code is wrong. It should be a number.
Comment 4 Pavel Novotny 2013-12-09 14:12:07 EST
Verified in is26/RC:
RHEVM: rhevm-3.3.0-0.38.rc.el6ev.noarch
VDSM: vdsm-4.13.2-0.1.rc.el6ev.x86_64

Verified according to reproducer in comment 0.

Results:

Migration failed as expected.

vdsm.log:

Thread-2773::DEBUG::2013-12-09 18:58:04,658::BindingXMLRPC::991::vds::(wrapper) return vmGetMigrationStatus with {'status': {'message': 'Fatal error during migration', 'code': 12}, 'progress': 0}
^^^ the status['code'] is correctly a number

No `ClassCastException` in the engine.log:

2013-12-09 18:57:38,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(HostName = srh-10, HostId = 6b104737-7581-47b0-8297-34454d645823, vmId=2a0a9c2c-b56d-4c85-9ecd-616c188b1294), log id: 3be0db03
2013-12-09 18:57:38,044 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) Failed in MigrateStatusVDS method
2013-12-09 18:57:38,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) Error code migrateErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Fatal error during migration
2013-12-09 18:57:38,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value 
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12, mMessage=Fatal error during migration]]
2013-12-09 18:57:38,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) HostName = srh-10
2013-12-09 18:57:38,053 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Fatal error during migration
2013-12-09 18:57:38,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 3be0db03
2013-12-09 18:57:38,113 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-47) Correlation ID: 1e63449b, Job ID: 6b5b4046-c885-495c-ab62-4d2dd7c86323, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Fatal error during migration. Trying to migrate to another Host (VM: rhel65-ci, Source: srh-10, Destination: srh-09).
2013-12-09 18:57:38,299 WARN  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed. Reasons:VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT
2013-12-09 18:57:38,299 INFO  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) Lock freed to object EngineLock [exclusiveLocks= key: 2a0a9c2c-b56d-4c85-9ecd-616c188b1294 value: VM
, sharedLocks= ]
2013-12-09 18:57:38,348 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-47) Correlation ID: 1e63449b, Job ID: 6b5b4046-c885-495c-ab62-4d2dd7c86323, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Fatal error during migration (VM: rhel65-ci, Source: srh-10, Destination: srh-09).
Comment 5 errata-xmlrpc 2014-01-21 11:17:26 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html

Note You need to log in before you can comment on or make changes to this bug.