Bug 1015887

Summary: Migration status verb fails on engine side due to change in response format
Product: Red Hat Enterprise Virtualization Manager Reporter: GenadiC <gcheresh>
Component: vdsmAssignee: Vinzenz Feenstra [evilissimo] <vfeenstr>
Status: CLOSED ERRATA QA Contact: Pavel Novotny <pnovotny>
Severity: high Docs Contact:
Priority: high    
Version: 3.3.0CC: acathrow, bazulay, iheim, lpeer, lyarwood, masayag, mavital, michal.skrivanek, myakove, Rhev-m-bugs, yeylon
Target Milestone: ---   
Target Release: 3.3.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: virt
Fixed In Version: is25 Doc Type: Bug Fix
Doc Text:
Cancelling a migration returned an error code in the wrong format. It has now been fixed, the error message now appears in the format of {'status': {'message': 'Migration canceled', 'code': 47}, 'progress': 1}
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-21 16:17:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 970645, 1038284    
Attachments:
Description Flags
engine log
none
origin vdsm log none

Description GenadiC 2013-10-06 14:12:07 UTC
Created attachment 808522 [details]
engine log

Description of problem:
Network migration fails on "Command MigrateStatusVDS execution failed. Exception: ClassCastException: java.util.HashMap cannot be cast to java.lang.Integer" when changing MTU to value different from default (9000 or 1480)

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create Network with MTU of 1480
2. Configure it as Display network and attach to Hosts with static IPs
3. Try to perform migration through that network

Actual results:
Migration fails after Timeout

Expected results:
Migration should work as expected over migration network

Additional info:
2013-10-06 16:54:12,519 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Rerun vm 0dcf947f-85aa-43a4-864b-d4abb71a1a5f. Called from vds silver-vdsb.qa.lab.tlv.redhat.com
2013-10-06 16:54:12,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-49) START, MigrateStatusVDSCommand(HostName = silver-vdsb.qa.lab.tlv.redhat.com, HostId = e937868c-88db-48c6-b714-295c52c2f798, vmId=0dcf947f-85aa-43a4-864b-d4abb71a1a5f), log id: 333c38cc
2013-10-06 16:54:12,530 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-49) Failed in MigrateStatusVDS method, for vds: silver-vdsb.qa.lab.tlv.redhat.com; host: 10.35.128.12
2013-10-06 16:54:12,531 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-49) Command MigrateStatusVDS execution failed. Exception: ClassCastException: java.util.HashMap cannot be cast to java.lang.Integer
2013-10-06 16:54:12,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-49) FINISH, MigrateStatusVDSCommand, log id: 333c38cc
2013-10-06 16:54:12,535 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-49) Correlation ID: 2dc985a1, Job ID: 392b15b2-2bab-43e8-a26d-557e6eb67d69, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Internal Engine Error. Trying to migrate to another Host (VM: VMTest1, Source: silver-vdsb.qa.lab.tlv.redhat.com, Destination: silver-vdsa.qa.lab.tlv.redhat.com).
2013-10-06 16:54:12,552 WARN  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-49) CanDoAction of action MigrateVm failed. Reasons:VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM
2013-10-06 16:54:12,552 INFO  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-49) Lock freed to object EngineLock [exclusiveLocks= key: 0dcf947f-85aa-43a4-864b-d4abb71a1a5f value: VM
, sharedLocks= ]
2013-10-06 16:54:12,557 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-49) Correlation ID: 2dc985a1, Job ID: 392b15b2-2bab-43e8-a26d-557e6eb67d69, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Internal Engine Error (VM: VMTest1, Source: silver-vdsb.qa.lab.tlv.redhat.com, Destination: silver-vdsa.qa.lab.tlv.redhat.com).

Comment 1 GenadiC 2013-10-06 14:13:35 UTC
Created attachment 808523 [details]
origin vdsm log

Comment 2 Moti Asayag 2013-10-10 14:03:22 UTC
vdsm version: vdsm-4.13.0-0.2.beta1.el6ev.x86_64

The error on engine.log:

"Command MigrateStatusVDS execution failed. Exception: ClassCastException: java.util.HashMap cannot be cast to java.lang.Integer" 

appears since the returned status from VSDM for migrateStatus verb was changed in http://gerrit.ovirt.org/#/c/11913/3/vdsm/vm.py:

Thread-9603::DEBUG::2013-10-10 14:36:12,042::BindingXMLRPC::981::vds::(wrapper) return vmGetMigrationStatus with {'status': {'message': 'Migration canceled', 'code': {'status': {'message': 
'Migration not in progress', 'code': 47}}}, 'progress': 1}

While the return value used to be simpler as status which contains message, code and progress.

Comment 3 Vinzenz Feenstra [evilissimo] 2013-11-14 07:53:28 UTC
(In reply to Moti Asayag from comment #2)
> vdsm version: vdsm-4.13.0-0.2.beta1.el6ev.x86_64
> 
> The error on engine.log:
> 
> "Command MigrateStatusVDS execution failed. Exception: ClassCastException:
> java.util.HashMap cannot be cast to java.lang.Integer" 
> 
> appears since the returned status from VSDM for migrateStatus verb was
> changed in http://gerrit.ovirt.org/#/c/11913/3/vdsm/vm.py:
This bug fix did not introduce this issue
> 
> Thread-9603::DEBUG::2013-10-10
> 14:36:12,042::BindingXMLRPC::981::vds::(wrapper) return vmGetMigrationStatus
> with {'status': {'message': 'Migration canceled', 'code': {'status':
> {'message': 
> 'Migration not in progress', 'code': 47}}}, 'progress': 1}
> 
> While the return value used to be simpler as status which contains message,
> code and progress.

The content of code is wrong. It should be a number.

Comment 4 Pavel Novotny 2013-12-09 19:12:07 UTC
Verified in is26/RC:
RHEVM: rhevm-3.3.0-0.38.rc.el6ev.noarch
VDSM: vdsm-4.13.2-0.1.rc.el6ev.x86_64

Verified according to reproducer in comment 0.

Results:

Migration failed as expected.

vdsm.log:

Thread-2773::DEBUG::2013-12-09 18:58:04,658::BindingXMLRPC::991::vds::(wrapper) return vmGetMigrationStatus with {'status': {'message': 'Fatal error during migration', 'code': 12}, 'progress': 0}
^^^ the status['code'] is correctly a number

No `ClassCastException` in the engine.log:

2013-12-09 18:57:38,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(HostName = srh-10, HostId = 6b104737-7581-47b0-8297-34454d645823, vmId=2a0a9c2c-b56d-4c85-9ecd-616c188b1294), log id: 3be0db03
2013-12-09 18:57:38,044 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) Failed in MigrateStatusVDS method
2013-12-09 18:57:38,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) Error code migrateErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Fatal error during migration
2013-12-09 18:57:38,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value 
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12, mMessage=Fatal error during migration]]
2013-12-09 18:57:38,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) HostName = srh-10
2013-12-09 18:57:38,053 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Fatal error during migration
2013-12-09 18:57:38,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 3be0db03
2013-12-09 18:57:38,113 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-47) Correlation ID: 1e63449b, Job ID: 6b5b4046-c885-495c-ab62-4d2dd7c86323, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Fatal error during migration. Trying to migrate to another Host (VM: rhel65-ci, Source: srh-10, Destination: srh-09).
2013-12-09 18:57:38,299 WARN  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed. Reasons:VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT
2013-12-09 18:57:38,299 INFO  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) Lock freed to object EngineLock [exclusiveLocks= key: 2a0a9c2c-b56d-4c85-9ecd-616c188b1294 value: VM
, sharedLocks= ]
2013-12-09 18:57:38,348 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-47) Correlation ID: 1e63449b, Job ID: 6b5b4046-c885-495c-ab62-4d2dd7c86323, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Fatal error during migration (VM: rhel65-ci, Source: srh-10, Destination: srh-09).

Comment 5 errata-xmlrpc 2014-01-21 16:17:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html