Bug 996005

Summary: ImportVmTemplateParameters fails (Unexpected token)
Product: Red Hat Enterprise Virtualization Manager Reporter: Meital Bourvine <mbourvin>
Component: ovirt-engineAssignee: Roy Golan <rgolan>
Status: CLOSED CURRENTRELEASE QA Contact: Meital Bourvine <mbourvin>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.3.0CC: acathrow, bzrh.bobd, cdhouch, eedri, herrold, iheim, lpeer, mbourvin, michal.skrivanek, nlevinki, obasan, pstehlik, Rhev-m-bugs, yeylon
Target Milestone: ---Keywords: TestBlocker
Target Release: 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: virt
Fixed In Version: is10 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Meital Bourvine 2013-08-12 07:41:02 UTC
Description of problem:
Import export fails when trying to Import exported VM after VM removal (iscsi).

http://jenkins.eng.lab.tlv.redhat.com/job/rhevm_3.3_automation_storage_export_import_server_factory/ENGINE=rest,HOST_HOSTGROUP=P-RHEVM-3.3-RHEL64-HOSTS,STORAGE_TYPE=iscsi,label=RHEVM-3.3-ENGINE-RHEL64/227/consoleFull

Version-Release number of selected component (if applicable):
is9.1

Additional info:

Additional logs: http://jenkins.eng.lab.tlv.redhat.com/job/rhevm_3.3_automation_storage_export_import_server_factory/ENGINE=rest,HOST_HOSTGROUP=P-RHEVM-3.3-RHEL64-HOSTS,STORAGE_TYPE=iscsi,label=RHEVM-3.3-ENGINE-RHEL64/227/artifact/logs/jenkins-rhevm_3.3_automation_storage_export_import_server_factory-ENGINE=rest,HOST_HOSTGROUP=P-RHEVM-3.3-RHEL64-HOSTS,STORAGE_TYPE=iscsi,label=RHEVM-3.3-ENGINE-RHEL64-227__08112013_18-30-47.tar.bz2

2013-08-11 16:22:21,368 ERROR [org.ovirt.engine.core.bll.CopyImageGroupCommand] (ajp-/127.0.0.1:8702-5) Error during CreateTask for command: org.ovirt.engine.core.bll.CopyImageGroupCommand. Exception org.apache.commons.lang.Serialization
Exception: org.codehaus.jackson.map.JsonMappingException: Unexpected token (START_OBJECT), expected START_ARRAY: need JSON Array to contain As.WRAPPER_ARRAY type information for class java.util.Map
 at [Source: java.io.StringReader@3543fc31; line: 157, column: 29] (through reference chain: org.ovirt.engine.core.common.action.ImportVmTemplateParameters["vmTemplate"]->org.ovirt.engine.core.common.businessentities.VmTemplate["diskMap"
]): org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: Unexpected token (START_OBJECT), expected START_ARRAY: need JSON Array to contain As.WRAPPER_ARRAY type information for class java.util.Ma
p
 at [Source: java.io.StringReader@3543fc31; line: 157, column: 29] (through reference chain: org.ovirt.engine.core.common.action.ImportVmTemplateParameters["vmTemplate"]->org.ovirt.engine.core.common.businessentities.VmTemplate["diskMap"])
        at org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.readJsonString(JsonObjectDeserializer.java:90) [utils.jar:]
        at org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.deserialize(JsonObjectDeserializer.java:57) [utils.jar:]
        at org.ovirt.engine.core.dao.AsyncTaskDAODbFacadeImpl$AsyncTaskRowMapper.deserializeParameters(AsyncTaskDAODbFacadeImpl.java:88) [dal.jar:]
        at org.ovirt.engine.core.dao.AsyncTaskDAODbFacadeImpl$AsyncTaskRowMapper.mapRow(AsyncTaskDAODbFacadeImpl.java:71) [dal.jar:]
        at org.ovirt.engine.core.dao.AsyncTaskDAODbFacadeImpl$AsyncTaskRowMapper.mapRow(AsyncTaskDAODbFacadeImpl.java:60) [dal.jar:]
        at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) [spring-jdbc.jar:3.1.3.RELEASE]
        at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:1) [spring-jdbc.jar:3.1.3.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:649) [spring-jdbc.jar:3.1.3.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:587) [spring-jdbc.jar:3.1.3.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.3.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.3.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.3.RELEASE]
        at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:]
        at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.3.RELEASE]
        at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:147) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:112) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:95) [dal.jar:]
        at org.ovirt.engine.core.dao.AsyncTaskDAODbFacadeImpl.get(AsyncTaskDAODbFacadeImpl.java:134) [dal.jar:]
        at org.ovirt.engine.core.bll.CommandBase.getAsyncTask(CommandBase.java:1617) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.concreteCreateTask(CommandBase.java:1600) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.createTaskImpl(CommandBase.java:1571) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.createTask(CommandBase.java:1541) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.createTask(CommandBase.java:1456) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.createTask(CommandBase.java:1506) [bll.jar:]
        at org.ovirt.engine.core.bll.CopyImageGroupCommand.executeCommand(CopyImageGroupCommand.java:100) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1130) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1215) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1879) [bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:151) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:118) [utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1235) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:365) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:408) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:387) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:628) [bll.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]

...

Comment 1 Roy Golan 2013-08-13 06:39:11 UTC
how reproducible?
steps to reproduce?

Comment 2 Ohad Basan 2013-08-13 13:13:10 UTC
I still see it on ci

Comment 3 Ohad Basan 2013-08-13 13:13:44 UTC
*** Bug 996571 has been marked as a duplicate of this bug. ***

Comment 4 Meital Bourvine 2013-08-13 13:27:06 UTC
I don't know how reproducible it is, since I only saw it on CI job - maybe eedri can give more info about it.

Steps to reproduce:
Create iSCSI data center (name=iscsiStorageTest)
Create iSCSI export storage domain (name=ImpExp_iscsi)
Attach export storage domain to data center
Activate export storage domain
Create shared ISO domain NFS (name=shared-iso-domain)
Attach shared ISO domain to data center iscsiStorageTest
Activate shared ISO storage domain
Create VM with rhel6 (name=iscsiTestVm)
Stop running VM (iscsiTestVm) to create template
Create template (name=iscsiTestTempl)
Create VM from template (name=iscsiTemplVm)
Wait till vm is down
Try to export VM (iscsiTemplVm) only to data domain (iscsiStorageTest_data_domain0)
Try to export VM (iscsiTemplVm) only to ISO domain (shared-iso-domain)
Export VM (iscsiTemplVm) to domain (ImpExp_iscsi)
checkVmVolume after VM (iscsiTemplVm) export to domain (ImpExp_iscsi)
Remove VM (iscsiTemplVm) from export domain (ImpExp_iscsi)
checkVmVolume after VM (iscsiTemplVm) removal from export domain (ImpExp_iscsi)
Export VM (iscsiTemplVm) after VMs removal from export domain to domain(ImpExp_iscsi)
checkVmVolume after the second VM export (iscsiTemplVm) in domain (ImpExp_iscsi)
Export template (iscsiTestTempl) to domain (ImpExp_iscsi)
checkTemplateVolume after template (iscsiTestTempl) export to (ImpExp_iscsi)
Export VM (iscsiTemplVm) after template export to (ImpExp_iscsi)
Wait for VM down status (iscsiTemplVm)
Remove VM (iscsiTemplVm) after template export
Remove template (iscsiTestTempl)
Import exported template (iscsiTestTempl) (import_storagedomain='iscsiStorageTest_data_domain0',export_storagedomain='ImpExp_iscsi')
Import exported VM after VM removal (iscsiTemplVm) (import_storagedomain='iscsiStorageTest_data_domain0',export_storagedomain='ImpExp_iscsi')
<here it failed>

Comment 5 Meital Bourvine 2013-08-21 07:33:33 UTC
Working on CI.

Comment 7 Arik 2013-08-28 14:47:06 UTC
*** Bug 996309 has been marked as a duplicate of this bug. ***

Comment 8 Bob Doolittle 2013-11-13 22:34:31 UTC
Is there plan to backport this bug to ovirt-stable? It is really nasty because a single import operation can corrupt the engine and cause it to fail completely.

Comment 9 Roy Golan 2013-11-17 09:21:42 UTC
the import flow is creating a VDSM task to copy in disks from the a template.  a JSON structure (the task details) is being saved to the DB. when the task is done
the engine is reading the JSON from DB back and trying to de-serialize it into a java object 

bad serialized form:
{ 
 diskMap: [ "java.util.Map", { ... the map} ]
}

while it should be 
{ 
 diskMap: [ "java.util.HashMap", { ... the map} ]
}

when this is being de-serialized back the Json mapper know which concrete Map type to create. (in detail java.util.Map is an Abstract class)

more of this in on the patch http://gerrit.ovirt.org/18001

* the engine isn't corrupted by this. this only mean the task could not be terminate and be cleared properly i.e affecting the template in question.

* workaround - UNTESTED, didn't try this but changing the JSON in db directly might make the de-serialization work:
 psql engine postgres -tq -c "select task_parameters from async_task where task_id = '{taskId}';" | sed 's/java\.util\.Map/java\.util\.HashMap/' | psql engine postgres -c "update async_task set task_parameters = '$(xargs < /dev/stdio)' where task_id = '{task_id}'

Comment 10 Itamar Heim 2014-01-21 22:31:21 UTC
Closing - RHEV 3.3 Released

Comment 11 Itamar Heim 2014-01-21 22:31:21 UTC
Closing - RHEV 3.3 Released

Comment 12 Itamar Heim 2014-01-21 22:34:06 UTC
Closing - RHEV 3.3 Released