Bug 1656092 - Importing OVA via Rest API failed
Summary: Importing OVA via Rest API failed
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Virt
Version: 4.3.0
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ovirt-4.3.0
: ---
Assignee: Steven Rosenberg
QA Contact: Pedut
Rolfe Dlugy-Hegwer
URL:
Whiteboard:
Depends On:
Blocks: 1562093
TreeView+ depends on / blocked
 
Reported: 2018-12-04 16:39 UTC by Pedut
Modified: 2019-02-13 07:44 UTC (History)
9 users (show)

Fixed In Version: ovirt-engine-4.3.0_rc2
Doc Type: Bug Fix
Doc Text:
Previously, when re-importing a virtual machine as an OVA file, duplicate Image IDs and Disk IDs caused errors while attempting to recreate the image. Also, after a failure, continuing to attempt to attach the image instead of failing immediately caused the error reported. Because the identifiers already existed, Red Hat Virtual Manager could not import the virtual machine OVA file even though the virtual machine name had been changed. This issue has been fixed in the current release. The Red Hat Virtual Manager can regenerate Identifiers. When copying the image, the Manager can use image mapping to correlate the previous Image ID to the new Image ID. Finally, the Manager can move the attach image handling so that it will not be called when creating a new image if the database fails. As a result, importing virtual machines using OVA files works.
Clone Of:
Environment:
Last Closed: 2019-02-13 07:44:58 UTC
oVirt Team: Virt
Embargoed:
rule-engine: ovirt-4.3+
rule-engine: blocker+


Attachments (Terms of Use)
logs (95.24 KB, application/x-xz)
2018-12-04 16:39 UTC, Pedut
no flags Details
logs 4.2.8-1 (92.09 KB, application/x-xz)
2018-12-06 13:30 UTC, Pedut
no flags Details
Import Request (15.47 KB, text/plain)
2018-12-13 09:23 UTC, Pedut
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 96268 0 master MERGED engine: Import OVA Duplicate Disk Image Failure 2021-01-17 08:06:51 UTC

Description Pedut 2018-12-04 16:39:26 UTC
Created attachment 1511380 [details]
logs

Description of problem:
Importing OVA via Rest API failed SQL ERROR: duplicate key value violates unique constraint "pk_images"

Version-Release number of selected component (if applicable):
Engine Version: 4.3.0-0.2.el7
OS Version: RHEL - 7.6 - 4.el7
LIBVIRT Version: libvirt-4.5.0-10.el7_6.3
VDSM Version: vdsm-4.30.3-71.el7


How reproducible:
100%

Steps to Reproduce:
1. Create VM
2. Export VM to host as OVA with one disk
3. Import the exported OVA(via Rest API)

Actual results:
Import failed with SQL ERROR: duplicate key value violates unique constraint "pk_images"

Expected results:
Import completed fine.

Additional info:
Caused by: org.springframework.dao.DuplicateKeyException: CallableStatementCallback; SQL [{call insertdiskvmelement(?, ?, ?, ?, ?, ?)}]; ERROR: duplicate key value violates unique constraint "pk_disk_vm_element"
  Detail: Key (vm_id, disk_id)=(c0986487-7161-4d90-a99a-586c7de95731, c2aa0075-e901-4fba-a603-dc60e27032b3) already exists.
  Where: SQL statement "INSERT INTO disk_vm_element (
        disk_id,
        vm_id,
        is_boot,
        pass_discard,
        disk_interface,
        is_using_scsi_reservation)
    VALUES (
        v_disk_id,
        v_vm_id,
        v_is_boot,
        v_pass_discard,
        v_disk_interface,
        v_is_using_scsi_reservation)"
PL/pgSQL function insertdiskvmelement(uuid,uuid,boolean,boolean,character varying,boolean) line 3 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pk_disk_vm_element"
  Detail: Key (vm_id, disk_id)=(c0986487-7161-4d90-a99a-586c7de95731, c2aa0075-e901-4fba-a603-dc60e27032b3) already exists.
  Where: SQL statement "INSERT INTO disk_vm_element (
        disk_id,
        vm_id,
        is_boot,
        pass_discard,
        disk_interface,
        is_using_scsi_reservation)
    VALUES (
        v_disk_id,
        v_vm_id,
        v_is_boot,
        v_pass_discard,
        v_disk_interface,
        v_is_using_scsi_reservation)"
PL/pgSQL function insertdiskvmelement(uuid,uuid,boolean,boolean,character varying,boolean) line 3 at SQL statement
	at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:239) [spring-jdbc.jar:4.3.9.RELEASE]
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) [spring-jdbc.jar:4.3.9.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1099) [spring-jdbc.jar:4.3.9.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1135) [spring-jdbc.jar:4.3.9.RELEASE]
	at org.springframework.jdbc.core.simple.AbstractJdbcCall.executeCallInternal(AbstractJdbcCall.java:405) [spring-jdbc.jar:4.3.9.RELEASE]
	at org.springframework.jdbc.core.simple.AbstractJdbcCall.doExecute(AbstractJdbcCall.java:365) [spring-jdbc.jar:4.3.9.RELEASE]
	at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198) [spring-jdbc.jar:4.3.9.RELEASE]
	at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:135) [dal.jar:]
	at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:130) [dal.jar:]
	at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeModification(SimpleJdbcCallsHandler.java:76) [dal.jar:]
	at org.ovirt.engine.core.dao.DefaultGenericDao.save(DefaultGenericDao.java:93) [dal.jar:]
	at org.ovirt.engine.core.bll.CommandCompensator.deletedOrUpdateEntity(CommandCompensator.java:124) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandCompensator.lambda$compensate$0(CommandCompensator.java:92) [bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:]
	at org.ovirt.engine.core.bll.CommandCompensator.compensate(CommandCompensator.java:63) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.compensate(CommandBase.java:475) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1173) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1299) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1948) [bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:]
	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1359) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:412) [bll.jar:]
	at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:638) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor493.invoke(Unknown Source) [:1.8.0_191]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:78)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.1.5.GA-redhat-00002.jar:7.1.5.GA-redhat-00002]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.1.5.GA-redhat-00002.jar:7.1.5.GA-redhat-00002]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.1.5.GA-redhat-00002.jar:7.1.5.GA-redhat-00002]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:264) [wildfly-ejb3-7.1.5.GA-redhat-00002.jar:7.1.5.GA-redhat-00002]
	... 324 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pk_disk_vm_element"
  Detail: Key (vm_id, disk_id)=(c0986487-7161-4d90-a99a-586c7de95731, c2aa0075-e901-4fba-a603-dc60e27032b3) already exists.
  Where: SQL statement "INSERT INTO disk_vm_element (
        disk_id,
        vm_id,
        is_boot,
        pass_discard,
        disk_interface,
        is_using_scsi_reservation)
    VALUES (
        v_disk_id,
        v_vm_id,
        v_is_boot,
        v_pass_discard,
        v_disk_interface,
        v_is_using_scsi_reservation)"
PL/pgSQL function insertdiskvmelement(uuid,uuid,boolean,boolean,character varying,boolean) line 3 at SQL statement
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
	at org.postgresql.jdbc.PgCallableStatement.executeWithFlags(PgCallableStatement.java:78)
	at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:144)
	at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:303)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:442)
	at org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1138) [spring-jdbc.jar:4.3.9.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1135) [spring-jdbc.jar:4.3.9.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1083) [spring-jdbc.jar:4.3.9.RELEASE]
	... 371 more

2018-11-26 01:16:18,058+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-29) [5d8832bf] EVENT_ID: IMPORTEXPORT_IMPORT_VM_FAILED(1,153), Failed to import Vm test_ova_export to Data Center golden_env_mixed, Cluster golden_env_mixed_1

Comment 1 Michal Skrivanek 2018-12-05 06:28:02 UTC
Regression from what version?

Comment 2 Michal Skrivanek 2018-12-05 08:17:22 UTC
we should do at least some validations in ImportVmFromExternalUrlCommand like we do in ImportVmCommand, at least for the VM name and disk images duplication

Comment 3 Red Hat Bugzilla Rules Engine 2018-12-05 11:17:55 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 4 Pedut 2018-12-06 12:01:40 UTC
Regression from 4.2.7-7.

Comment 5 Michal Skrivanek 2018-12-06 13:11:14 UTC
that would be surprising, there were no changes in validation AFAIR. Do you have a engine.log in 4.2.7 where this works?
The fix we have in plan for this bug will just add validation so you will get a clear and concise error message, but it still won't import the VM when you just do export a VM without removing it from the system before importing it back.

Comment 6 Pedut 2018-12-06 13:30:55 UTC
Created attachment 1512139 [details]
logs 4.2.8-1

Comment 7 Pedut 2018-12-06 13:32:48 UTC
Michal I see that in 4.2.8-1 it works too. 
I attached the logs from the last run of our tests in 4.2.8-1 environment.

Comment 8 Ryan Barry 2018-12-10 13:46:36 UTC
If this works in 4.2.8, I'll close. Please re-open if it comes up again.

Comment 9 Pedut 2018-12-10 14:12:26 UTC
But it doesn't work in 4.3 at all.

Comment 10 Ryan Barry 2018-12-10 14:14:20 UTC
Surprising. Re-opened

Comment 11 Pedut 2018-12-10 14:20:18 UTC
I want to emphasize that OVA it self works, but this scenario never worked in 4.3.

Comment 12 Michal Skrivanek 2018-12-10 14:53:04 UTC
can you clarify if you remove the exported VM in 4.2 and 4.3 before importing it? is the REST API request exactly the same?

Comment 13 Michal Skrivanek 2018-12-12 06:56:31 UTC
Is it not the same case as in bug 1658249?

Comment 14 Pedut 2018-12-12 12:30:23 UTC
The exported VM wasn't removed.
As written in bug 1658249 the root cause is that both VMs have the same UUID.
The difference between the bugs is that in bug 1658249 the operations were performed via the UI and in this bug the operations were performed via the Rest API.

Comment 15 Steven Rosenberg 2018-12-13 08:51:14 UTC
Some clarifications for this issue have been requested:

1. It is not clear how to generate the message body for the import command.   
2. Exporting a VM with one disk does not seem to create the ova file on the host, it creates a temp file whereas exporting a VM with no disks seemed to work. I was wondering if there was steps missing. For example the assumption is that the second step of the instructions means that we export via the UI and only the Import on the third step is via the REST API. Please clarify. The question is if there is if something is missing in the instructions or if this is another bug or something QE did not report.

Comment 16 Nisim Simsolo 2018-12-13 08:59:55 UTC
This issue is related to automation REST API. 
Both export and import are done using REST API. 
Please ask Pedut about the message body.

Comment 17 Pedut 2018-12-13 09:23:11 UTC
Created attachment 1513964 [details]
Import Request

Comment 18 Pedut 2018-12-13 11:20:48 UTC
In the attached file you can find the import request as we send it via the Rest API.
The whole steps are done via the Rest API. 
You can find all the sequence of how to reproduce it in the art log that I attached when I opened the bug.

Comment 19 Steven Rosenberg 2018-12-13 11:24:04 UTC
Please provide the request including the message body for the export REST API command as well. It is not clear to me how the message body of the externalvmimports is obtained.

Also did you check the host for the export file to see if it is a valid ova file? In my testing the export did not complete.

Comment 20 Pedut 2018-12-13 11:58:59 UTC
Please take a look at the art log - all the requests are there.
I didn't check the exported OVA since the export mechanism should work in a certain way. If you want you can extract the tar file and check it.

Comment 21 Steven Rosenberg 2018-12-17 17:39:47 UTC
Notes for testing. The actual error was a failure to insert the Disk Image due to the duplicate Disk Image ID:

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pk_images"
  Detail: Key (image_guid)=(dc4bcbfa-0cde-419c-bb1a-7b46035448ae) already exists.
  Where: SQL statement "INSERT INTO images (
        creation_date,
        image_guid,
        it_guid,
        size,
        ParentId,
        imageStatus,
        lastModified,
        vm_snapshot_id,
        volume_type,
        image_group_id,
        volume_format,
        active,
        volume_classification
        )
    VALUES (
        v_creation_date,
        v_image_guid,
        v_it_guid,
        v_size,
        v_ParentId,
        v_imageStatus,
        v_lastModified,
        v_vm_snapshot_id,
        v_volume_type,
        v_image_group_id,
        v_volume_format,
        v_active,
        v_volume_classification
        )"
PL/pgSQL function insertimage(timestamp with time zone,uuid,uuid,bigint,uuid,integer,timestamp with time zone,uuid,integer,integer,uuid,boolean,smallint) line 3 at SQL statement


The fix validates the DiskImageID against the database table and fails with an error when the Disk Image already exists.

A small change was made to avoid the failure to insert to the disk_vm_element when the insert to the images table fails.

The issue can be simulated via the Web Admin if the VM Name is changed before the ova importing. The export can also be performed within the Web Admin.


The REST API command for simulating this issue can simply be rendered as such:

Method:

POST http://<hostname>:8080/ovirt-engine/api/externalvmimports

Note: Enter the host name of the engine


Body Content Type:

application/xml


Message Body:


<external_vm_import>
	<name>test_ova_export</name>
	<provider>kvm</provider>
	<sparse>false</sparse>
	<url>ova:///root/export/VM.ova</url>
	<host id=""/>
	<cluster id=""/>
	<storage_domain id=""/>
</external_vm_import>


Ensure the host, cluster and storage domain ids are valid and can be obtained from the Web Admin or via the rest api as follows:


GET http://<hostname>:8080/ovirt-engine/api/clusters

GET http://<hostname>:8080/ovirt-engine/api/hosts

GET http://<hostname>:8080/ovirt-engine/api/storagedomains


Also, ensure the name field is different than the name of the ova file, otherwise the process will fail because the VM name was not unique.

Comment 22 Steven Rosenberg 2018-12-18 13:44:47 UTC
There is an issue with whether to validate and block the Importing of Images with duplicate Image IDs / Disk IDs as per Michal's comment 2 above:

https://bugzilla.redhat.com/show_bug.cgi?id=1656092#c2

or to regenerate the Disk and Image ID which is what Arik is asking for within the patch that addressed the Validation only:  

https://gerrit.ovirt.org/#/c/96268/


In my testing I saw that we do validate and prevent the generation of Templates from a VM with Disks as well as when exporting a VM with a Disk to an External Domain. We should be consistent across our platform, so we do need a decision on whether to validate and fail or to regenerate the IDs which it seems we do for snapshots.

Comment 23 Martin Tessun 2018-12-18 14:18:40 UTC
(In reply to Steven Rosenberg from comment #22)
> There is an issue with whether to validate and block the Importing of Images
> with duplicate Image IDs / Disk IDs as per Michal's comment 2 above:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1656092#c2
> 
> or to regenerate the Disk and Image ID which is what Arik is asking for
> within the patch that addressed the Validation only:  
> 
> https://gerrit.ovirt.org/#/c/96268/
> 
> 
> In my testing I saw that we do validate and prevent the generation of
> Templates from a VM with Disks as well as when exporting a VM with a Disk to
> an External Domain. We should be consistent across our platform, so we do
> need a decision on whether to validate and fail or to regenerate the IDs
> which it seems we do for snapshots.

As the IDs are only used internally (and for REST-API scripts are hopefully checked/retrieved before acted upon), I would vote for regenerating an ID in case of a conflict. Customer's typically don't have a way to workaround such issues, so we should "help" them in doing so.

Comment 25 Pedut 2019-01-02 11:15:01 UTC
Verified on 4.3.0-0.4.master.20181231193012.git1f27a84.el7.

Comment 26 Rolfe Dlugy-Hegwer 2019-01-18 02:14:32 UTC
Please closely review the Doc Text. This is as close as I can get it with the information available. However, I'm sure this text is incorrect. Please contact me if you want to work together on wordsmithing the final draft.

Comment 27 Steven Rosenberg 2019-01-20 09:31:38 UTC
As per my e-mail: Please advise what you believe is not correct and we can discuss your concerns accordingly. The changes are in the oVirt Engine, to be more precise.

Comment 28 Rolfe Dlugy-Hegwer 2019-01-20 15:32:49 UTC
Thanks, Steven. I appreciate the quick turn-around. Please note, I had commented that the proposed text was *incorrect* and needed further attention. Thanks for improving it.

Comment 29 Steven Rosenberg 2019-01-20 16:08:26 UTC
You are welcome Rolfe.

Comment 30 Sandro Bonazzola 2019-02-13 07:44:58 UTC
This bugzilla is included in oVirt 4.3.0 release, published on February 4th 2019.

Since the problem described in this bug report should be
resolved in oVirt 4.3.0 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.