Bug 1825638

Summary: OVA export is failing writing disk headers
Product: [oVirt] ovirt-engine Reporter: Liran Rotenberg <lrotenbe>
Component: BLL.VirtAssignee: Liran Rotenberg <lrotenbe>
Status: CLOSED CURRENTRELEASE QA Contact: Nisim Simsolo <nsimsolo>
Severity: urgent Docs Contact:
Priority: high    
Version: 4.4.0CC: aoconnor, bpelled, bugs, eshames, gveitmic, lsvaty, mavital, michal.skrivanek, mnecas, mperina, nsimsolo, rbarry, tamir
Target Milestone: ovirt-4.4.1Keywords: Regression
Target Release: ---Flags: pm-rhel: ovirt-4.4+
aoconnor: blocker+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rhv-4.4.0-35 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-08 08:25:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1684266, 1733843, 1743543, 1748879, 1771333    
Attachments:
Description Flags
ova ansible log none

Description Liran Rotenberg 2020-04-19 14:26:26 UTC 
Created attachment 1680023 [details]
ova ansible log

Description of problem:
Exporting OVA failing due to error when writing disk headers.

How reproducible:
100%

Steps to Reproduce:
1. Create a VM or a template with disk.
2. Export as OVA.

Actual results:
Fail to create OVA.

Expected results:
OVA creation should succeed. 

Additional info:
In the engine the traceback shows:
2020-04-19 15:22:14,454+03 ERROR [org.ovirt.engine.core.bll.CreateOvaCommand] (EE-ManagedThreadFactory-engine-Thread-647) [4d0775e4] Command 'org.ovirt.engine.core.bll.CreateOvaCommand' failed: Task Check OVA creation process result failed to execute: 
2020-04-19 15:22:14,454+03 ERROR [org.ovirt.engine.core.bll.CreateOvaCommand] (EE-ManagedThreadFactory-engine-Thread-647) [4d0775e4] Exception: org.ovirt.engine.core.common.utils.ansible.AnsibleRunnerCallException: Task Check OVA creation process result failed to execute: 
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleRunnerHTTPClient.processEvents(AnsibleRunnerHTTPClient.java:239)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor.runCommand(AnsibleExecutor.java:170)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor.runCommand(AnsibleExecutor.java:59)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor.runCommand(AnsibleExecutor.java:46)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CreateOvaCommand.runAnsiblePackOvaPlaybook(CreateOvaCommand.java:211)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CreateOvaCommand.executeCommand(CreateOvaCommand.java:86)

...

	at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)

2020-04-19 15:22:14,456+03 ERROR [org.ovirt.engine.core.bll.exportimport.ExportVmTemplateToOvaCommand] (EE-ManagedThreadFactory-engine-Thread-647) [4d0775e4] Failed to create OVA file
2020-04-19 15:22:14,456+03 ERROR [org.ovirt.engine.core.bll.exportimport.ExportVmTemplateToOvaCommand] (EE-ManagedThreadFactory-engine-Thread-647) [4d0775e4] Command 'org.ovirt.engine.core.bll.exportimport.ExportVmTemplateToOvaCommand' failed: EngineException: ENGINE (Failed with error ENGINE and code 5001)
2020-04-19 15:22:14,460+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-647) [4d0775e4] EVENT_ID: IMPORTEXPORT_EXPORT_TEMPLATE_TO_OVA_FAILED(1,228), Failed to export Template ova as a Virtual Appliance to path /var/ova.ova on Host host_mixed_1

The error is within the ova export log - ovirt-export-ova-ansible:
020-04-19 16:05:33 IDT - TASK [ovirt-ova-pack : Run packing script] *************************************
2020-04-19 16:05:33 IDT - fatal: [10.46.9.68]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Shared connection to 10.46.9.68 closed.\r\n", "stderr_lines": [
"Shared connection to 10.46.9.68 closed."], "stdout": "writing ovf: <?xml version=1.0 encoding=UTF-8?><ovf:Envelope xmlns:ovf=http://schemas.dmtf.org/ovf/envelope/1 xmlns:rasd=http://schemas
.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData xmlns:vssd=http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData xmlns:xsi=http://www.w3.org/20
01/XMLSchema-instance xmlns=http://schemas.dmtf.org/ovf/envelope/1 xmlns:ovirt=http://www.ovirt.org/ovf><References><File ovf:href=d87d7c5b-e64b-4751-a85b-6757a097d4f8 ovf:id=d87d7c5b-e64b-4
751-a85b-6757a097d4f8 ovf:size=1074135040></File></References><NetworkSection><Info>List of networks</Info></NetworkSection><DiskSection><Info>List of Virtual Disks</Info><Disk ovf:diskId=28
7c5f27-6c73-429f-b6a7-8e8fe3b91aa1 ovf:capacity=1 ovf:capacityAllocationUnits=byte\r\nTraceback (most recent call last):\r\n  File \"/root/.ansible/tmp/ansible-tmp-1587301528.376324-20903482
1475250/pack_ova.py\", line 99, in <module>\r\n    write_disk_headers(ova_file, disks_info.split('+'))\r\n  File \"/root/.ansible/tmp/ansible-tmp-1587301528.376324-209034821475250/pack_ova.p
y\", line 71, in write_disk_headers\r\n    idx = disk_info.index('::')\r\nValueError: substring not found\r\n", "stdout_lines": ["writing ovf: <?xml version=1.0 encoding=UTF-8?><ovf:Envelope
 xmlns:ovf=http://schemas.dmtf.org/ovf/envelope/1 xmlns:rasd=http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData xmlns:vssd=http://schemas.dmtf.org/wbem/wscim
/1/cim-schema/2/CIM_VirtualSystemSettingData xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xmlns=http://schemas.dmtf.org/ovf/envelope/1 xmlns:ovirt=http://www.ovirt.org/ovf><References
><File ovf:href=d87d7c5b-e64b-4751-a85b-6757a097d4f8 ovf:id=d87d7c5b-e64b-4751-a85b-6757a097d4f8 ovf:size=1074135040></File></References><NetworkSection><Info>List of networks</Info></Networ
kSection><DiskSection><Info>List of Virtual Disks</Info><Disk ovf:diskId=287c5f27-6c73-429f-b6a7-8e8fe3b91aa1 ovf:capacity=1 ovf:capacityAllocationUnits=byte", "Traceback (most recent call l
ast):", "  File \"/root/.ansible/tmp/ansible-tmp-1587301528.376324-209034821475250/pack_ova.py\", line 99, in <module>", "    write_disk_headers(ova_file, disks_info.split('+'))", "  File \"
/root/.ansible/tmp/ansible-tmp-1587301528.376324-209034821475250/pack_ova.py\", line 71, in write_disk_headers", "    idx = disk_info.index('::')", "ValueError: substring not found"]}
Comment 1 Ryan Barry 2020-04-20 00:42:10 UTC 
Network ok? The connection shouldn't be closed for this
Comment 2 Liran Rotenberg 2020-04-20 06:57:56 UTC 
(In reply to Ryan Barry from comment #1)
> Network ok? The connection shouldn't be closed for this

Network OK, the error is within stderr_lines:
"Traceback (most recent call last):",
"File \"/root/.ansible/tmp/ansible-tmp-1587301528.376324-209034821475250/pack_ova.py\",
line 99, in <module>", "write_disk_headers(ova_file, disks_info.split('+'))", 
"File \" /root/.ansible/tmp/ansible-tmp-1587301528.376324-209034821475250/pack_ova.py\",
line 71, in write_disk_headers","idx = disk_info.index('::')",
"ValueError: substring not found"

The pack_ova.py returns non-zero and the connection is closed.
I checked it on two different environments.
Comment 3 Liran Rotenberg 2020-04-20 15:25:12 UTC 
*** Bug 1825842 has been marked as a duplicate of this bug. ***
Comment 4 Nisim Simsolo 2020-04-20 17:06:01 UTC 
Also seen this issue manually using build ovirt-engine-4.4.0-0.33.master.el8ev
Comment 5 Liran Rotenberg 2020-04-21 08:13:12 UTC 
The main issue we see here should be fixed in(tested locally and it did work): https://github.com/ansible/ansible-runner-service/pull/52
The problem is, when ansible running the python script, it adds files and directories in $HOME at the destination host as arguments for the python script.
Therefore, we are having wrong arguments, failing the script.

Another problem is regarding template export - an NPE caused by: BZ 1733843.
As well, we had problems importing both VM and template, caused by changes in escaping in: https://gerrit.ovirt.org/#/c/107683/

I will post the fix for the escaping and the template export, but this bug still needs the PR for ansible-runner and a ansible-runner version bump.
Comment 6 Lukas Svaty 2020-04-21 10:35:50 UTC 
Does this happen on export of VMs or only templates?
Comment 7 Nisim Simsolo 2020-04-21 11:56:52 UTC 
(In reply to Lukas Svaty from comment #6)
> Does this happen on export of VMs or only templates?

Issue is true for both VM export and template export
Comment 8 Michal Skrivanek 2020-04-27 04:25:32 UTC 
*** Bug 1828025 has been marked as a duplicate of this bug. ***
Comment 9 Michal Skrivanek 2020-04-27 04:29:12 UTC 
*** Bug 1828040 has been marked as a duplicate of this bug. ***
Comment 10 Liran Rotenberg 2020-04-29 10:57:49 UTC 
Patch is merged and ansible-runner-service-dev-1.0.2 released.
Comment 11 Liran Rotenberg 2020-05-11 08:24:45 UTC 
*** Bug 1833851 has been marked as a duplicate of this bug. ***
Comment 12 Nisim Simsolo 2020-06-02 12:14:57 UTC 
Verification builds:
ovirt-engine-4.4.1.1-0.5.el8ev
vdsm-4.40.17-1.el8ev.x86_64
libvirt-daemon-6.0.0-22.module+el8.2.1+6815+1c792dc8.x86_64
qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64


Verification scenario:
1. Export VM with 4 disks as OVA.
   Verify VM exported successfully.
2. Import OVA. 
   Run VM and verify VM is running.
3. Make template from VM with 4 disks. export template as OVA.
   Verify template exported as OVA successfully.
4. Import template as clone from OVA.
   Verify Template imported successfully.
5. Create VM from imported template.
   Run VM and verify VM is running.
Comment 13 Sandro Bonazzola 2020-07-08 08:25:20 UTC 
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.