Bug 1377442

Summary: Clone from snapshot on Cinder hangs
Product: [oVirt] ovirt-engine Reporter: Andrew Richards <andrew.richards>
Component: Backend.CoreAssignee: Tal Nisan <tnisan>
Status: CLOSED CURRENTRELEASE QA Contact: Aharon Canan <acanan>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.0.6.3CC: bugs, tnisan
Target Milestone: ovirt-4.0.5Flags: tnisan: ovirt-4.0.z?
rule-engine: planning_ack?
tnisan: devel_ack+
rule-engine: testing_ack?
Target Release: 4.0.5   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-05 08:53:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log/ovirt-engine/engine.log context of error referenced none

Description Andrew Richards 2016-09-19 17:37:39 UTC 
Created attachment 1202569 [details]
/var/log/ovirt-engine/engine.log context of error referenced

Description of problem: Trying to clone a VM from s snapshot of an RBD Cinder volume results in an error and a task that will retry indefinitely. Live snapshot taken of running CentOS 7 guest is used to create a clone, but the clone task hangs. Cinder shows the new volume successfully created and available. No errors in the Cinder logs.


Version-Release number of selected component (if applicable): 
oVirt Engine v4.0.3-1.el7
Cinder (via RDO Liberty) v7.0.1-1.el7


How reproducible: Consistently reproducible 


Steps to Reproduce: all via oVirt GUI Administrative panel
1. Create VM with Cinder RBD boot volume
2. Create snapshot of this VM
3. Clone a new VM from that snapshot


Actual results: Task to create the clone never completes, but never fails either.


Expected results: Clone completes and uses the newly cloned RBD image as its boot device.


Additional info: /var/log/ovirt-engine/engine.log initial error message:
 
ERROR [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (default task-93) [50af631a] Command 'org.ovirt.engine.core.bll.AddVmFromSnapshotCommand' failed: null

ERROR [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (default task-93) [50af631a] Exception: java.lang.NullPointerException

...which is then followed repeatedly by:

ERROR [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (DefaultQuartzScheduler8) [50af631a] Ending command 'org.ovirt.engine.core.bll.AddVmFromSnapshotCommand' with failure.

ERROR [org.ovirt.engine.core.bll.storage.disk.cinder.CloneSingleCinderDiskCommand] (DefaultQuartzScheduler8) [50af631a] Ending command 'org.ovirt.engine.core.bll.storage.disk.cinder.CloneSingleCinderDiskCommand' with failure.

INFO  [org.ovirt.engine.core.utils.transaction.TransactionSupport] (DefaultQuartzScheduler8) [] transaction rolled back

ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [] Correlation ID: 284e62be, Job ID: 1899e73a-e5f8-4cda-b166-fbd5d444dc7c, Call Stack: null, Custom Event ID: -1, Message: Failed to complete VM testRBDclone1 creation.

ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler8) [] Failed invoking callback end method 'onFailed' for command '81f6cbd0-6298-4033-99a2-6a740e10c61f' with exception 'null', the callback is marked for end method retries

Full log snippet here: http://pastebin.com/ZLqRDSNW as well as attached TXT.

Cinder is from RDO repo, Liberty version. Cinder API is running v2.