Bug 1412646 - Restore/Preview of RAM snapshot failed
Summary: Restore/Preview of RAM snapshot failed
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: General
Version: 4.19.1
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ovirt-4.1.0-beta
: ---
Assignee: Milan Zamazal
QA Contact: Raz Tamir
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-12 14:05 UTC by Raz Tamir
Modified: 2017-02-15 14:53 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-15 14:53:02 UTC
oVirt Team: Virt
rule-engine: ovirt-4.1+
rule-engine: blocker+
rule-engine: planning_ack+
ahadas: devel_ack+
rule-engine: testing_ack+


Attachments (Terms of Use)
engine and vdsm logs (321.41 KB, application/x-gzip)
2017-01-12 14:05 UTC, Raz Tamir
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 69899 0 None None None 2017-01-16 12:30:46 UTC
oVirt gerrit 70323 0 None None None 2017-01-16 08:58:30 UTC

Description Raz Tamir 2017-01-12 14:05:54 UTC
Created attachment 1239940 [details]
engine and vdsm logs

Description of problem:
When trying to restore a VM to snapshot with memory, the operation succeeds but the memory state is not restored.

In the engine.log:
2017-01-12 11:43:07,973+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-5) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm_TestCase5134_REST_ISCSI_1211373723 is down with error. Exit message: Wake up from hibernation failed:'MutableDomainDescriptor' object has no attribute '_devices'.
2017-01-12 11:43:07,974+02 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-5) [] add VM 'c1c3c2af-456b-485a-b0b0-0a7037426c51'(vm_TestCase5134_REST_ISCSI_1211373723) to rerun treatment
2017-01-12 11:43:07,981+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-5) [] Rerun VM 'c1c3c2af-456b-485a-b0b0-0a7037426c51'. Called from VDS 'host_mixed_3'
2017-01-12 11:43:07,995+02 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] Correlation ID: vms_syncAction_d01938e6-1199-45e0, Job ID: 504bda6d-67fe-41ee-af6d-ffaf4f669f08, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM vm_TestCase5134_REST_ISCSI_1211373723 on Host host_mixed_3.


In vdsm.log:
2017-01-12 11:43:07,078 INFO  (vm/c1c3c2af) [dispatcher] Run and protect: prepareImage, Return response: {'info': {'domainID': '13790281-69c2-4a63-9ec4-0781d4cb33d3', 'volType': 'path', 'leaseOffset': 119537664, 'path': '/rhev/data-center/mnt/blockSD/13790281-69c2-4a63-9ec4-0781d4cb33d3/images/1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d/eeaea5de-35d4-47ff-b213-0c85cf91930c', 'volumeID': 'eeaea5de-35d4-47ff-b213-0c85cf91930c', 'leasePath': '/dev/13790281-69c2-4a63-9ec4-0781d4cb33d3/leases', 'imageID': '1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d'}, 'path': '/rhev/data-center/ad32aa2c-fda4-4eab-83b2-a78568d48bd1/13790281-69c2-4a63-9ec4-0781d4cb33d3/images/1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d/eeaea5de-35d4-47ff-b213-0c85cf91930c', 'imgVolumesInfo': [{'domainID': '13790281-69c2-4a63-9ec4-0781d4cb33d3', 'volType': 'path', 'leaseOffset': 109051904, 'path': '/rhev/data-center/mnt/blockSD/13790281-69c2-4a63-9ec4-0781d4cb33d3/images/1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d/08e1b37f-eff4-4bac-a567-53e90bee5c75', 'volumeID': '08e1b37f-eff4-4bac-a567-53e90bee5c75', 'leasePath': '/dev/13790281-69c2-4a63-9ec4-0781d4cb33d3/leases', 'imageID': '1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d'}, {'domainID': '13790281-69c2-4a63-9ec4-0781d4cb33d3', 'volType': 'path', 'leaseOffset': 119537664, 'path': '/rhev/data-center/mnt/blockSD/13790281-69c2-4a63-9ec4-0781d4cb33d3/images/1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d/eeaea5de-35d4-47ff-b213-0c85cf91930c', 'volumeID': 'eeaea5de-35d4-47ff-b213-0c85cf91930c', 'leasePath': '/dev/13790281-69c2-4a63-9ec4-0781d4cb33d3/leases', 'imageID': '1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d'}, {'domainID': '13790281-69c2-4a63-9ec4-0781d4cb33d3', 'volType': 'path', 'leaseOffset': 114294784, 'path': '/rhev/data-center/mnt/blockSD/13790281-69c2-4a63-9ec4-0781d4cb33d3/images/1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d/d96b059a-f631-4ed6-a334-1271fc546b65', 'volumeID': 'd96b059a-f631-4ed6-a334-1271fc546b65', 'leasePath': '/dev/13790281-69c2-4a63-9ec4-0781d4cb33d3/leases', 'imageID': '1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d'}, {'domainID': '13790281-69c2-4a63-9ec4-0781d4cb33d3', 'volType': 'path', 'leaseOffset': 117440512, 'path': '/rhev/data-center/mnt/blockSD/13790281-69c2-4a63-9ec4-0781d4cb33d3/images/1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d/a042979c-83da-4442-849b-cc7de904237d', 'volumeID': 'a042979c-83da-4442-849b-cc7de904237d', 'leasePath': '/dev/13790281-69c2-4a63-9ec4-0781d4cb33d3/leases', 'imageID': '1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d'}]} (logUtils:52)
2017-01-12 11:43:07,078 INFO  (vm/c1c3c2af) [vds] prepared volume path: /rhev/data-center/ad32aa2c-fda4-4eab-83b2-a78568d48bd1/13790281-69c2-4a63-9ec4-0781d4cb33d3/images/1720e3e1-b1a9-44ad-a18b-cf6c6f292e0d/eeaea5de-35d4-47ff-b213-0c85cf91930c (clientIF:374)
2017-01-12 11:43:07,104 ERROR (vm/c1c3c2af) [virt.vm] (vmId='c1c3c2af-456b-485a-b0b0-0a7037426c51') The vm start process failed (vm:616)
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/vm.py", line 552, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/virt/vm.py", line 1953, in _run
    srcDomXML = self._correctDiskVolumes(srcDomXML)
  File "/usr/share/vdsm/virt/vm.py", line 2028, in _correctDiskVolumes
    for element in domain.get_device_elements('disk'):
  File "/usr/share/vdsm/virt/domain_descriptor.py", line 41, in get_device_elements
    return vmxml.find_all(self._devices, tagName)
AttributeError: 'MutableDomainDescriptor' object has no attribute '_devices'
2017-01-12 11:43:07,105 INFO  (vm/c1c3c2af) [virt.vm] (vmId='c1c3c2af-456b-485a-b0b0-0a7037426c51') Changed state to Down: 'MutableDomainDescriptor' object has no attribute '_devices' (code=1) (vm:1198)



Version-Release number of selected component (if applicable):
ovirt-engine-4.1.0-0.4.master.20170111000229.git9ce0636.el7.centos.noarch
vdsm-4.19.1-24.git7747cad.el7.centos.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Create a VM with 1 disk and OS installed
2. Run the VM and start a cat process --> $ cat &
3. Create a snapshot (VM running) with memory state
4. Power off the VM
5. Preview the snapshot
6. Start the VM and check for the process --> $ pgrep cat


Actual results:
The process doesn't exists

Expected results:


Additional info:
This is a regression from 4.0.6.3-0.1.el7ev

Comment 1 Yaniv Kaul 2017-01-12 15:13:55 UTC
virt issue?

Comment 2 Tal Nisan 2017-01-12 16:00:25 UTC
Daniel, any chance it's related to you latest change in the snapshots mechanism?

Whether it's virt or storage it needs to be targeted to 4.1 beta so I'm targeting.

Comment 3 Daniel Erez 2017-01-15 13:25:14 UTC
@Raz - is it reproducible without an installed OS? or without step 2 ("start a cat process --> $ cat &"). I've couldn't reproduce the issue on a local env.

Comment 4 Raz Tamir 2017-01-15 15:54:17 UTC
Daniel,
The snapshot operation works fine, and therefore, there is nothing interesting to check this scenario without an OS because the issue relays on comparing the processes that are running when taking the ram snapshot and after restoring/previewing the snapshot

Comment 5 Daniel Erez 2017-01-15 18:09:04 UTC
> Daniel,
> The snapshot operation works fine, and therefore, there is nothing
> interesting to check this scenario without an OS because the issue relays on
> comparing the processes that are running when taking the ram snapshot and
> after restoring/previewing the snapshot

@Arik - what do you think? familiar with this issue?

Comment 6 Arik 2017-01-16 08:21:37 UTC
(In reply to Daniel Erez from comment #5) 
> @Arik - what do you think? familiar with this issue?

Seems like a regression on VDSM related to refactoring that was done in VIRT related code.

Comment 7 Milan Zamazal 2017-01-16 08:54:54 UTC
This has already been fixed in master (commit f83f3a0) and ovirt-4.1 (commit e19ff7a).

Comment 8 Tal Nisan 2017-01-16 12:31:32 UTC
Milan, please verify this fix, I'd like to make sure it's verified before the build

Comment 9 Milan Zamazal 2017-01-16 14:16:55 UTC
I tried the initial Steps to Reproduce with Engine master and with current Vdsm ovirt-4.1 branch. It works for me, so there should be no problem anymore on the Vdsm side.

Comment 10 Raz Tamir 2017-01-23 15:35:31 UTC
Executed all our tier 1 and tier 2 on ovirt-engine-4.1.0.3-0.0.master.20170122091652.gitc6fc2c2.el7.centos and all passed.
Moving to verified


Note You need to log in before you can comment on or make changes to this bug.