Bug 1332453 - Snapshot deletion failed.
Summary: Snapshot deletion failed.
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: ovirt-guest-tools
Classification: oVirt
Component: General
Version: ---
Hardware: x86_64
OS: Linux
unspecified
high vote
Target Milestone: ---
: ---
Assignee: Lev Veyde
QA Contact: Pavel Stehlik
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-03 09:01 UTC by boruvka.michal
Modified: 2017-05-11 09:23 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-01 13:29:52 UTC
oVirt Team: Virt
amureini: ovirt-3.6.z?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
logs from engine, host and spm (5.80 MB, application/x-gzip)
2016-05-03 09:01 UTC, boruvka.michal
no flags Details
libvirt log (6.00 KB, text/plain)
2016-05-17 09:51 UTC, boruvka.michal
no flags Details
dumpxml (6.03 KB, text/plain)
2016-05-18 08:46 UTC, boruvka.michal
no flags Details
running services on Windows guest (253.87 KB, image/jpeg)
2016-06-16 07:35 UTC, boruvka.michal
no flags Details
Snapshot deletion fail logs 28.8.2016 (3.51 MB, application/x-gzip)
2016-08-29 06:53 UTC, boruvka.michal
no flags Details

Description boruvka.michal 2016-05-03 09:01:16 UTC
Created attachment 1153289 [details]
logs from engine, host and spm

Description of problem:
Snapshot deletion failed.

Version-Release number of selected component (if applicable):
3.6.5.3-1.el7.centos
vdsm 4.17.26-0.el7.centos

How reproducible:
sometime

Steps to Reproduce:
After fail snapshot disk is in illegal state.
If I stop/start VM and run SN deletion again, SN is deleted, but filesystem is destroyed.

Actual results:
2016-05-02 23:06:47,008 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [1a26d1f0] Correlation ID: 1a26d1f0, Job ID: 8e0c8a78-e7e4-4be9-a9b6-d35a250fe70c, Call Stack: null, Custom Event ID: -1, Message: Failed to delete snapshot 'pred profily3' for VM 'terminal1'.


Expected results:


Additional info:
logs in attachment

Comment 1 Allon Mureinik 2016-05-05 07:55:03 UTC
What do you mean by "filesystem is destroyed"? What exactly are you seeing?

Comment 2 boruvka.michal 2016-05-05 19:52:43 UTC
After start VM check disk is running and some files are often corrupt. (VM was stopped regularly)

Comment 3 Ala Hino 2016-05-15 14:19:04 UTC
From the logs, I see that qemu-guest-agent is not available:

jsonrpc.Executor/6::WARNING::2016-05-03 00:09:01,848::vm::3082::virt.vm::(freeze) vmId=`ae83a7b7-82af-4707-b655-b88a1d94ee5b`::Unable to freeze guest filesystems: Guest agent is not responding: Guest agent not available for now

After starting the VM please run fsck.

In general, if qemu-guest-agent is not available, the snapshot isn't consistent. Please make sure the agent is available before taking snapshots.

Comment 4 boruvka.michal 2016-05-16 08:07:21 UTC
I have Guest agent installed, but in Windows it is not working. If I run: "vssadmin list providers", I see: "QEMU Guest Agent VSS Provider" but ovirt tell me: "Could not detect Guest Agent on the VM". 

In linux it is working fine.

But I'm not sure if its my problem. My problem is in this situation:

1. create snapshot (consistent/no consistent)
2. delete snapshot - delete failed and snapshot disk is in ILLEGAL state
3. stop/start VM (its OK)
4. delete snapshot again - snapshot is deleted
5. stop/start VM - filesystem is corrupted

It is sometime happening.

Comment 5 Ala Hino 2016-05-16 19:55:57 UTC
Can you please make sure you followed the steps described in:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/chap-QEMU_Guest_Agent.html

See section 11.1.2 - Configuring the QEMU guest agent on a Windows guest
And section 11.2 - USING THE QEMU GUEST AGENT WITH LIBVIRT

Comment 6 boruvka.michal 2016-05-17 09:09:25 UTC
Yes. I followed this steps. In windows I have installed virtio-serial driver and qemu-ga-x64.msi. Services are running.

Comment 7 Ala Hino 2016-05-17 09:20:54 UTC
Can you please add libvirt logs?

Comment 8 boruvka.michal 2016-05-17 09:51:06 UTC
Created attachment 1158234 [details]
libvirt log

windows VM with qemu guest agent installed

Comment 9 Ala Hino 2016-05-18 08:42:23 UTC
can you please run following command on the host where the VM is running and send the (XML) output?

virsh -r dumpxml sogo2-w12

(where sogo2-w12 is the name of the Win guest)

Comment 10 boruvka.michal 2016-05-18 08:46:22 UTC
Created attachment 1158649 [details]
dumpxml

Comment 11 Vinzenz Feenstra [evilissimo] 2016-05-18 09:06:23 UTC
Hi,

Do you have installed the OVirtGuestTools on Windows and the OVirtGuestService running? 

If not please try that.

From what I remember, we're trying to check for the presence of _our_ guest agent to deduce that the QEMU Guest agent is there (it's a dependency of our service)

Comment 12 boruvka.michal 2016-05-18 09:16:19 UTC
Yes. I have it installed and service OvirtGuestService is running.

Comment 13 Allon Mureinik 2016-05-18 13:38:16 UTC
Moving to the guest-tools component to continue the investigation there

Comment 14 Vinzenz Feenstra [evilissimo] 2016-06-16 07:10:02 UTC
Mr Boruvka, 

could you please check that the following service is installed and mainly that it is running:

"QEMU Guest Agent"

Thank you.

Comment 15 boruvka.michal 2016-06-16 07:29:41 UTC
Yes, service QEMU Guest Agent is running.

Comment 16 boruvka.michal 2016-06-16 07:35:47 UTC
Created attachment 1168582 [details]
running services on Windows guest

Comment 17 Vinzenz Feenstra [evilissimo] 2016-06-23 11:46:49 UTC
Could you please try to reproduce this once more now and provide the logs from vdsm on the host the VM is running on and the qemu guest agent logs from within the guest.

Thank you.

Comment 18 Michal Skrivanek 2016-07-26 08:12:21 UTC
any update? Otherwise the most likely conclusion is that at the time of the snapshot the agent was not running

Comment 19 boruvka.michal 2016-08-29 06:53:21 UTC
Created attachment 1195186 [details]
Snapshot deletion fail logs 28.8.2016

Comment 20 boruvka.michal 2016-08-29 06:55:39 UTC
Snapshot deletion failed in my ovirt again. Logs are attached.


2016-08-28 21:20:38,141 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-13) [1e9e0119] Merging of snapshot '2dcd3a68-a222-4049-aa79-0dde2ec5cbf5' images '0affe31c-a40a-45c6-a8c1-a3ee0e54e518'..'990c5f2f-4561-4a8e-a0b0-da3a9ce0ef92' failed. Images have been marked illegal and can no longer be previewed or reverted to. Please retry Live Merge on the snapshot to complete the operation.

Comment 21 Tomas Jelinek 2016-09-01 10:04:47 UTC
Could you please also provide the qemu guest agent logs from within the guest?

Comment 22 boruvka.michal 2016-09-01 12:32:22 UTC
I didn't have qemu guest agent at this VM installed.

Comment 23 Tomas Jelinek 2016-09-01 13:29:52 UTC
So, that seems to be the reason, you need to have both the qemu and ovirt guest agents installed. 
Im closing this as "not a bug", please feel free to reopen once you will reproduce this with a VM where the ovirt and qemu guest agent is installed and running and provide all the logs here.


Note You need to log in before you can comment on or make changes to this bug.