Bug 1285236 - [RFE] provide feedback about the status of the recovery
[RFE] provide feedback about the status of the recovery
Product: vdsm
Classification: oVirt
Component: Core (Show other bugs)
Unspecified Unspecified
medium Severity low (vote)
: ovirt-3.6.2
: 4.17.14
Assigned To: Francesco Romani
sefi litmanovich
: FutureFeature
Depends On:
  Show dependency treegraph
Reported: 2015-11-25 04:33 EST by Francesco Romani
Modified: 2016-02-18 06:11 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Improve feedback during the recovery flow, adding more logs Reason: Helps the administrator understanding what VDSM is doing during the recovery, and provide progress report. Result: Vdsm gained more logs during the recovery process, explaining the progress of current operations and the steps performed
Story Points: ---
Clone Of:
Last Closed: 2016-02-18 06:11:51 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Virt
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
michal.skrivanek: ovirt‑3.6.z?
rule-engine: planning_ack?
michal.skrivanek: devel_ack+
rule-engine: testing_ack+

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 49140 ovirt-3.6 MERGED clientIF: add logs during the recovery Never

  None (edit)
Description Francesco Romani 2015-11-25 04:33:12 EST
Description of problem:
The recovery procedure may take some time, so it is useful to have feedback, at least in VDSM logs, about the status, and also an estimate of the total duration.

Furthermore, recovery should clearly signal when it is completed. This is especially useful on post-mortem analysis, when one can no longer poll VDSM to learn when recovery is done

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. restart VDSM when it is running some VMs

Actual results:
Hard to know when the recovery is completed. One has to closely monitor the VDSM logs, knowing what to look for, and/or poll VDSM, for example using vdsClient. When VDSM start to answer to queries, recovery is concluded.

Expected results:
VDSM should clearly report, at least in the logs, how recovery is going and when it is completed.

Additional info:
logs are useful for post-mortem analysis, so it is good approach to fix.
Adding events to send to Engine could be a nice plus for a second step.
Comment 1 Red Hat Bugzilla Rules Engine 2015-11-25 05:24:06 EST
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 2 Red Hat Bugzilla Rules Engine 2015-11-25 05:24:06 EST
This bug is not marked for z-stream, yet the milestone is for a z-stream version, therefore the milestone has been reset.
Please set the correct milestone or add the z-stream flag.
Comment 3 Sandro Bonazzola 2015-12-22 06:03:28 EST
Can you please add some doc-text?
Comment 4 Sandro Bonazzola 2015-12-23 08:44:31 EST
oVirt 3.6.2 RC1 has been released for testing, moving to ON_QA
Comment 5 Francesco Romani 2016-01-04 07:05:49 EST
Added doc text, even though I don't think this deserves mentioning in the documentation, maybe just one short sentence like "Vdsm logs the progress of recovering"
Comment 6 sefi litmanovich 2016-02-16 07:51:52 EST
Verified with:

Engine: rhevm-3.6.3-0.1.el6.noarch
Host: vdsm-4.17.19-0.el7ev.noarch

1. Started 2 windows VMs with floppy disk payload with sysprep.
2. After the VMs started and are on up state - restart vdsmd service on the host.

Result (there was a failure in recovery but out of the scope of this bz - opening a seperate one):

[root@{hostname}]# tail -F /var/log/vdsm/vdsm.log | egrep -i --color "recovery"
clientIFinit::DEBUG::2016-02-10 14:11:42,613::clientIF::462::vds::(_recoverExistingVms) recovery: started
clientIFinit::INFO::2016-02-10 14:11:42,675::clientIF::479::vds::(_recoverExistingVms) recovery [1:1/2]: recovered domain 0a0bfec9-713c-4b6a-bdab-bb1b762ed432 from libvirt
clientIFinit::INFO::2016-02-10 14:11:42,679::clientIF::479::vds::(_recoverExistingVms) recovery [1:2/2]: recovered domain 7d8429f5-5482-4a64-bb18-1c300ca1bf5c from libvirt
clientIFinit::INFO::2016-02-10 14:11:42,679::clientIF::521::vds::(_recoverExistingVms) recovery: waiting for 2 domains to go up
clientIFinit::INFO::2016-02-10 14:11:43,681::clientIF::532::vds::(_recoverExistingVms) recovery: waiting for storage pool to go up
clientIFinit::INFO::2016-02-10 14:11:48,684::clientIF::532::vds::(_recoverExistingVms) recovery: waiting for storage pool to go up
clientIFinit::INFO::2016-02-10 14:11:53,687::clientIF::544::vds::(_recoverExistingVms) recovery [1/2]: preparing paths for domain 7d8429f5-5482-4a64-bb18-1c300ca1bf5c
clientIFinit::ERROR::2016-02-10 14:11:53,705::clientIF::550::vds::(_recoverExistingVms) recovery [1/2]: failed for vm 7d8429f5-5482-4a64-bb18-1c300ca1bf5c
clientIFinit::INFO::2016-02-10 14:11:53,706::clientIF::544::vds::(_recoverExistingVms) recovery [2/2]: preparing paths for domain 0a0bfec9-713c-4b6a-bdab-bb1b762ed432
clientIFinit::ERROR::2016-02-10 14:11:53,711::clientIF::550::vds::(_recoverExistingVms) recovery [2/2]: failed for vm 0a0bfec9-713c-4b6a-bdab-bb1b762ed432
clientIFinit::INFO::2016-02-10 14:11:53,711::clientIF::553::vds::(_recoverExistingVms) recovery: completed in 11s

also tested with linux vm with no payload. No failure in recovery, reports are correct.

Note You need to log in before you can comment on or make changes to this bug.