Bug 873145 - 3.2 - vdsm [Storage Live Migration]: vm changes state to pause for a few seconds during storage live migration
Summary: 3.2 - vdsm [Storage Live Migration]: vm changes state to pause for a few seco...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.2.0
Hardware: ia64
OS: Linux
urgent
urgent
Target Milestone: ---
: 3.2.0
Assignee: Federico Simoncelli
QA Contact: Dafna Ron
URL:
Whiteboard: storage
Depends On: 890003 913253
Blocks: 883939 896511 915537
TreeView+ depends on / blocked
 
Reported: 2012-11-05 09:09 UTC by Dafna Ron
Modified: 2022-07-09 05:51 UTC (History)
10 users (show)

Fixed In Version: vdsm-4.10.2-2.0
Doc Type: Bug Fix
Doc Text:
Previously, during live storage migration, some virtual machines changed their state, becoming paused. VDSM was patched so that it extends volumes doubling the chunks size during live migration. This doubles the size of the watermark limit as well, and gives VDSM more time to accomplish the storage operations necessary for live migration. Virtual machines no longer pause during live storage migration.
Clone Of:
Environment:
Last Closed: 2013-06-10 20:33:58 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
logs (704.22 KB, application/x-gzip)
2012-11-05 09:09 UTC, Dafna Ron
no flags Details
## Logs vdsm, rhevm, libvirt, screen-shots (924.37 KB, application/x-gzip)
2012-12-11 12:32 UTC, vvyazmin@redhat.com
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-47028 0 None None None 2022-07-09 05:51:12 UTC
Red Hat Product Errata RHSA-2013:0886 0 normal SHIPPED_LIVE Moderate: rhev 3.2 - vdsm security and bug fix update 2013-06-11 00:25:02 UTC

Description Dafna Ron 2012-11-05 09:09:03 UTC
Created attachment 638428 [details]
logs

Description of problem:

I noticed that some of the vm change state to pause during live storage migration. 
looking at vdsm log I can see the following error: 

libvirtEventLoop::ERROR::2012-11-05 10:37:14,239::libvirtvm::1982::vm.Vm::(_onBlockJobEvent) vmId=`06c4d8e1-b2d1-4b1d-be75-7eb486201709`::Live merge completed for an unexpected path: /rhev/data-center/2fe7a6d8-3a87-42a4-993e-0b4cdaa1e32
7/ebe96ea4-b0d6-4320-ac35-fb3eb15c7b9e/images/19a3be6d-ad3d-48a6-bb22-536145ecb503/651dcb10-8bd1-4992-8932-52cffaf66c49


Version-Release number of selected component (if applicable):

si23
vdsm-4.9.6-40.0.el6_3.x86_64
libvirt-0.9.10-21.el6_3.5.x86_64
qemu-kvm-rhev-0.12.1.2-2.295.el6_3.5.x86_64

How reproducible:

intermittent

Steps to Reproduce:
1. run a vm and move disk
2.
3.
  
Actual results:

vm changes state to pasue for a few seconds

Expected results:

vm should not change state to pause

Additional info: attaching vdsm,libvirt, engine and domain log

the storage I used was iscsi
the vm has 3 disks with no OS installed. 


in attached libvirt log we can see errors while sync is sent: 

2012-11-05 08:36:05.178+0000: 2108: debug : qemuAgentGuestSync:937 : Sending guest-sync command with ID: 1352104565178
2012-11-05 08:36:08.379+0000: 2107: debug : virDomainInterfaceStats:7223 : dom=0x7f24cc0d6540, (VM: name=VN, uuid=06c4d8e1-b2d1-4b1d-be75-7eb486201709), path=vnet0, stats=0x7f24fa2beb00, size=64
2012-11-05 08:36:10.000+0000: 2108: error : qemuAgentSend:877 : internal error Guest agent not available for now
2012-11-05 08:36:10.000+0000: 2108: debug : qemuAgentGuestSync:941 : qemuAgentSend returned: -2
2012-11-05 08:36:10.000+0000: 2108: debug : qemuAgentGuestSync:937 : Sending guest-sync command with ID: 1352104570000
2012-11-05 08:36:15.000+0000: 2108: error : qemuAgentSend:877 : internal error Guest agent not available for now
2012-11-05 08:36:15.000+0000: 2108: debug : qemuAgentGuestSync:941 : qemuAgentSend returned: -2

and domain log shows the following: 

Domain id=34 is tainted: custom-monitor
handle_dev_stop: stop
handle_dev_stop: stop
handle_dev_stop: stop

Comment 5 Ayal Baron 2012-11-12 15:07:01 UTC
Fix should be to double size of extends (2G instead of 1G)

Comment 7 Federico Simoncelli 2012-12-03 12:54:35 UTC
commit 679b45308ac69a5baed98318a11c16771e339336
Author: Federico Simoncelli <fsimonce>
Date:   Mon Nov 12 08:18:12 2012 -0500

    vm: increase the volume extension on storage migration
    
    During live migration VDSM needs to subsequently extend two volumes
    instead of one; doubling the size of the chunk to extend the watermark
    limit is doubled and VDSM has more time to accomplish the operations.
    
    Change-Id: Ib61375613712feb7118a80c50b73e678d257f251
    Signed-off-by: Federico Simoncelli <fsimonce>

http://gerrit.ovirt.org/#/c/9200/

Comment 9 vvyazmin@redhat.com 2012-12-11 12:32:55 UTC
Created attachment 661400 [details]
## Logs vdsm, rhevm, libvirt, screen-shots

Comment 13 Dafna Ron 2012-12-24 13:08:36 UTC
cannot be tested on sf1 - blocked by: https://bugzilla.redhat.com/show_bug.cgi?id=890003

Comment 14 Dafna Ron 2013-02-21 11:03:36 UTC
blocked by 913253

Comment 16 Dafna Ron 2013-03-13 13:17:34 UTC
verified on sf10 with vdsm-4.10.2-11.0.el6ev.x86_64 and libvirt-0.10.2-18.el6_4.eblake.2.x86_64

1e919292-b5a9-4042-b178-c3a133b4b9be::DEBUG::2013-03-13 14:36:47,635::threadPool::200::Misc.ThreadPool.WorkerThread::(run) Task: 1e919292-b5a9-4042-b178-c3a133b4b9be running: <
function runTask at 0x1bd6488> with: (<bound method LibvirtVm.__afterVolumeExtension of <libvirtvm.LibvirtVm object at 0x7fc6900b9750>>, {'newSize': 3072, 'domainID': '52e4917b
-10bc-4b03-9b82-a2e15732c15b', 'name': u'hda', 'volumeID': 'f421e710-86c7-4c74-9f06-aae7b07656f0', 'imageID': '2a70839f-2282-469d-89c1-59970c082736', 'poolID': '696adcda-73f3-4
6b3-ae74-a188c459dfcc'})

Comment 17 Cheryn Tan 2013-04-03 07:01:54 UTC
This bug is currently attached to errata RHBA-2012:14332. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.

* Consequence: What happens when the bug presents.

* Fix: What was done to fix the bug.

* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes

Thanks in advance.

Comment 18 Federico Simoncelli 2013-06-07 10:02:58 UTC
Few changes in the Doc Text.

Comment 20 errata-xmlrpc 2013-06-10 20:33:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0886.html


Note You need to log in before you can comment on or make changes to this bug.