Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 986914

Summary:

[vdsm][SLM] Cannot live migrate disk due to replication is already running

Product:

Red Hat Enterprise Virtualization Manager

Reporter:

Jakub Libosvar <jlibosva>

Component:

vdsm

Assignee:

Ayal Baron <abaron>

Status:

CLOSED DUPLICATE

QA Contact:

Aharon Canan <acanan>

Severity:

urgent

Docs Contact:

Priority:

unspecified

Version:

3.3.0

CC:

abaron, bazulay, hateya, iheim, lpeer, mbourvin, scohen

Target Milestone:

---

Keywords:

Regression, TestBlocker, Triaged

Target Release:

3.3.0

Flags:

scohen: Triaged+

Hardware:

All

OS:

Linux

Whiteboard:

storage

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2013-08-04 12:34:35 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

Storage

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
collected logs	none

Description Jakub Libosvar 2013-07-22 12:21:43 UTC

Created attachment 776884 [details]
collected logs

Description of problem:
When performing live migration of a disk, it fails with error in vdsm that Disk 'vda' already has an ongoing replication. The request for migration is done via API:

2013-07-22 12:15:58,977 - Thread-101 - disks - DEBUG - Action request content is --  url:/api/vms/6c113a5d-7a4f-46ec-9d10-aa4b62eb565c/disks/c034da5c-140e-45ec-9935-07ad7fc53c41/move body:<action>
    <async>false</async>
    <grace_period>
        <expiry>10</expiry>
    </grace_period>
    <storage_domain href="/api/storagedomains/0d52c346-1640-4ced-a1b6-73d71abdc1e9" id="0d52c346-1640-4ced-a1b6-73d71abdc1e9">
        <name>iscsi_1</name>
        <link href="/api/storagedomains/0d52c346-1640-4ced-a1b6-73d71abdc1e9/permissions" rel="permissions"/>
        <link href="/api/storagedomains/0d52c346-1640-4ced-a1b6-73d71abdc1e9/disks" rel="disks"/>
        <type>data</type>
        <master>false</master>
        <storage>
            <type>iscsi</type>
            <volume_group id="0aEhCn-qS73-3in2-WkZf-6VrG-meEx-P7XPrd">
                <logical_unit id="3514f0c54d5c001ab">
                    <address>10.35.160.7</address>
                    <port>3260</port>
                    <target>iqn.2008-05.com.xtremio:001e675b8ee0</target>
                    <serial>SXtremIO_XtremApp_4d2590bfc18d4e4093657764290dccaf</serial>
                    <vendor_id>XtremIO</vendor_id>
                    <product_id>XtremApp</product_id>
                    <lun_mapping>15</lun_mapping>
                    <portal>10.35.160.7:3260,1</portal>
                    <size>322122547200</size>
                    <paths>0</paths>
                    <volume_group_id>0aEhCn-qS73-3in2-WkZf-6VrG-meEx-P7XPrd</volume_group_id>
                    <storage_domain_id>0d52c346-1640-4ced-a1b6-73d71abdc1e9</storage_domain_id>
                </logical_unit>
            </volume_group>
        </storage>
        <available>316753838080</available>
        <used>4294967296</used>
        <committed>0</committed>
        <storage_format>v3</storage_format>
    </storage_domain>
</action>

2013-07-22 12:17:22,571 INFO  [org.ovirt.engine.core.bll.lsm.LiveMigrateDiskCommand] (pool-5-thread-50) [123a951] Running command: LiveMigrateDiskCommand Task handler: VmReplicateDiskStartTaskHandler internal: false. Entities affected :  ID: c034da5c-140e-45ec-9935-07ad7fc53c41 Type: Disk,  ID: 0d52c346-1640-4ced-a1b6-73d71abdc1e9 Type: Storage
2013-07-22 12:17:22,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-50) [123a951] START, VmReplicateDiskStartVDSCommand(HostName = 10.35.160.41, HostId = 0d4a70d5-4254-4a1e-8de0-c0a71561a0d1, vmId=6c113a5d-7a4f-46ec-9d10-aa4b62eb565c), log id: 2badfb0c
2013-07-22 12:17:22,576 INFO  [org.ovirt.engine.core.bll.lsm.LiveMigrateDiskCommand] (pool-5-thread-49) [36c6000d] Ending command successfully: org.ovirt.engine.core.bll.lsm.LiveMigrateDiskCommand
2013-07-22 12:17:22,580 INFO  [org.ovirt.engine.core.bll.lsm.LiveMigrateDiskCommand] (pool-5-thread-49) [36c6000d] Running command: LiveMigrateDiskCommand Task handler: VmReplicateDiskStartTaskHandler internal: false. Entities affected :  ID: aae4f168-8b5f-4a3e-8707-f6d5dd348bf0 Type: Disk,  ID: 0d52c346-1640-4ced-a1b6-73d71abdc1e9 Type: Storage
2013-07-22 12:17:22,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-49) [36c6000d] START, VmReplicateDiskStartVDSCommand(HostName = 10.35.160.41, HostId = 0d4a70d5-4254-4a1e-8de0-c0a71561a0d1, vmId=40dd0d44-7f40-4290-817e-35246760a0f6), log id: 2c247f34
2013-07-22 12:17:22,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-49) [36c6000d] Failed in VmReplicateDiskStartVDS method
2013-07-22 12:17:22,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-50) [123a951] Failed in VmReplicateDiskStartVDS method
2013-07-22 12:17:22,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-49) [36c6000d] Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to VmReplicateDiskStartVDS, error = Unexpected exception
2013-07-22 12:17:22,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-50) [123a951] Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to VmReplicateDiskStartVDS, error = Unexpected exception
2013-07-22 12:17:22,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-49) [36c6000d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand return value
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=16, mMessage=Unexpected exception]]
2013-07-22 12:17:22,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-50) [123a951] Command org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand return value
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=16, mMessage=Unexpected exception]]
2013-07-22 12:17:22,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-50) [123a951] HostName = 10.35.160.41
2013-07-22 12:17:22,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-49) [36c6000d] HostName = 10.35.160.41
2013-07-22 12:17:22,989 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-50) [123a951] Command VmReplicateDiskStartVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskStartVDS, error = Unexpected exception
2013-07-22 12:17:22,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-50) [123a951] FINISH, VmReplicateDiskStartVDSCommand, log id: 2badfb0c
2013-07-22 12:17:22,990 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-49) [36c6000d] Command VmReplicateDiskStartVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskStartVDS, error = Unexpected exception
2013-07-22 12:17:22,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (pool-5-thread-49) [36c6000d] FINISH, VmReplicateDiskStartVDSCommand, log id: 2c247f34
2013-07-22 12:17:22,990 ERROR [org.ovirt.engine.core.bll.lsm.VmReplicateDiskStartTaskHandler] (pool-5-thread-50) [123a951] Failed VmReplicateDiskStart (Disk c034da5c-140e-45ec-9935-07ad7fc53c41 , VM 6c113a5d-7a4f-46ec-9d10-aa4b62eb565c)
2013-07-22 12:17:22,997 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateDiskCommand] (pool-5-thread-50) [123a951] Command org.ovirt.engine.core.bll.lsm.LiveMigrateDiskCommand throw Vdc Bll exception. With error message VdcBLLException: Unexpected exception (Failed with VDSM error unexpected and code 16)
2013-07-22 12:17:22,990 ERROR [org.ovirt.engine.core.bll.lsm.VmReplicateDiskStartTaskHandler] (pool-5-thread-49) [36c6000d] Failed VmReplicateDiskStart (Disk aae4f168-8b5f-4a3e-8707-f6d5dd348bf0 , VM 40dd0d44-7f40-4290-817e-35246760a0f6)


Thread-659::DEBUG::2013-07-22 12:17:22,717::BindingXMLRPC::979::vds::(wrapper) client [10.35.161.89]::call vmDiskReplicateStart with ('6c113a5d-7a4f-46ec-9d10-aa4b62eb565c', {'device': 'disk', 'domainID': '6146e022-f7c0-4c94-a927-2e44b821bb6c', 'volumeID': '36d1d53c-5c01-4769-9977-4bc81bd6ba44', 'poolID': '57719734-f9a5-4de7-a212-9548273fbf08', 'imageID': 'c034da5c-140e-45ec-9935-07ad7fc53c41'}, {'device': 'disk', 'domainID': '0d52c346-1640-4ced-a1b6-73d71abdc1e9', 'volumeID': '36d1d53c-5c01-4769-9977-4bc81bd6ba44', 'poolID': '57719734-f9a5-4de7-a212-9548273fbf08', 'imageID': 'c034da5c-140e-45ec-9935-07ad7fc53c41'}) {} flowID [123a951]
Thread-659::ERROR::2013-07-22 12:17:23,096::BindingXMLRPC::998::vds::(wrapper) unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 984, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/BindingXMLRPC.py", line 373, in vmDiskReplicateStart
    return vm.diskReplicateStart(srcDisk, dstDisk)
  File "/usr/share/vdsm/API.py", line 529, in diskReplicateStart
    return v.diskReplicateStart(srcDisk, dstDisk)
  File "/usr/share/vdsm/vm.py", line 3828, in diskReplicateStart
    self._setDiskReplica(srcDrive, dstDisk)
  File "/usr/share/vdsm/vm.py", line 3778, in _setDiskReplica
    "replication" % srcDrive.name)
RuntimeError: Disk 'vda' already has an ongoing replication


Version-Release number of selected component (if applicable):
rhevm-3.3.0-0.9.master.el6ev.noarch
vdsm-4.12.0-rc1.12.git8ee6885.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Migrate a disk of running VM
2.
3.

Actual results:
Fails

Expected results:
Succeeds

Additional info:
It worked in is5

Comment 3 Ayal Baron 2013-08-04 12:34:35 UTC


*** This bug has been marked as a duplicate of bug 985735 ***