Bug 1250590

Summary: Migration with combined local and network storage fails on assertion error
Product: Red Hat Enterprise Linux 7 Reporter: Jiri Lunacek <jiri.lunacek>
Component: qemu-kvm-rhevAssignee: Ademar Reis <areis>
Status: CLOSED WORKSFORME QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.1CC: huding, jiri.lunacek, juzhang, knoel, pezhang, shu, virt-bugs, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-15 23:41:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Spec file of used build none

Description Jiri Lunacek 2015-08-05 14:02:40 UTC
Description of problem:
When a guest has both local raw block storage and remote (directly attached) iscsi storage, migration fails after local storage has been trasnfered.

Log on source states:
qemu-kvm: block/mirror.c:359: mirror_run: Assertion `n > 0' failed.
2015-08-05 13:42:39.769+0000: shutting down

Guest is not running on either host when this happens.

Version-Release number of selected component (if applicable):
1.5.3-86.el7_1.5

Steps to Reproduce:
1. Create a guest with similar block device configuration to this (snippet from my testing guest):
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <source dev='/dev/HostVG/virtual_56070'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <auth username='csuser910'>
        <secret type='iscsi' usage='iqn.2011-09.cz.hosting90:target774_csuser910'/>
      </auth>
      <source protocol='iscsi' name='iqn.2011-09.cz.hosting90:target774/1'>
        <host name='10.0.3.1' port='3260'/>
      </source>
      <target dev='sdb' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
2. Prepare secret objects on target host for the iscsi
3. Start migration to another host with VIR_MIGRATE_NON_SHARED_DISK flag set

Actual results:
The local storage is copied using nbd but at the end it fails with
qemu-kvm: block/mirror.c:359: mirror_run: Assertion `n > 0' failed.

Expected results:
The guest should successfully migrate

Additional info:
The guest is using virtio-scsi bus.

Comment 2 Ademar Reis 2015-08-07 16:49:29 UTC
Jiri, thanks for taking the time to enter a bug report with us. We use reports like yours to keep improving the quality of our products and releases. That said, we're not able to guarantee the timeliness or suitability of a resolution for issues entered here because this is not a mechanism for requesting support.

If this issue is critical or in any way time sensitive, please raise a ticket through your regular Red Hat support channels to make certain it receives the proper attention and prioritization that will result in a timely resolution.

For information on how to contact the Red Hat production support team, please visit: https://www.redhat.com/support/process/production/#howto

Setting NEEDINFO(QE) to try to reproduce it.

Comment 3 juzhang 2015-08-10 06:31:33 UTC
Hi Shaolong,

Could you have a try and update it the bz?

Best Regards,
Junyi

Comment 4 Shaolong Hu 2015-08-12 06:59:09 UTC
Hi Jiri,

In Redhat product, only qemu-kvm-rhev version supports storage migration function, qemu-kvm-1.5.3-86.el7_1.5(without rhev) does not, could you confirm your version?


Bests,
Shaolong

Comment 5 Jiri Lunacek 2015-08-18 08:25:47 UTC
Created attachment 1064225 [details]
Spec file of used build

I am aware of that. Since we are using CentOS which currently has no official rhev build of qemu-kvm (we filled a bug for this: https://bugs.centos.org/view.php?id=8281), we are using custom build.

Source rpm version is:
qemu-kvm-1.5.3-86.el7_1.5

I am attaching spec file of the build.

Comment 6 Jiri Lunacek 2015-09-22 09:31:19 UTC
Note:
The bug is also present in package version qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64 from CentOS oVirt repository.

Due to this I filed a bug in CentOS bug tracker too: https://bugs.centos.org/view.php?id=9497

Comment 9 Jiri Lunacek 2015-10-13 07:53:42 UTC
Workaround/solution:

Adding <shareable/> tag to the iSCSI storage solves the issue and results in desired behaviour. I.e. the local storage gets migrated and the iSCSI storage is only re-attached on remote side.

This whole thing may have been a missconfiguration. However a more desciptive error message may be a good idea.

Comment 10 Ademar Reis 2015-10-15 23:41:54 UTC
Jiri, I checked with our QE and they couldn't reproduce it, so it seems it was a configuration mistake indeed.

Since we don't support direct configuration (without using a management tool), I'm closing this bug.

Anyway, thanks for reporting it.