Bug 1250590 - Migration with combined local and network storage fails on assertion error
Migration with combined local and network storage fails on assertion error
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.1
x86_64 All
unspecified Severity high
: rc
: ---
Assigned To: Ademar Reis
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-05 10:02 EDT by Jiri Lunacek
Modified: 2015-10-15 19:41 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-15 19:41:54 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Spec file of used build (403.42 KB, text/plain)
2015-08-18 04:25 EDT, Jiri Lunacek
no flags Details

  None (edit)
Description Jiri Lunacek 2015-08-05 10:02:40 EDT
Description of problem:
When a guest has both local raw block storage and remote (directly attached) iscsi storage, migration fails after local storage has been trasnfered.

Log on source states:
qemu-kvm: block/mirror.c:359: mirror_run: Assertion `n > 0' failed.
2015-08-05 13:42:39.769+0000: shutting down

Guest is not running on either host when this happens.

Version-Release number of selected component (if applicable):
1.5.3-86.el7_1.5

Steps to Reproduce:
1. Create a guest with similar block device configuration to this (snippet from my testing guest):
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <source dev='/dev/HostVG/virtual_56070'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <auth username='csuser910'>
        <secret type='iscsi' usage='iqn.2011-09.cz.hosting90:target774_csuser910'/>
      </auth>
      <source protocol='iscsi' name='iqn.2011-09.cz.hosting90:target774/1'>
        <host name='10.0.3.1' port='3260'/>
      </source>
      <target dev='sdb' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
2. Prepare secret objects on target host for the iscsi
3. Start migration to another host with VIR_MIGRATE_NON_SHARED_DISK flag set

Actual results:
The local storage is copied using nbd but at the end it fails with
qemu-kvm: block/mirror.c:359: mirror_run: Assertion `n > 0' failed.

Expected results:
The guest should successfully migrate

Additional info:
The guest is using virtio-scsi bus.
Comment 2 Ademar Reis 2015-08-07 12:49:29 EDT
Jiri, thanks for taking the time to enter a bug report with us. We use reports like yours to keep improving the quality of our products and releases. That said, we're not able to guarantee the timeliness or suitability of a resolution for issues entered here because this is not a mechanism for requesting support.

If this issue is critical or in any way time sensitive, please raise a ticket through your regular Red Hat support channels to make certain it receives the proper attention and prioritization that will result in a timely resolution.

For information on how to contact the Red Hat production support team, please visit: https://www.redhat.com/support/process/production/#howto

Setting NEEDINFO(QE) to try to reproduce it.
Comment 3 juzhang 2015-08-10 02:31:33 EDT
Hi Shaolong,

Could you have a try and update it the bz?

Best Regards,
Junyi
Comment 4 Shaolong Hu 2015-08-12 02:59:09 EDT
Hi Jiri,

In Redhat product, only qemu-kvm-rhev version supports storage migration function, qemu-kvm-1.5.3-86.el7_1.5(without rhev) does not, could you confirm your version?


Bests,
Shaolong
Comment 5 Jiri Lunacek 2015-08-18 04:25:47 EDT
Created attachment 1064225 [details]
Spec file of used build

I am aware of that. Since we are using CentOS which currently has no official rhev build of qemu-kvm (we filled a bug for this: https://bugs.centos.org/view.php?id=8281), we are using custom build.

Source rpm version is:
qemu-kvm-1.5.3-86.el7_1.5

I am attaching spec file of the build.
Comment 6 Jiri Lunacek 2015-09-22 05:31:19 EDT
Note:
The bug is also present in package version qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64 from CentOS oVirt repository.

Due to this I filed a bug in CentOS bug tracker too: https://bugs.centos.org/view.php?id=9497
Comment 9 Jiri Lunacek 2015-10-13 03:53:42 EDT
Workaround/solution:

Adding <shareable/> tag to the iSCSI storage solves the issue and results in desired behaviour. I.e. the local storage gets migrated and the iSCSI storage is only re-attached on remote side.

This whole thing may have been a missconfiguration. However a more desciptive error message may be a good idea.
Comment 10 Ademar Reis 2015-10-15 19:41:54 EDT
Jiri, I checked with our QE and they couldn't reproduce it, so it seems it was a configuration mistake indeed.

Since we don't support direct configuration (without using a management tool), I'm closing this bug.

Anyway, thanks for reporting it.

Note You need to log in before you can comment on or make changes to this bug.