Bug 1073368

Summary: [libvirt] can create live snapshot of passthrough device (iSCSI LUN or block device)
Product: Red Hat Enterprise Linux 7 Reporter: chhu
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: areis, chayang, dyuan, juzhang, michen, mzhan, pbonzini, pkrempa, qzhang, rbalakri, shyu, sluo, virt-maint, xfu, yanyang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-1.2.7-1.el7 Doc Type: Bug Fix
Doc Text:
Cause: Libvirt didn't treat the SCSI passthrough device as a host passthrough device which would usually block migration/snapshot and others. Consequence: Libvirt allowed the 'lun' passthrough devices for both local and iSCSI targets to be used with snapshot which wasn't supposed to work and caused qemu to crash or produce error message that didn't help diagnosing the problem. Fix: libvirt now correctly recognizes the 'lun' passthrough devices and forbids unsupported operations with a better error message.
Story Points: ---
Clone Of: 1034993
: 1131974 (view as bug list) Environment:
Last Closed: 2015-03-05 07:31:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1032369, 1032393, 1034993    
Bug Blocks: 1131974    

Comment 1 chhu 2014-03-12 08:31:54 UTC
1. The patch in bug1034993 comment 2 fix the issue with xml below, which is verified on that bug.
<disk type='block' device='lun'> 
  <driver name='qemu' type='raw'/>
  <source dev='/dev/disk/by-path/ip-10.66.7.108:3260-iscsi- iqn.2013-10.com.example:iscsi-lun-1'/>
......

2. This bug will track test among these xmls, which are without any patches by far:

2.1 test with xml below: 
<disk type='block' device='lun'>
  <driver name='qemu' type='qcow2'/>
  <source dev='/dev/disk/by-path/ip-10.66.7.108:3260-iscsi-iqn.2013-10.com.example:iscsi-lun-1'/>
......

==Result: start the guest met the same error as qemu-kvm bug1032369

2.2 test with xml below:
<disk type='network' device='lun'>
  <driver name='qemu' type='raw'/>
  <source protocol='iscsi' name='iqn.2013-10.com.example:iscsi/1'>
......

qemu-kvm command line: scsi-block
-drive file=iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/1,if=none,id=drive-scsi0-0-0-0,format=raw -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1

==Results: external active snapshot failed with error:
error: internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied

2.3 test with xml below:    
<disk type='network' device='lun'>
  <driver name='qemu' type='qcow2'/>
  <source protocol='iscsi' name='iqn.2013-10.com.example:iscsi/1'>

==Result: start the guest met the same error as qemu-kvm bug1032369

2.4 test with xml below:    
<disk type='network' device='disk'>
  <driver name='qemu' type='qcow2'/>
  <source protocol='iscsi' name='iqn.2013-10.com.example:iscsi/1'>

qemu-kvm command line: scsi-hd
-drive file=iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/1,if=none,id=drive-scsi0-0-0-0,format=qcow2 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1

==Result: 
1) internal active snapshot succeeded 
2) external active snapshot failed with error:
error: internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied

2.5 test with xml below:
<disk type='network' device='disk'>
  <driver name='qemu' type='raw'/>
  <source protocol='iscsi' name='iqn.2013-10.com.example:iscsi/1'>

qemu-kvm command line:
-drive file=iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/1,if=none,id=drive-scsi0-0-0-0,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1

==Result:
1) internal active snapshot not support: PASS
2) external active snapshot failed with error:
error: internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied

2.6 test with xml below: 
<disk type='volume' device='disk'>
  <driver name='qemu'/>
  <source pool='pool-iscsi' volume='unit:0:0:1' mode='direct'/>

qemu-kvm command line: scsi-hd
-drive file=iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/1,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 

==Result:
1) internal active snapshot succeeded 
2) external active snapshot failed with error:
error: internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied

2.7 test with xml below:
<disk type='volume' device='disk'>
<driver name='qemu'/>
<source pool='pool-iscsi' volume='unit:0:0:1' mode='host'/>

qemu-kvm command line: scsi-hd
-drive file=/dev/disk/by-path/ip-10.66.7.108:3260-iscsi-iqn.2013-10.com.example:iscsi-lun-1,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1

==Result:
1) internal active snapshot succeeded 
2) external active snapshot succeeded

Comment 3 Peter Krempa 2014-07-18 15:32:45 UTC
Fixed in v1.2.6-196-g1f4933f:

commit 1f4933f0f40446136b290694a276cf29df567dfe
Author: Peter Krempa <pkrempa>
Date:   Fri Jul 18 11:07:01 2014 +0200

    qemu: snapshot: Forbid snapshots of iSCSI passthrough devices
    
    As with the local SCSI passthrough devicesm qemu can't support snapshots
    on those as the block ops are handled by the device. This is also true
    for iSCSI backing of the disk. Remove the check for the local block
    device and just forbid snapshot when the disk is of type 'lun'.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1073368

commit 2a483038009d0e4136af4edb045d5902ec102aa8
Author: Peter Krempa <pkrempa>
Date:   Fri Jul 18 11:05:08 2014 +0200

    doc: domain: Clarify that disk type 'lun' works with iSCSI too
    
    Disk type 'lun' enables SCSI command passthrough for a disk. We stated
    that it works only with "block" disks. Qemu supports it also when using
    the iSCSI protocol.

Comment 5 Yang Yang 2014-09-30 10:08:30 UTC
Hi Peter,
I found that the error messages are not friendly enough 
when doing external memory only snapshot on scsi 
passthrough devices.
Does it deserve a fix?

Test on libvirt-1.2.8-4.el7.x86_64 and qemu-kvm-rhev-2.1.2-1.el7.x86_64

Test with active domain qe-con
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    qe-con                         running

Steps:
1. virsh dumpxml qe-con
<disk type='network' device='lun'>
      <driver name='qemu' type='raw' cache='none'/>
      <source protocol='iscsi' name='iqn.yy:server.target1/1'>
        <host name='10.66.100.101' port='3260'/>
      </source>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# virsh snapshot-create-as qe-con s1
error: unsupported configuration: internal snapshot for disk sda unsupported for storage type raw

# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,file=/tmp/qe-con.s1
error: unsupported configuration: external active snapshots are not supported on scsi passthrough devices

***************************
Hi Peter,
The error messages are not friendly enough 
when doing external memory only snapshot.
Does it deserve a fix?

# virsh snapshot-create-as qe-con s1 --memspec 
snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
error: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device '0000:00:07.0/ich9_ahci'

***************************

# virsh snapshot-create-as qe-con s1 --disk-only --diskspec sda,file=/tmp/qe-con.s1
error: unsupported configuration: external active snapshots are not supported on scsi passthrough devices

2. virsh dumpxml qe-con
<disk type='network' device='lun'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source protocol='iscsi' name='iqn.yy:server.target2/1'>
        <host name='10.66.100.101' port='3260'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# virsh start qe-con
error: Failed to start domain qe-con
error: internal error: process exited while connecting to monitor: 2014-09-28T03:54:54.149128Z qemu-kvm: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1: cannot get SG_IO version number: Operation not supported.  Is this a SCSI device?
2014-09-28T03:54:54.149185Z qemu-kvm: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1: Device initialization failed.
2014-09-28T03:54:54.149202Z qemu-kvm: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1: Device 'scsi-block' could not be initialized

3. virsh dumpxml qe-con
<disk type='block' device='lun'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.66.100.101:3260-iscsi-iqn.yy:server.target1-lun-1'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# virsh snapshot-create-as qe-con s1
error: unsupported configuration: internal snapshot for disk sda unsupported for storage type raw

# virsh snapshot-create-as qe-con s1 --memspec file=/tmp/s1 --diskspec sda,snapshot=external,file=/tmp/qe-con.s1
error: unsupported configuration: external active snapshots are not supported on scsi passthrough devices


***************************
Hi Peter,
The error messages are not friendly enough 
when doing external memory only snapshot.
Does it deserve a fix?

# virsh snapshot-create-as qe-con s1 --memspec 
snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
error: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device '0000:00:07.0/ich9_ahci'

***************************

# virsh snapshot-create-as qe-con s1 --disk-only --diskspec sda,snapshot=external,file=/tmp/qe-con.s1
error: unsupported configuration: external active snapshots are not supported on scsi passthrough devices

4. virsh dumpxml qe-con
<disk type='block' device='lun'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.66.100.101:3260-iscsi-iqn.yy:server.target2-lun-1'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
# virsh start qe-con
error: Failed to start domain qe-con
error: internal error: process exited while connecting to monitor: 2014-09-28T05:02:31.837167Z qemu-kvm: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1: cannot get SG_IO version number: Operation not supported.  Is this a SCSI device?
2014-09-28T05:02:31.837218Z qemu-kvm: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1: Device initialization failed.
2014-09-28T05:02:31.837236Z qemu-kvm: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1: Device 'scsi-block' could not be initialized

5. virsh dumpxml qe-con
<disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source protocol='iscsi' name='iqn.yy:server.target1/1'>
        <host name='$ip' port='3260'/>
      </source>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# virsh snapshot-create-as qe-con s1
error: unsupported configuration: internal snapshot for disk sda unsupported for storage type raw


# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,file=/tmp/qe-con.s1
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --disk-only --diskspec sda,file=/tmp/qe-con.s1
Domain snapshot s1 created

6. virsh dumpxml qe-con
<disk type='network' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source protocol='iscsi' name='iqn.yy:server.target2/1'>
        <host name='$ip' port='3260'/>
      </source>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>


# virsh snapshot-create-as qe-con s1
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,file=/tmp/qe-con.s1
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --disk-only --diskspec sda,file=/tmp/qe-con.s1
Domain snapshot s1 created

7. test with
<disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <source pool='iscsi' volume='unit:0:0:1' mode='direct'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# virsh snapshot-create-as qe-con s1
error: unsupported configuration: internal snapshot for disk sda unsupported for storage type raw

# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,file=/tmp/qe-con.s1
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --disk-only --diskspec sda,file=/tmp/qe-con.s1
Domain snapshot s1 created

8. test with
 <disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <source pool='iscsi' volume='unit:0:0:1' mode='host'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# virsh snapshot-create-as qe-con s1
error: unsupported configuration: internal snapshot for disk sda unsupported for storage type raw

# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s2 --memspec snapshot=external,file=/tmp/s2 --diskspec sda,file=/tmp/qe-con.s2
Domain snapshot s2 created

# virsh snapshot-create-as qe-con s1 --disk-only --diskspec sda,file=/tmp/qe-con.s1
Domain snapshot s1 created

9. test with
 <disk type='volume' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source pool='iscsi' volume='unit:0:0:1' mode='direct'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# virsh snapshot-create-as qe-con s1
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,file=/tmp/qe-con.s1
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --disk-only --diskspec sda,file=/tmp/qe-con.s1
Domain snapshot s1 created

10. test with
<disk type='volume' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source pool='iscsi' volume='unit:0:0:1' mode='host'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>


# virsh snapshot-create-as qe-con s1
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
Domain snapshot s1 created

# virsh snapshot-create-as qe-con s2 --memspec snapshot=external,file=/tmp/s2 --diskspec sda,file=/tmp/qe-con.s2
Domain snapshot s2 created

# virsh snapshot-create-as qe-con s1 --disk-only --diskspec sda,file=/tmp/qe-con.s1
Domain snapshot s1 created

Comment 6 Paolo Bonzini 2014-09-30 13:05:58 UTC
> The error messages are not friendly enough 
> when doing external memory only snapshot.
> Does it deserve a fix?
> 
> # virsh snapshot-create-as qe-con s1 --memspec 
> snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
> error: internal error: unable to execute QEMU command 'migrate': State blocked > by non-migratable device '0000:00:07.0/ich9_ahci'

Memory only snapshots are okay.  They should work if you use the i440fx machine type instead of q35.

Comment 7 Yang Yang 2014-10-08 08:54:53 UTC
(In reply to Paolo Bonzini from comment #6)
> > The error messages are not friendly enough 
> > when doing external memory only snapshot.
> > Does it deserve a fix?
> > 
> > # virsh snapshot-create-as qe-con s1 --memspec 
> > snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
> > error: internal error: unable to execute QEMU command 'migrate': State blocked > by non-migratable device '0000:00:07.0/ich9_ahci'
> 
> Memory only snapshots are okay.  They should work if you use the i440fx
> machine type instead of q35.

Thanks Paolo.

Re-test with clean env. Memory only snapshots are created without error.

1. test with the following xml
<disk type='network' device='lun'>
      <driver name='qemu' type='raw' cache='none'/>
      <source protocol='iscsi' name='iqn.2008-09.com.example:server.yy3/1'>
        <host name='$ip' port='3260'/>
      </source>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
Domain snapshot s1 created

2. test with the following xml
<disk type='block' device='lun'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.66.106.36:3260-iscsi-iqn.2008-09.com.example:server.yy3-lun-1'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
# virsh snapshot-create-as qe-con s1 --memspec snapshot=external,file=/tmp/s1 --diskspec sda,snapshot=no
Domain snapshot s1 created

Thanks 
Yang Yang

Comment 8 Yang Yang 2014-10-08 09:00:12 UTC
Accodding to comment 5 and 7, I'd like to set it to verified

Comment 10 errata-xmlrpc 2015-03-05 07:31:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html