Bug 1200206 - Guest fail to restore when disk is iscsi lun using authenticate pool/volume source with direct mode
Summary: Guest fail to restore when disk is iscsi lun using authenticate pool/volume s...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: John Ferlan
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-10 01:38 UTC by Pei Zhang
Modified: 2015-11-19 06:19 UTC (History)
7 users (show)

Fixed In Version: libvirt-1.2.17-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-19 06:19:48 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2202 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2015-11-19 08:17:58 UTC

Description Pei Zhang 2015-03-10 01:38:40 UTC
Description of problem:
Guest fail to restore when disk is iscsi lun using authenticate pool/volume source with direct mode. Guest save and restore successfully when using host mode.

Version-Release number of selected component (if applicable):
libvirt-1.2.8-16.el7.x86_64
qemu-kvm-rhev-2.2.0-5.el7.x86_64
kernel-3.10.0-229.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare authenticate iscsi target server 
#vim /etc/tgt/targets.conf
......
<target iqn.2008-09.5.165.3ffe.104:server.target1>
    backing-store /mnt/server/iscsi/iscsi.img
    incominguser rhat rhatrhat
</target>
......
#service tgtd restart 

2.on iscsi client using authenticate iscsi lun. 
2.1 define a secret 
# cat iscsi-secret.xml 
 <secret ephemeral='no' private='yes'>
         <description>iSCSI secret</description>
         <auth type='chap' username='rhat'/>
         <usage type='iscsi'>
            <target>libvirtiscsi</target>
         </usage>
      </secret>
# virsh secret-define iscsi-secret.xml 
Secret 653f8bfe-7e1a-4e55-904f-608ced9d563b created

# virsh secret-list 
 UUID                                  Usage
--------------------------------------------------------------------------------
 653f8bfe-7e1a-4e55-904f-608ced9d563b  iscsi libvirtiscsi

2.2 set value for secret 

# MYSECRET=`printf %s "rhatrhat" | base64`
# virsh secret-set-value 653f8bfe-7e1a-4e55-904f-608ced9d563b $MYSECRET
Secret value set

2.3 create a iscsi pool with authentication

# virsh pool-dumpxml iscsi-pool
<pool type='iscsi'>
  <name>iscsi-pool</name>
  ...
  <source>
    <host name='ip' port='3260'/>
    <device path='iqn.2008-09.5.165.3ffe.104:server.target1'/>
    <auth type='chap' username='rhat'>
      <secret type='iscsi' usage='libvirtiscsi'/>
    </auth>
  </source>
  <target>
    <path>/dev/disk/by-path</path>
   ...
</pool>
# virsh pool-define iscsi-pool-secret.xml 
Pool iscsi-pool defined from iscsi-pool-secret.xml

# virsh pool-start iscsi-pool
Pool iscsi-pool started

# virsh vol-list iscsi-pool
 Name                 Path                                    
------------------------------------------------------------------------------
 unit:0:0:1           /dev/disk/by-path/ip-$ip:3260-iscsi-iqn.2008-09.5.165.3ffe.104:server.target1-lun-1

3.define and start a guest with following XML :
start a guest with source mode is direct .
......
<disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <source pool='iscsi-pool' volume='unit:0:0:1' mode='direct'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
......

3.1  save and restore 
# virsh save iscsi-vol iscsi.save

Domain iscsi-vol saved to iscsi.save

# virsh restore iscsi.save 
error: Failed to restore domain from iscsi.save
error: internal error: invalid secret type 'iscsi'


Actual results:
As step3.1 guest fail to restore .

Expected results:
Guest restore successfully.

Additional info:
Also fail to create snapshot .
# virsh snapshot-create-as iscsi-vol s1 --disk-only --diskspec vda,file=/tmp/snap1
error: internal error: invalid secret type 'iscsi'

Comment 1 Yang Yang 2015-03-10 02:37:09 UTC
The real problem is that <auth> field will be automatically generated in domain's disk element when vm starts up. However, as described about 'auth' element in libvirt.org, it seems that auth element is NOT supported for a disk type 'volume'. e.g.
auth
    The auth element is supported for a disk type "network" that is using a source element with the protocol attributes "rbd" or "iscsi".

If I defined an iscsi type pool not specifying secret type in source element, secret type will be 'null' in the automatically generated 'auth' element in domain's disk element. If I defined an iscsi type pool specifying secret type with 'iscsi' in source element, secret type will be 'iscsi' in the automatically generated 'auth' element in domain's disk element.

For example:
scenario 1:
1. define and start iscsi pool with secret type 'null'
# virsh pool-dumpxml iscsi
<pool type='iscsi'>
  <name>iscsi</name>
  <uuid>4e7eed01-b53e-41c1-9d3b-41c23955496c</uuid>
  <capacity unit='bytes'>64424509440</capacity>
  <allocation unit='bytes'>64424509440</allocation>
  <available unit='bytes'>0</available>
  <source>
    <host name='10.66.x.x' port='3260'/>
    <device path='iqn.2008-09.5.165.3ffe.104:server.target1'/>
    <auth type='chap' username='rhat'>
      <secret usage='libvirtiscsi'/>
    </auth>
  </source>
  <target>
    <path>/dev/disk/by-path</path>
    <permissions>
      <mode>0755</mode>
      <owner>-1</owner>
      <group>-1</group>
    </permissions>
  </target>

2. start vm with disk type 'volume' and source mode 'direct'
<disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <source pool='iscsi' volume='unit:0:0:1' mode='direct'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
</pool>

3. check the domain xml, the auth element is generated with secret type 'null'
<disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <auth type='chap' username='rhat'>
        <secret usage='libvirtiscsi'/>
      </auth>
      <source pool='iscsi' volume='unit:0:0:1' mode='direct'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

4. create snapshot
# virsh snapshot-create-as virt-tests-vm1 --disk-only s1 --diskspec vda,file=/tmp/s1
error: unsupported configuration: invalid secret type (null)

Scenario 2
1. define and start iscsi pool with secret type 'null'
# virsh pool-dumpxml iscsi
<pool type='iscsi'>
  <name>iscsi</name>
  <uuid>4e7eed01-b53e-41c1-9d3b-41c23955496c</uuid>
  <capacity unit='bytes'>64424509440</capacity>
  <allocation unit='bytes'>64424509440</allocation>
  <available unit='bytes'>0</available>
  <source>
    <host name='10.66.x.x' port='3260'/>
    <device path='iqn.2008-09.5.165.3ffe.104:server.target1'/>
    <auth type='chap' username='rhat'>
      <secret type='iscsi' usage='libvirtiscsi'/>
    </auth>
  </source>
  <target>
    <path>/dev/disk/by-path</path>
    <permissions>
      <mode>0755</mode>
      <owner>-1</owner>
      <group>-1</group>
    </permissions>
  </target>

2. start vm with disk type 'volume' and source mode 'direct'
<disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <source pool='iscsi' volume='unit:0:0:1' mode='direct'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
</pool>

3. check the domain xml, the auth element is generated with secret type 'null'
<disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <auth type='chap' username='rhat'>
        <secret type='iscsi' usage='libvirtiscsi'/>
      </auth>
      <source pool='iscsi' volume='unit:0:0:1' mode='direct'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

4. create snapshot
# virsh snapshot-create-as virt-tests-vm1 --disk-only s1 --diskspec vda,file=/tmp/s1
error: internal error: invalid secret type 'iscsi'

Comment 2 John Ferlan 2015-06-08 23:01:26 UTC
Patch posted upstream:

http://www.redhat.com/archives/libvir-list/2015-June/msg00329.html

Comment 3 John Ferlan 2015-06-15 11:28:35 UTC
Patch has been pushed upstream:

commit 1feaccf0001794e167b0374abf75d97024414e27
Author: John Ferlan <jferlan@redhat.com>
Date:   Mon Jun 8 18:41:28 2015 -0400

    storage: Need to set secrettype for direct iscsi disk volume

    Commit id '1b4eaa61' added the ability to have a mode='direct' for
    an iscsi disk volume.  It relied on virStorageTranslateDiskSourcePool
    in order to copy any disk source pool authentication information to
    the direct disk volume, but it neglected to also copy the 'secrettype'
    field which ends up being used in the domain volume formatting code.
    Adding a secrettype for this case will allow for proper formatting later
    and allow disk snapshotting to work properly
    
    Additionally libvirtd restart processing would fail to find the domain
    since the translation processing code is run after domain xml processing,
    so handle the the case where the authdef could have an empty secrettype
    field when processing the auth and additionally ignore performing the
    actual and expected auth secret type checks for a DISK_VOLUME since that
    data will be reassembled later during translation processing of the
    running domain.

$ git describe 1feaccf0001794e167b0374abf75d97024414e27
v1.2.16-148-g1feaccf
$

Comment 5 Pei Zhang 2015-08-20 09:32:38 UTC
Verify version:
libvirt-1.2.17-5.el7.x86_64
qemu-kvm-rhev-2.3.0-18.el7.x86_64

steps:
1.prepare a iscsi server with secret conf.
2.on iscsi client create iscsi pool with secret info.
3.check pool and volume info.
# virsh pool-dumpxml iscsi-pool
<pool type='iscsi'>
  <name>iscsi-pool</name>
  <uuid>73a72456-e7ef-460a-99f7-926a6f20a74c</uuid>
  <capacity unit='bytes'>32212254720</capacity>
  <allocation unit='bytes'>32212254720</allocation>
  <available unit='bytes'>0</available>
  <source>
    <host name='$IP' port='3260'/>
    <device path='iqn.2008-09.5.165.3ffe.104:server.target3'/>
    <auth type='chap' username='redhat'>
      <secret usage='libvirtiscsi'/>
    </auth>
  </source>
  <target>
    <path>/dev/disk/by-path</path>
  </target>
</pool>

# virsh vol-list iscsi-pool
 Name                 Path                                    
------------------------------------------------------------------------------
 unit:0:0:1           /dev/disk/by-path/ip-$IP:3260-iscsi-iqn.2008-09.5.165.3ffe.104:server.target3-lun-1

4.define and start a guest like following
# virsh dumpxml r708 --inactive |grep disk -A 9
 <disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <source pool='iscsi-pool' volume='unit:0:0:1' mode='direct'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

5.do save and restore
# virsh save r708 r708.save

Domain r708 saved to r708.save

# virsh restore r708.save
Domain restored from r708.save

# virsh dumpxml r708|grep disk -A 9
    <disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <auth type='chap' username='redhat'>
        <secret type='iscsi' usage='libvirtiscsi'/>
      </auth>
      <source pool='iscsi-pool' volume='unit:0:0:1' mode='direct'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

6.do external snapshot
# virsh snapshot-create-as r708 --disk-only s1 --diskspec sda,file=/tmp/s1
Domain snapshot s1 created
# virsh snapshot-info r708 s1
Name:           s1
Domain:         r708
Current:        yes
State:          disk-snapshot
Location:       external
Parent:         -
Children:       0
Descendants:    0
Metadata:       yes

7. restart libvirtd then guest should still running.
# service libvirtd restart 
Redirecting to /bin/systemctl restart  libvirtd.service

# virsh list 
 Id    Name                           State
----------------------------------------------------
 6     r708                           running

Now, it works well.

And John, a small issue need your help. Thanks in advance.
As you said in patch :
"For the domain disk, there is no <auth> 'type' field, while there is a
<secret> 'type' field." 
I found that "<auth type='chap' ...>" will be added into domain xml automatically. details as step 4 and step 5. This xml will validate fail via virt-xml-validate .
I was wondering if this issue also should be modified. 
Thanks.

Comment 6 John Ferlan 2015-08-25 15:49:45 UTC
(In reply to Pei Zhang from comment #5)

> 
> And John, a small issue need your help. Thanks in advance.
> As you said in patch :
> "For the domain disk, there is no <auth> 'type' field, while there is a
> <secret> 'type' field." 
> I found that "<auth type='chap' ...>" will be added into domain xml
> automatically. details as step 4 and step 5. This xml will validate fail via
> virt-xml-validate .
> I was wondering if this issue also should be modified. 
> Thanks.

Hmm... yeah that shows up I guess. Beyond the XML validation failing virt-xml-validate it's no big deal.  The reason it's there has more to do with common code shared between pool and disk as well as how the disk data is filled in from the pool data.

Feel free to generate a RHEL7.3 bug to track a fix.

Comment 7 Pei Zhang 2015-08-26 03:37:12 UTC
Thanks for your info John.
File a new bug on rhel7.3 to track <auth> type issue.
https://bugzilla.redhat.com/show_bug.cgi?id=1256999

and according to comment 5, move this bug to verified.

Comment 9 errata-xmlrpc 2015-11-19 06:19:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html


Note You need to log in before you can comment on or make changes to this bug.