Bug 1773399 - virsh failed to attach disk which has a luks encrypted backing file
Summary: virsh failed to attach disk which has a luks encrypted backing file
Keywords:
Status: CLOSED DUPLICATE of bug 1375855
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.0
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: rc
: 8.0
Assignee: Libvirt Maintainers
QA Contact: Han Han
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-18 03:37 UTC by Peter Luo
Modified: 2019-12-25 09:05 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-18 08:40:10 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Peter Luo 2019-11-18 03:37:47 UTC
Description of problem:

VM is running. I'd like to live attach a disk which has a luks encrypted backing file.

However it seems that Libvirt can't support this scenario, and just print out the error message generated from Qemu.

# virsh attach-device i-oy9651gi attach-sn1.xml
error: Failed to attach device from attach-sn1.xml
error: operation failed: Could not open backing file: No secret with id 'sec0'

# qemu-img info sn1.img
image: sn1.img
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 196K
cluster_size: 65536
backing file: json:{"encrypt.key-secret": "sec0", "driver": "qcow2", "file": {"driver": "file", "filename": "/base-e.img"}}
Format specific information:
    compat: 1.1
    lazy refcounts: false
Version-Release number of selected component (if applicable):

# cat attach-sn1.xml
<disk type='file' device='disk'>
      <driver cache="writeback" discard="ignore" io="threads" name="qemu" type="qcow2"/>
      <source file='/sn1.img'/>
      <backingStore type='file' index='1'>
	  <format type='qcow2'/>
            <source file='/base-e.img'/>
            <encryption format='luks'>
              <secret type='passphrase' uuid='0a81f5b2-8403-0001-c8d6-21bbb2f80d6f'/>
            </encryption>
            <backingStore />
         </backingStore>
      <target dev='vdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
</disk>

Version:
QEMU emulator version 3.1.0
libvirt 5.7.0

How reproducible: Constantly reproducible

Steps to Reproduce:
0. VM is running, domain name is i-oy9651gi.
1. Create the luks encrypted base file. 
qemu-img create --object secret,id=sec0,data=123456 -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 base-e 1G
2. Create the plain file backed by base-e.img 
qemu-img create  -f qcow2 -b base-e.img sn1.img 1G
3. Create and edit attach-sn1.xml as above in the description section.
4. Create and edit the Libvirt secret object, uuid is 0a81f5b2-8403-0001-c8d6-21bbb2f80d6f, data value is 123456.
5. Attach the disk, virsh attach-device i-oy9651gi attach-sn1.xml

Actual results:
error: Failed to attach device from attach-sn1.xml
error: operation failed: Could not open backing file: No secret with id 'sec0'

Expected results:
Disk attached successfully

Additional info:
It seems that Libvirt will ask Qemu to create a Qom secret object to hold the secret value, and pass the encrypt-key to Qemu which points to the secret object. This is working fine when there is no backing file. However in the case of backing file, I don't see any code in Libvirt to create the secret object for  backing file, and t he alias name seems to be always virtio-diskXXX-luks-secret0 for a.particular disk.

Comment 1 yisun 2019-11-18 08:40:10 UTC
When you use following disk xml (no <encryption> part), you may get same result.
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/luks.snap1'/>
      <backingStore type='file'>
        <format type='raw'/>
        <source file='/var/lib/libvirt/images/luks.img'/>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

Encryption part is not supported inside <backingStore>, as libvirt-rhel/docs/schemas/domaincommon.rng shows:
1559 | <define name="diskBackingStore">                                              
1560 |   <element name="backingStore">                                               
1561 |   | <optional>                                                                
1562 |   |   <attribute name="index">                                                
1563 |   |   | <ref name="positiveInteger"/>                                         
1564 |   |   </attribute>                                                            
1565 |   | </optional>                                                               
1566 |   | <interleave>                                                              
1567 |   |   <ref name="diskSource"/>                                                
1568 |   |   <ref name="diskBackingChain"/>                                          
1569 |   |   <ref name="diskFormat"/>                                                
1570 |   | </interleave>                                                             
1571 |   </element>                                                                  
1572 | </define> 

This is definitely an issue, and all of this could be tracked in "Bug 1375855 - LUKS encryption not supported for snapshots' backing files."
So I'm closing this as a 'duplicate' for now.

*** This bug has been marked as a duplicate of bug 1375855 ***

Comment 2 Peter Luo 2019-12-21 15:55:48 UTC
Hi yisun,

I encountered another issue recently which is very similar to this one and could be the same cause. 
I'd like to get some advise from you how to support this case in libvirt 5.7.0, and I noticed that in the bug 1375855,  Peter Krempa  mentioned that using -blockdev should be able to support encrypted backing file. Could you please paste some scripts or commands I can have a try? Thanks in advance.


Below is the issue description I recently got when I use the virsh blockcommit against the encrypted backing file.

root@host:~/peterluo/211# virsh blockcommit --domain i-tcbss9e8 vdd --top /root/peterluo/211/1221-snaptest1 --shallow --active --pivot --wait
error: invalid argument: could not find image 'json:{"encrypt.format": "luks", "encrypt.key-secret": "virtio-disk3-luks-secret0", "driver": "qcow2", "file": {"driver": "file", "filename": "/root/peterluo/211/1211-encry1.img"}}' beneath '/root/peterluo/211/1221-snaptest1' in chain for '/root/peterluo/211/1221-snaptest1'

root@host:~/peterluo/211# qemu-img info /root/peterluo/211/1221-snaptest1
image: /root/peterluo/211/1221-snaptest1
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 196K
cluster_size: 65536
backing file: json:{"encrypt.format": "luks", "encrypt.key-secret": "virtio-disk3-luks-secret0", "driver": "qcow2", "file": {"driver": "file", "filename": "/root/peterluo/211/1211-encry1.img"}}
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false

Comment 3 yisun 2019-12-22 14:30:50 UTC
(In reply to Peter Luo from comment #2)
> Hi yisun,
> 
> I encountered another issue recently which is very similar to this one and
> could be the same cause. 
> I'd like to get some advise from you how to support this case in libvirt
> 5.7.0, and I noticed that in the bug 1375855,  Peter Krempa  mentioned that
> using -blockdev should be able to support encrypted backing file. Could you
> please paste some scripts or commands I can have a try? Thanks in advance.
> 
> 
> Below is the issue description I recently got when I use the virsh
> blockcommit against the encrypted backing file.
> 
> root@host:~/peterluo/211# virsh blockcommit --domain i-tcbss9e8 vdd --top
> /root/peterluo/211/1221-snaptest1 --shallow --active --pivot --wait
> error: invalid argument: could not find image 'json:{"encrypt.format":
> "luks", "encrypt.key-secret": "virtio-disk3-luks-secret0", "driver":
> "qcow2", "file": {"driver": "file", "filename":
> "/root/peterluo/211/1211-encry1.img"}}' beneath
> '/root/peterluo/211/1221-snaptest1' in chain for
> '/root/peterluo/211/1221-snaptest1'
> 
> root@host:~/peterluo/211# qemu-img info /root/peterluo/211/1221-snaptest1
> image: /root/peterluo/211/1221-snaptest1
> file format: qcow2
> virtual size: 10G (10737418240 bytes)
> disk size: 196K
> cluster_size: 65536
> backing file: json:{"encrypt.format": "luks", "encrypt.key-secret":
> "virtio-disk3-luks-secret0", "driver": "qcow2", "file": {"driver": "file",
> "filename": "/root/peterluo/211/1211-encry1.img"}}
> backing file format: qcow2
> Format specific information:
>     compat: 1.1
>     lazy refcounts: false

For "using -blockdev", it means libvirt will invoke qemu-kvm command line with "-blockdev" to indicate disks, but not "-drive" anymore. With libvirt 5.7.0, we can edit vm's xml with following info:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
…
  <qemu:capabilities>
    <qemu:add capability='blockdev'/>
    <qemu:del capability='drive'/>
  </qemu:capabilities>
</domain>

With above setting, when you start vm, the "-blockdev" will be used. As well, if use libvirt-5.10.0 or later versions, you don't need above setting and libvirt will use -blockdev by default. 
You can have a try with this, but bz1375855 has not been tested, and -blockdev enablement actually not been fully tested yet, the function maybe not stable...

Comment 4 Peter Luo 2019-12-25 07:19:05 UTC
I tried with the below xml configuration today, but I failed to create the external snapshot when using the blockdev instead of drive. Does the virsh snapshot-create-as command need any change accordingly when using the blockdev?

<qemu:capabilities>
    <qemu:add capability='blockdev'/>
    <qemu:del capability='drive'/>
</qemu:capabilities>

root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh domblklist vm1
 Target   Source
-------------------------------------------------------------------------------
 vda      /root/peterluo/bionic3x64
 vdb      /root/peterluo/211/i-oy9651gi/virshtestforbacking/case20/base-e.img

root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh snapshot-create-as vm1  snap --atomic --no-metadata --disk-only --diskspec vdb,snapshot=external,file=/root/peterluo/211/i-oy9651gi/virshtestforbacking/case20/snap1 --diskspec vda,snapshot=no
error: internal error: unable to execute QEMU command 'transaction': Cannot find device=drive-virtio-disk1 nor node_name=


The VM1 is created with the blockdev option enabled.

root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# ps -elf | grep qemu-system
...
-blockdev {"driver":"file","filename":"/root/peterluo/bionic3x64","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} 
-blockdev {"node-name":"libvirt-2-format","read-only":false,"discard":"ignore","cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":null} -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=libvirt-2-format,id=virtio-disk0,bootindex=1,write-cache=on 
...
-object secret,id=libvirt-1-format-luks-secret0,data=X/DMmVc5Nw0hdEFFnapp4g==,keyid=masterKey0,iv=kmjhuyfojBgqtZxT8XdXHg==,format=base64 
-blockdev {"driver":"file","filename":"/root/peterluo/211/i-oy9651gi/virshtestforbacking/case20/base-e.img","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} 
-blockdev {"node-name":"libvirt-1-format","read-only":false,"discard":"ignore","cache":{"direct":false,"no-flush":false},"driver":"qcow2","encrypt":{"format":"luks","key-secret":"libvirt-1-format-luks-secret0"},"file":"libvirt-1-storage","backing":null} -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk1,write-cache=on 
...


root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh version
Compiled against library: libvirt 5.7.0
Using library: libvirt 5.7.0
Using API: QEMU 5.7.0

root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# qemu-system-x86_64  --version
QEMU emulator version 3.1.0

Comment 5 Han Han 2019-12-25 08:00:29 UTC
(In reply to Peter Luo from comment #4)
> I tried with the below xml configuration today, but I failed to create the
> external snapshot when using the blockdev instead of drive. Does the virsh
> snapshot-create-as command need any change accordingly when using the
> blockdev?
> 
> <qemu:capabilities>
>     <qemu:add capability='blockdev'/>
>     <qemu:del capability='drive'/>
> </qemu:capabilities>
> 
> root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh
> domblklist vm1
>  Target   Source
> -----------------------------------------------------------------------------
> --
>  vda      /root/peterluo/bionic3x64
>  vdb      /root/peterluo/211/i-oy9651gi/virshtestforbacking/case20/base-e.img
Do not user /root as image dir, qemu process will be denied by SELinux. Use /var/lib/libvirt/images instead.
> 
> root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh
> snapshot-create-as vm1  snap --atomic --no-metadata --disk-only --diskspec
> vdb,snapshot=external,file=/root/peterluo/211/i-oy9651gi/virshtestforbacking/
> case20/snap1 --diskspec vda,snapshot=no
> error: internal error: unable to execute QEMU command 'transaction': Cannot
> find device=drive-virtio-disk1 nor node_name=

This issue should have been fixed in libvirt 5.10.
Please use libvirt 5.10 to test it.
Since blockdev is defaultly enabled in libvirt 5.10, <qemu:capabilities>...</qemu:capabilities> is not required.

And it is better to use qemu 4.2 release, because you may encounter some snapshot releated bug like https://bugzilla.redhat.com/show_bug.cgi?id=1763937

> 
> 
> The VM1 is created with the blockdev option enabled.
> 
> root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# ps -elf |
> grep qemu-system
> ...
> -blockdev
> {"driver":"file","filename":"/root/peterluo/bionic3x64","aio":"threads",
> "node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},
> "auto-read-only":true,"discard":"unmap"} 
> -blockdev
> {"node-name":"libvirt-2-format","read-only":false,"discard":"ignore","cache":
> {"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2-
> storage","backing":null} -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=libvirt-2-format,id=virtio-
> disk0,bootindex=1,write-cache=on 
> ...
> -object
> secret,id=libvirt-1-format-luks-secret0,data=X/DMmVc5Nw0hdEFFnapp4g==,
> keyid=masterKey0,iv=kmjhuyfojBgqtZxT8XdXHg==,format=base64 
> -blockdev
> {"driver":"file","filename":"/root/peterluo/211/i-oy9651gi/
> virshtestforbacking/case20/base-e.img","aio":"threads","node-name":"libvirt-
> 1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,
> "discard":"unmap"} 
> -blockdev
> {"node-name":"libvirt-1-format","read-only":false,"discard":"ignore","cache":
> {"direct":false,"no-flush":false},"driver":"qcow2","encrypt":{"format":
> "luks","key-secret":"libvirt-1-format-luks-secret0"},"file":"libvirt-1-
> storage","backing":null} -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-
> disk1,write-cache=on 
> ...
> 
> 
> root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh version
> Compiled against library: libvirt 5.7.0
> Using library: libvirt 5.7.0
> Using API: QEMU 5.7.0
> 
> root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20#
> qemu-system-x86_64  --version
> QEMU emulator version 3.1.0

Comment 6 Peter Krempa 2019-12-25 09:05:18 UTC
(In reply to Han Han from comment #5)
> (In reply to Peter Luo from comment #4)
> > I tried with the below xml configuration today, but I failed to create the
> > external snapshot when using the blockdev instead of drive. Does the virsh
> > snapshot-create-as command need any change accordingly when using the
> > blockdev?
> > 
> > <qemu:capabilities>
> >     <qemu:add capability='blockdev'/>
> >     <qemu:del capability='drive'/>
> > </qemu:capabilities>

This configuration is really meant only for testing purposes. You tried to enable an unfinished (at the 5.7 time) feature.

> > 
> > root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh
> > domblklist vm1
> >  Target   Source
> > -----------------------------------------------------------------------------
> > --
> >  vda      /root/peterluo/bionic3x64
> >  vdb      /root/peterluo/211/i-oy9651gi/virshtestforbacking/case20/base-e.img
> Do not user /root as image dir, qemu process will be denied by SELinux. Use
> /var/lib/libvirt/images instead.
> > 
> > root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh
> > snapshot-create-as vm1  snap --atomic --no-metadata --disk-only --diskspec
> > vdb,snapshot=external,file=/root/peterluo/211/i-oy9651gi/virshtestforbacking/
> > case20/snap1 --diskspec vda,snapshot=no
> > error: internal error: unable to execute QEMU command 'transaction': Cannot
> > find device=drive-virtio-disk1 nor node_name=
> 
> This issue should have been fixed in libvirt 5.10.
> Please use libvirt 5.10 to test it.
> Since blockdev is defaultly enabled in libvirt 5.10,
> <qemu:capabilities>...</qemu:capabilities> is not required.
> 
> And it is better to use qemu 4.2 release, because you may encounter some

No. You MUST use qemu 4.2. Blockdev is only usable with qemu-4.2 and will not be enabled in any lower release.
> snapshot releated bug like
> https://bugzilla.redhat.com/show_bug.cgi?id=1763937


Note You need to log in before you can comment on or make changes to this bug.