Hide Forgot
Description of problem: VM is running. I'd like to live attach a disk which has a luks encrypted backing file. However it seems that Libvirt can't support this scenario, and just print out the error message generated from Qemu. # virsh attach-device i-oy9651gi attach-sn1.xml error: Failed to attach device from attach-sn1.xml error: operation failed: Could not open backing file: No secret with id 'sec0' # qemu-img info sn1.img image: sn1.img file format: qcow2 virtual size: 1.0G (1073741824 bytes) disk size: 196K cluster_size: 65536 backing file: json:{"encrypt.key-secret": "sec0", "driver": "qcow2", "file": {"driver": "file", "filename": "/base-e.img"}} Format specific information: compat: 1.1 lazy refcounts: false Version-Release number of selected component (if applicable): # cat attach-sn1.xml <disk type='file' device='disk'> <driver cache="writeback" discard="ignore" io="threads" name="qemu" type="qcow2"/> <source file='/sn1.img'/> <backingStore type='file' index='1'> <format type='qcow2'/> <source file='/base-e.img'/> <encryption format='luks'> <secret type='passphrase' uuid='0a81f5b2-8403-0001-c8d6-21bbb2f80d6f'/> </encryption> <backingStore /> </backingStore> <target dev='vdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </disk> Version: QEMU emulator version 3.1.0 libvirt 5.7.0 How reproducible: Constantly reproducible Steps to Reproduce: 0. VM is running, domain name is i-oy9651gi. 1. Create the luks encrypted base file. qemu-img create --object secret,id=sec0,data=123456 -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 base-e 1G 2. Create the plain file backed by base-e.img qemu-img create -f qcow2 -b base-e.img sn1.img 1G 3. Create and edit attach-sn1.xml as above in the description section. 4. Create and edit the Libvirt secret object, uuid is 0a81f5b2-8403-0001-c8d6-21bbb2f80d6f, data value is 123456. 5. Attach the disk, virsh attach-device i-oy9651gi attach-sn1.xml Actual results: error: Failed to attach device from attach-sn1.xml error: operation failed: Could not open backing file: No secret with id 'sec0' Expected results: Disk attached successfully Additional info: It seems that Libvirt will ask Qemu to create a Qom secret object to hold the secret value, and pass the encrypt-key to Qemu which points to the secret object. This is working fine when there is no backing file. However in the case of backing file, I don't see any code in Libvirt to create the secret object for backing file, and t he alias name seems to be always virtio-diskXXX-luks-secret0 for a.particular disk.
When you use following disk xml (no <encryption> part), you may get same result. <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/luks.snap1'/> <backingStore type='file'> <format type='raw'/> <source file='/var/lib/libvirt/images/luks.img'/> </backingStore> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </disk> Encryption part is not supported inside <backingStore>, as libvirt-rhel/docs/schemas/domaincommon.rng shows: 1559 | <define name="diskBackingStore"> 1560 | <element name="backingStore"> 1561 | | <optional> 1562 | | <attribute name="index"> 1563 | | | <ref name="positiveInteger"/> 1564 | | </attribute> 1565 | | </optional> 1566 | | <interleave> 1567 | | <ref name="diskSource"/> 1568 | | <ref name="diskBackingChain"/> 1569 | | <ref name="diskFormat"/> 1570 | | </interleave> 1571 | </element> 1572 | </define> This is definitely an issue, and all of this could be tracked in "Bug 1375855 - LUKS encryption not supported for snapshots' backing files." So I'm closing this as a 'duplicate' for now. *** This bug has been marked as a duplicate of bug 1375855 ***
Hi yisun, I encountered another issue recently which is very similar to this one and could be the same cause. I'd like to get some advise from you how to support this case in libvirt 5.7.0, and I noticed that in the bug 1375855, Peter Krempa mentioned that using -blockdev should be able to support encrypted backing file. Could you please paste some scripts or commands I can have a try? Thanks in advance. Below is the issue description I recently got when I use the virsh blockcommit against the encrypted backing file. root@host:~/peterluo/211# virsh blockcommit --domain i-tcbss9e8 vdd --top /root/peterluo/211/1221-snaptest1 --shallow --active --pivot --wait error: invalid argument: could not find image 'json:{"encrypt.format": "luks", "encrypt.key-secret": "virtio-disk3-luks-secret0", "driver": "qcow2", "file": {"driver": "file", "filename": "/root/peterluo/211/1211-encry1.img"}}' beneath '/root/peterluo/211/1221-snaptest1' in chain for '/root/peterluo/211/1221-snaptest1' root@host:~/peterluo/211# qemu-img info /root/peterluo/211/1221-snaptest1 image: /root/peterluo/211/1221-snaptest1 file format: qcow2 virtual size: 10G (10737418240 bytes) disk size: 196K cluster_size: 65536 backing file: json:{"encrypt.format": "luks", "encrypt.key-secret": "virtio-disk3-luks-secret0", "driver": "qcow2", "file": {"driver": "file", "filename": "/root/peterluo/211/1211-encry1.img"}} backing file format: qcow2 Format specific information: compat: 1.1 lazy refcounts: false
(In reply to Peter Luo from comment #2) > Hi yisun, > > I encountered another issue recently which is very similar to this one and > could be the same cause. > I'd like to get some advise from you how to support this case in libvirt > 5.7.0, and I noticed that in the bug 1375855, Peter Krempa mentioned that > using -blockdev should be able to support encrypted backing file. Could you > please paste some scripts or commands I can have a try? Thanks in advance. > > > Below is the issue description I recently got when I use the virsh > blockcommit against the encrypted backing file. > > root@host:~/peterluo/211# virsh blockcommit --domain i-tcbss9e8 vdd --top > /root/peterluo/211/1221-snaptest1 --shallow --active --pivot --wait > error: invalid argument: could not find image 'json:{"encrypt.format": > "luks", "encrypt.key-secret": "virtio-disk3-luks-secret0", "driver": > "qcow2", "file": {"driver": "file", "filename": > "/root/peterluo/211/1211-encry1.img"}}' beneath > '/root/peterluo/211/1221-snaptest1' in chain for > '/root/peterluo/211/1221-snaptest1' > > root@host:~/peterluo/211# qemu-img info /root/peterluo/211/1221-snaptest1 > image: /root/peterluo/211/1221-snaptest1 > file format: qcow2 > virtual size: 10G (10737418240 bytes) > disk size: 196K > cluster_size: 65536 > backing file: json:{"encrypt.format": "luks", "encrypt.key-secret": > "virtio-disk3-luks-secret0", "driver": "qcow2", "file": {"driver": "file", > "filename": "/root/peterluo/211/1211-encry1.img"}} > backing file format: qcow2 > Format specific information: > compat: 1.1 > lazy refcounts: false For "using -blockdev", it means libvirt will invoke qemu-kvm command line with "-blockdev" to indicate disks, but not "-drive" anymore. With libvirt 5.7.0, we can edit vm's xml with following info: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> … <qemu:capabilities> <qemu:add capability='blockdev'/> <qemu:del capability='drive'/> </qemu:capabilities> </domain> With above setting, when you start vm, the "-blockdev" will be used. As well, if use libvirt-5.10.0 or later versions, you don't need above setting and libvirt will use -blockdev by default. You can have a try with this, but bz1375855 has not been tested, and -blockdev enablement actually not been fully tested yet, the function maybe not stable...
I tried with the below xml configuration today, but I failed to create the external snapshot when using the blockdev instead of drive. Does the virsh snapshot-create-as command need any change accordingly when using the blockdev? <qemu:capabilities> <qemu:add capability='blockdev'/> <qemu:del capability='drive'/> </qemu:capabilities> root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh domblklist vm1 Target Source ------------------------------------------------------------------------------- vda /root/peterluo/bionic3x64 vdb /root/peterluo/211/i-oy9651gi/virshtestforbacking/case20/base-e.img root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh snapshot-create-as vm1 snap --atomic --no-metadata --disk-only --diskspec vdb,snapshot=external,file=/root/peterluo/211/i-oy9651gi/virshtestforbacking/case20/snap1 --diskspec vda,snapshot=no error: internal error: unable to execute QEMU command 'transaction': Cannot find device=drive-virtio-disk1 nor node_name= The VM1 is created with the blockdev option enabled. root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# ps -elf | grep qemu-system ... -blockdev {"driver":"file","filename":"/root/peterluo/bionic3x64","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"discard":"ignore","cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":null} -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=libvirt-2-format,id=virtio-disk0,bootindex=1,write-cache=on ... -object secret,id=libvirt-1-format-luks-secret0,data=X/DMmVc5Nw0hdEFFnapp4g==,keyid=masterKey0,iv=kmjhuyfojBgqtZxT8XdXHg==,format=base64 -blockdev {"driver":"file","filename":"/root/peterluo/211/i-oy9651gi/virshtestforbacking/case20/base-e.img","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"discard":"ignore","cache":{"direct":false,"no-flush":false},"driver":"qcow2","encrypt":{"format":"luks","key-secret":"libvirt-1-format-luks-secret0"},"file":"libvirt-1-storage","backing":null} -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk1,write-cache=on ... root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh version Compiled against library: libvirt 5.7.0 Using library: libvirt 5.7.0 Using API: QEMU 5.7.0 root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# qemu-system-x86_64 --version QEMU emulator version 3.1.0
(In reply to Peter Luo from comment #4) > I tried with the below xml configuration today, but I failed to create the > external snapshot when using the blockdev instead of drive. Does the virsh > snapshot-create-as command need any change accordingly when using the > blockdev? > > <qemu:capabilities> > <qemu:add capability='blockdev'/> > <qemu:del capability='drive'/> > </qemu:capabilities> > > root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh > domblklist vm1 > Target Source > ----------------------------------------------------------------------------- > -- > vda /root/peterluo/bionic3x64 > vdb /root/peterluo/211/i-oy9651gi/virshtestforbacking/case20/base-e.img Do not user /root as image dir, qemu process will be denied by SELinux. Use /var/lib/libvirt/images instead. > > root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh > snapshot-create-as vm1 snap --atomic --no-metadata --disk-only --diskspec > vdb,snapshot=external,file=/root/peterluo/211/i-oy9651gi/virshtestforbacking/ > case20/snap1 --diskspec vda,snapshot=no > error: internal error: unable to execute QEMU command 'transaction': Cannot > find device=drive-virtio-disk1 nor node_name= This issue should have been fixed in libvirt 5.10. Please use libvirt 5.10 to test it. Since blockdev is defaultly enabled in libvirt 5.10, <qemu:capabilities>...</qemu:capabilities> is not required. And it is better to use qemu 4.2 release, because you may encounter some snapshot releated bug like https://bugzilla.redhat.com/show_bug.cgi?id=1763937 > > > The VM1 is created with the blockdev option enabled. > > root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# ps -elf | > grep qemu-system > ... > -blockdev > {"driver":"file","filename":"/root/peterluo/bionic3x64","aio":"threads", > "node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false}, > "auto-read-only":true,"discard":"unmap"} > -blockdev > {"node-name":"libvirt-2-format","read-only":false,"discard":"ignore","cache": > {"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2- > storage","backing":null} -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=libvirt-2-format,id=virtio- > disk0,bootindex=1,write-cache=on > ... > -object > secret,id=libvirt-1-format-luks-secret0,data=X/DMmVc5Nw0hdEFFnapp4g==, > keyid=masterKey0,iv=kmjhuyfojBgqtZxT8XdXHg==,format=base64 > -blockdev > {"driver":"file","filename":"/root/peterluo/211/i-oy9651gi/ > virshtestforbacking/case20/base-e.img","aio":"threads","node-name":"libvirt- > 1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true, > "discard":"unmap"} > -blockdev > {"node-name":"libvirt-1-format","read-only":false,"discard":"ignore","cache": > {"direct":false,"no-flush":false},"driver":"qcow2","encrypt":{"format": > "luks","key-secret":"libvirt-1-format-luks-secret0"},"file":"libvirt-1- > storage","backing":null} -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio- > disk1,write-cache=on > ... > > > root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh version > Compiled against library: libvirt 5.7.0 > Using library: libvirt 5.7.0 > Using API: QEMU 5.7.0 > > root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# > qemu-system-x86_64 --version > QEMU emulator version 3.1.0
(In reply to Han Han from comment #5) > (In reply to Peter Luo from comment #4) > > I tried with the below xml configuration today, but I failed to create the > > external snapshot when using the blockdev instead of drive. Does the virsh > > snapshot-create-as command need any change accordingly when using the > > blockdev? > > > > <qemu:capabilities> > > <qemu:add capability='blockdev'/> > > <qemu:del capability='drive'/> > > </qemu:capabilities> This configuration is really meant only for testing purposes. You tried to enable an unfinished (at the 5.7 time) feature. > > > > root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh > > domblklist vm1 > > Target Source > > ----------------------------------------------------------------------------- > > -- > > vda /root/peterluo/bionic3x64 > > vdb /root/peterluo/211/i-oy9651gi/virshtestforbacking/case20/base-e.img > Do not user /root as image dir, qemu process will be denied by SELinux. Use > /var/lib/libvirt/images instead. > > > > root@host:~/peterluo/211/i-oy9651gi/virshtestforbacking/case20# virsh > > snapshot-create-as vm1 snap --atomic --no-metadata --disk-only --diskspec > > vdb,snapshot=external,file=/root/peterluo/211/i-oy9651gi/virshtestforbacking/ > > case20/snap1 --diskspec vda,snapshot=no > > error: internal error: unable to execute QEMU command 'transaction': Cannot > > find device=drive-virtio-disk1 nor node_name= > > This issue should have been fixed in libvirt 5.10. > Please use libvirt 5.10 to test it. > Since blockdev is defaultly enabled in libvirt 5.10, > <qemu:capabilities>...</qemu:capabilities> is not required. > > And it is better to use qemu 4.2 release, because you may encounter some No. You MUST use qemu 4.2. Blockdev is only usable with qemu-4.2 and will not be enabled in any lower release. > snapshot releated bug like > https://bugzilla.redhat.com/show_bug.cgi?id=1763937