Bug 1872567

Created attachment 1712627 [details]
libvirtd-debug.log

This really isn't a libvirt problem. If the destination is a block device and it's too small it's a user error and nothing libvirt can do about. Users are advised that block device must be prepared properly and it's suggested users use an image rather than a block device anyways. (step 6 -> notabug)

While a raw block device doesn't encode the size but I think that a full backup to a larger block device (step 7) should not fail on qemu's side regardless of the format. Obviously the result will be bigger. Moving to qemu for investigation.

(In reply to yisun from comment #0)
> Description:
> [incremental_backup][luks] Cannot do push backup to a block device with
> type=raw and luks encryption enabled
> 
> 
> Versions:
> libvirt-6.6.0-2.module+el8.3.0+7567+dc41c0a9.x86_64
> qemu-kvm-5.1.0-3.module+el8.3.0+7708+740a1315.x86_64
> 
> How reproducible:
> 100%
> 
> Steps:
> 1. prepare a 100M block device /dev/sdb and clear it
> [root@dell-per740xd-11 images]# lsblk | grep sdb
> sdb                                8:16   0    100M  0 disk 
> 
> 
> [root@dell-per740xd-11 images]# dd if=/dev/zero of=/dev/sdb 
> dd: writing to '/dev/sdb': No space left on device
> 204801+0 records in
> 204800+0 records out
> 104857600 bytes (105 MB, 100 MiB) copied, 1.40947 s, 74.4 MB/s
> 
> 2. having a vm with vdb pointing to a 100M qcow2 file
> [root@dell-per740xd-11 images]# virsh domblklist vm1
>  Target   Source
> --------------------------------------------------------
> ...
>  vdb      /var/lib/libvirt/images/vdb.qcow2
> 
> [root@dell-per740xd-11 images]# qemu-img info
> /var/lib/libvirt/images/vdb.qcow2 -U
> image: /var/lib/libvirt/images/vdb.qcow2
> file format: qcow2
> virtual size: 100 MiB (104857600 bytes)
> disk size: 1 MiB
> 
> 3. clear libvirtd log
> [root@dell-per740xd-11 images]# systemctl restart libvirtd
> [root@dell-per740xd-11 images]# echo "" > /var/log/libvirtd-debug.log 
> 
> 4. prepare a libvirt secret
> [root@dell-per740xd-11 images]# virsh secret-dumpxml
> bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb
> <secret ephemeral='no' private='no'>
>   <uuid>bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb</uuid>
>   <description>LUKS passphrase</description>
>   <usage type='volume'>
>     <volume>/luks/img</volume>
>   </usage>
> </secret>
> 
> [root@dell-per740xd-11 images]# virsh secret-get-value
> bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb
> cmVkaGF0
> 
> 
> 5. prepare a backup to /dev/sdb xml with type=raw and luks enabled
> [root@dell-per740xd-11 images]# cat backup_push_full_blk.xml 
> <domainbackup mode="push">
>   <disks>
>     <disk name='vda' backup='no'/>
>     <disk name='vdb' type='block'>
>       <driver type='raw'/>
>       <target dev='/dev/sdb'>
>         <encryption format='luks'>
>           <secret type='passphrase'
> uuid='bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb'/>
>         </encryption>
>       </target>
>     </disk>
>   </disks>
> </domainbackup>
> 
> 6. start the backup job, failed with error
> [root@dell-per740xd-11 images]# virsh backup-begin vm1
> backup_push_full_blk.xml 
> error: operation failed: failed to format image: 'Cannot grow device files'
> 
> (attached the libvirtd log now)
> 
> 7. if we use a smaller vdb image (/var/lib/libvirt/images/vdb.qcow2 is 80M),
> hit another expected error.
> [root@dell-per740xd-11 images]# qemu-img create -f qcow2
> /var/lib/libvirt/images/vdb.qcow2 80M
> Formatting '/var/lib/libvirt/images/vdb.qcow2', fmt=qcow2 cluster_size=65536
> compression_type=zlib size=83886080 lazy_refcounts=off refcount_bits=16
> [root@dell-per740xd-11 images]# virsh start vm1
> Domain vm1 started
> 
> [root@dell-per740xd-11 images]# virsh backup-begin vm1
> backup_push_full_blk.xml 
> error: internal error: unable to execute QEMU command 'transaction': Source
> and target image have different sizes
> 

It's a behavior change since qemu-kvm-4.2.0-24.module+el8.2.1+6959+9b840e7c, please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1778593#c19
> 
> 
> Expected result:
> backup job should be successful in step 6  or step 7
> 
> Actual result:
> Cannot do push backup
> 
> Additional info:
> Seems luks metadata will take some space and causing the problem, if disable
> luks, nothing wrong
> 
> [root@dell-per740xd-11 images]# cat bkup.xml 
> <domainbackup mode="push">
>   <disks>
>     <disk name='vda' backup='no'/>
>     <disk name='vdb' type='block'>
>       <driver type='raw'/>
>       <target dev='/dev/sdb'/>
>     </disk>
>   </disks>
> </domainbackup>
> [root@dell-per740xd-11 images]# virsh backup-begin vm1 bkup.xml 
> Backup started

(In reply to yisun from comment #0)
> Description:
> [incremental_backup][luks] Cannot do push backup to a block device with
> type=raw and luks encryption enabled
> 
> 
> Versions:
> libvirt-6.6.0-2.module+el8.3.0+7567+dc41c0a9.x86_64
> qemu-kvm-5.1.0-3.module+el8.3.0+7708+740a1315.x86_64
> 
> How reproducible:
> 100%
> 
> Steps:
> 1. prepare a 100M block device /dev/sdb and clear it
> [root@dell-per740xd-11 images]# lsblk | grep sdb
> sdb                                8:16   0    100M  0 disk 
> 
> 
> [root@dell-per740xd-11 images]# dd if=/dev/zero of=/dev/sdb 
> dd: writing to '/dev/sdb': No space left on device
> 204801+0 records in
> 204800+0 records out
> 104857600 bytes (105 MB, 100 MiB) copied, 1.40947 s, 74.4 MB/s
> 
> 2. having a vm with vdb pointing to a 100M qcow2 file
> [root@dell-per740xd-11 images]# virsh domblklist vm1
>  Target   Source
> --------------------------------------------------------
> ...
>  vdb      /var/lib/libvirt/images/vdb.qcow2
> 
> [root@dell-per740xd-11 images]# qemu-img info
> /var/lib/libvirt/images/vdb.qcow2 -U
> image: /var/lib/libvirt/images/vdb.qcow2
> file format: qcow2
> virtual size: 100 MiB (104857600 bytes)
> disk size: 1 MiB
> 
> 3. clear libvirtd log
> [root@dell-per740xd-11 images]# systemctl restart libvirtd
> [root@dell-per740xd-11 images]# echo "" > /var/log/libvirtd-debug.log 
> 
> 4. prepare a libvirt secret
> [root@dell-per740xd-11 images]# virsh secret-dumpxml
> bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb
> <secret ephemeral='no' private='no'>
>   <uuid>bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb</uuid>
>   <description>LUKS passphrase</description>
>   <usage type='volume'>
>     <volume>/luks/img</volume>
>   </usage>
> </secret>
> 
> [root@dell-per740xd-11 images]# virsh secret-get-value
> bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb
> cmVkaGF0
> 
> 
> 5. prepare a backup to /dev/sdb xml with type=raw and luks enabled
> [root@dell-per740xd-11 images]# cat backup_push_full_blk.xml 
> <domainbackup mode="push">
>   <disks>
>     <disk name='vda' backup='no'/>
>     <disk name='vdb' type='block'>
>       <driver type='raw'/>
>       <target dev='/dev/sdb'>
>         <encryption format='luks'>
>           <secret type='passphrase'
> uuid='bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb'/>
>         </encryption>
>       </target>
>     </disk>
>   </disks>
> </domainbackup>
> 
> 6. start the backup job, failed with error
> [root@dell-per740xd-11 images]# virsh backup-begin vm1
> backup_push_full_blk.xml 
> error: operation failed: failed to format image: 'Cannot grow device files'

Hi, Sunyi
 I reproduce it by following steps:
 1.Create a 100M image
   #dd if=/dev/urandom of=disk.img 100M

 2.Mount this image to /dev/loop1
   #losetup /dev/loop1 /home/disk.img

 3.Create luks image on this device
   #qemu-img create --object secret,id=sec0,data=backing -f luks /dev/loop1 -o key-secret=sec0 100M
Formatting '/dev/loop1', fmt=luks size=104857600 key-secret=sec0
qemu-img: /dev/loop1: Cannot grow device files

 I think maybe it's the root cause.

> 
> (attached the libvirtd log now)
> 
> 7. if we use a smaller vdb image (/var/lib/libvirt/images/vdb.qcow2 is 80M),
> hit another expected error.
> [root@dell-per740xd-11 images]# qemu-img create -f qcow2
> /var/lib/libvirt/images/vdb.qcow2 80M
> Formatting '/var/lib/libvirt/images/vdb.qcow2', fmt=qcow2 cluster_size=65536
> compression_type=zlib size=83886080 lazy_refcounts=off refcount_bits=16
> [root@dell-per740xd-11 images]# virsh start vm1
> Domain vm1 started
> 
> [root@dell-per740xd-11 images]# virsh backup-begin vm1
> backup_push_full_blk.xml 
> error: internal error: unable to execute QEMU command 'transaction': Source
> and target image have different sizes
> 
> 
> 
> Expected result:
> backup job should be successful in step 6  or step 7
> 
> Actual result:
> Cannot do push backup
> 
> Additional info:
> Seems luks metadata will take some space and causing the problem, if disable
> luks, nothing wrong
> 
> [root@dell-per740xd-11 images]# cat bkup.xml 
> <domainbackup mode="push">
>   <disks>
>     <disk name='vda' backup='no'/>
>     <disk name='vdb' type='block'>
>       <driver type='raw'/>
>       <target dev='/dev/sdb'/>
>     </disk>
>   </disks>
> </domainbackup>
> [root@dell-per740xd-11 images]# virsh backup-begin vm1 bkup.xml 
> Backup started

(In reply to aihua liang from comment #4)
> Hi, Sunyi
>  I reproduce it by following steps:
>  1.Create a 100M image
>    #dd if=/dev/urandom of=disk.img 100M
> 
>  2.Mount this image to /dev/loop1
>    #losetup /dev/loop1 /home/disk.img
> 
>  3.Create luks image on this device
>    #qemu-img create --object secret,id=sec0,data=backing -f luks /dev/loop1
> -o key-secret=sec0 100M
> Formatting '/dev/loop1', fmt=luks size=104857600 key-secret=sec0
> qemu-img: /dev/loop1: Cannot grow device files
> 
>  I think maybe it's the root cause.

This is the failure cause of comment 0 step 6, the luks image contains some metadata such as key slots and encryption params, so a exactly same size block device is not big enough.
But as comment 0 step 7 shows, if we use a bigger block device, the backup job will hit another failure - "Source and target image have different sizes". This means the backup job cannot be sucessful anyway. 
Per Perter said in comment 2, we may need to allow target block device larger than source image when do backup to avoid such kind of problem.
So the problematic step should be comment 0 step 7 now. Thanks.

(In reply to yisun from comment #5)
> (In reply to aihua liang from comment #4)
> > Hi, Sunyi
> >  I reproduce it by following steps:
> >  1.Create a 100M image
> >    #dd if=/dev/urandom of=disk.img 100M
> > 
> >  2.Mount this image to /dev/loop1
> >    #losetup /dev/loop1 /home/disk.img
> > 
> >  3.Create luks image on this device
> >    #qemu-img create --object secret,id=sec0,data=backing -f luks /dev/loop1
> > -o key-secret=sec0 100M
> > Formatting '/dev/loop1', fmt=luks size=104857600 key-secret=sec0
> > qemu-img: /dev/loop1: Cannot grow device files
> > 
> >  I think maybe it's the root cause.
> 
> This is the failure cause of comment 0 step 6, the luks image contains some
> metadata such as key slots and encryption params, so a exactly same size
> block device is not big enough.
> But as comment 0 step 7 shows, if we use a bigger block device, the backup
> job will hit another failure - "Source and target image have different
> sizes". This means the backup job cannot be sucessful anyway. 
> Per Perter said in comment 2, we may need to allow target block device
> larger than source image when do backup to avoid such kind of problem.
> So the problematic step should be comment 0 step 7 now. Thanks.

Hi, SunYi

  Please read comment 3, for step7, it's not a bug, just be a behavior change.
  Now, we must do mirror/backup with the same size between src and dst.
  So, I think libvirt should make some changes to adapt this behavior change.

BR,
Aliang

(In reply to aihua liang from comment #6)
> Hi, SunYi
> 
>   Please read comment 3, for step7, it's not a bug, just be a behavior
> change.
>   Now, we must do mirror/backup with the same size between src and dst.
>   So, I think libvirt should make some changes to adapt this behavior change.

Libvirt is not in control of the block device that is passed so there's nothing we can do on that front. Arguably I'd consider this as a regression, but if it's intended behaviour change it should be explained/justified and closed as a qemu bug.